西刺代理是一個國內(nèi)IP代理，由于代理倒閉了，所以我就把原來的代碼放出來供大家學(xué)習(xí)吧。
鏡像地址：https://www.blib.cn/url/xcdl.html
首先找到所有的tr標(biāo)簽，與class="odd"的標(biāo)簽，然后提取出來。
然后再依次找到tr標(biāo)簽里面的所有td標(biāo)簽，然后只提取出里面的[1,2,5,9]這四個標(biāo)簽的位置，其他的不提取。
最后可以寫出提取單一頁面的代碼，提取后將其保存到文件中。
import sys,re,threading
import requests,lxml
from queue import Queue
import argparse
from bs4 import BeautifulSoup
head = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.100 Safari/537.36"}
if __name__ == "__main__":
ip_list=[]
fp = open("SpiderAddr.json","a+",encoding="utf-8")
url = "https://www.blib.cn/url/xcdl.html"
request = requests.get(url=url,headers=head)
soup = BeautifulSoup(request.content,"lxml")
data = soup.find_all(name="tr",attrs={"class": re.compile("|[^odd]")})
for item in data:
soup_proxy = BeautifulSoup(str(item),"lxml")
proxy_list = soup_proxy.find_all(name="td")
for i in [1,2,5,9]:
ip_list.append(proxy_list[i].string)
print("[+] 爬行列表: {} 已轉(zhuǎn)存".format(ip_list))
fp.write(str(ip_list) + '\n')
ip_list.clear()
爬取后會將文件保存為 SpiderAddr.json 格式。
最后再使用另一段代碼，將其轉(zhuǎn)換為一個SSR代理工具直接能識別的格式，{'http': 'http://119.101.112.31:9999'}
import sys,re,threading
import requests,lxml
from queue import Queue
import argparse
from bs4 import BeautifulSoup
if __name__ == "__main__":
result = []
fp = open("SpiderAddr.json","r")
data = fp.readlines()
for item in data:
dic = {}
read_line = eval(item.replace("\n",""))
Protocol = read_line[2].lower()
if Protocol == "http":
dic[Protocol] = "http://" + read_line[0] + ":" + read_line[1]
else:
dic[Protocol] = "https://" + read_line[0] + ":" + read_line[1]
result.append(dic)
print(result)
完整多線程版代碼如下所示。
import sys,re,threading
import requests,lxml
from queue import Queue
import argparse
from bs4 import BeautifulSoup
head = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.100 Safari/537.36"}
class AgentSpider(threading.Thread):
def __init__(self,queue):
threading.Thread.__init__(self)
self._queue = queue
def run(self):
ip_list=[]
fp = open("SpiderAddr.json","a+",encoding="utf-8")
while not self._queue.empty():
url = self._queue.get()
try:
request = requests.get(url=url,headers=head)
soup = BeautifulSoup(request.content,"lxml")
data = soup.find_all(name="tr",attrs={"class": re.compile("|[^odd]")})
for item in data:
soup_proxy = BeautifulSoup(str(item),"lxml")
proxy_list = soup_proxy.find_all(name="td")
for i in [1,2,5,9]:
ip_list.append(proxy_list[i].string)
print("[+] 爬行列表: {} 已轉(zhuǎn)存".format(ip_list))
fp.write(str(ip_list) + '\n')
ip_list.clear()
except Exception:
pass
def StartThread(count):
queue = Queue()
threads = []
for item in range(1,int(count)+1):
url = "https://www.xicidaili.com/nn/{}".format(item)
queue.put(url)
print("[+] 生成爬行鏈接 {}".format(url))
for item in range(count):
threads.append(AgentSpider(queue))
for t in threads:
t.start()
for t in threads:
t.join()
# 轉(zhuǎn)換函數(shù)
def ConversionAgentIP(FileName):
result = []
fp = open(FileName,"r")
data = fp.readlines()
for item in data:
dic = {}
read_line = eval(item.replace("\n",""))
Protocol = read_line[2].lower()
if Protocol == "http":
dic[Protocol] = "http://" + read_line[0] + ":" + read_line[1]
else:
dic[Protocol] = "https://" + read_line[0] + ":" + read_line[1]
result.append(dic)
return result
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument("-p","--page",dest="page",help="指定爬行多少頁")
parser.add_argument("-f","--file",dest="file",help="將爬取到的結(jié)果轉(zhuǎn)化為代理格式 SpiderAddr.json")
args = parser.parse_args()
if args.page:
StartThread(int(args.page))
elif args.file:
dic = ConversionAgentIP(args.file)
for item in dic:
print(item)
else:
parser.print_help()
以上就是python多線程爬取西刺代理的示例代碼的詳細(xì)內(nèi)容，更多關(guān)于python多線程爬取代理的資料請關(guān)注腳本之家其它相關(guān)文章！
您可能感興趣的文章:- Python基礎(chǔ)進(jìn)階之海量表情包多線程爬蟲功能的實現(xiàn)
- python3爬蟲中多線程進(jìn)行解鎖操作實例
- Python如何使用隊列方式實現(xiàn)多線程爬蟲
- python爬蟲開發(fā)之使用Python爬蟲庫requests多線程抓取貓眼電影TOP100實例
- python支持多線程的爬蟲實例
- python爬蟲中多線程的使用詳解
- python面向?qū)ο蠖嗑€程爬蟲爬取搜狐頁面的實例代碼
- python爬蟲爬取快手視頻多線程下載功能
- Python之多線程爬蟲抓取網(wǎng)頁圖片的示例代碼
- Python3多線程爬蟲實例講解代碼
- php與python實現(xiàn)的線程池多線程爬蟲功能示例
- Python 爬蟲多線程詳解及實例代碼
