用Python做大批量請求發送

大批量請求發送需要考慮的幾個因素:

1. 伺服器承載能力(網路帶寬/硬體配置);

2. 客戶端IO情況, 客戶端帶寬, 硬體配置;

方案:

1. 方案都是相對的;

2. 因為這裡我的情況是客戶機只有一台,所以不能考慮使用分散式了, 伺服器承載能力也非常有限(經過不斷調試得知);

3. 這裡沒有使用Jmeter, 雖然jmeter也是可以做到的.

Python裡面支持發送大批量的方案有很多, 這裡只介紹我所用過的幾種:

grequests可以一次性發送超大批量的請求, 但是底層聽說修改了socket通信, 可能不穩定或者不安全? 而且如果你需要校驗對比每個請求的發送信息與返回信息, 比較不方便, 因為它是批量發送,然後批量收回, 示例代碼:

1 import grequests
2 import time
3 from collections import OrderedDict
4 import hashlib
5 import os
6 import xlrd
7 import json
8 import datetime
9
10 class voiceSearchInterface:
11 @classmethod
12 def formatUrlAndHeader(self, des, singer, songName):
13 #演算法邏輯, 根據傳進來的參數生成url和header45 return url, h
46
47 @classmethod
48 def exeRequests(self):
49 errorLog = open("errorLog.txt","w+")
50 startTime = datetime.datetime.now
51 rightCount = 0
52 errCount = 0
53 descr = ["播放", "搜索", "搜", "聽", "我要聽", "我想聽", "來一首", "來一個", "來一段", "來一曲", "來首", "來個", "來段", "來曲"]
54 orgPath = os.path.join(os.path.dirname(os.getcwd), "test","SongsAndSingers","singersAndSongs3.txt")
55 f = open(orgPath,"rb")
56 i = 0
57 urlsAndHs =
58 for line in f.readlines:
59 temp = line.decode.split(" ")
60 orgSinger = temp[0]
61 orgSong = temp[1].replace("
","")
62 for k in descr:
63 urlAndH = self.formatUrlAndHeader(k, orgSinger,orgSong)
64 urlsAndHs.append(urlAndH)
65 f.close
66 rs = (grequests.get(u[0], headers = u[1], stream = False) for u in urlsAndHs)
#size表示一次發出去的請求數量 67 rsText = grequests.imap(rs, size=20) 68 for r in rsText: 69 executingLog = open("Log.txt","w+") 70 i+=1 71 try: 72 searchResult = json.loads(r.text) 73 searchItem = searchResult["data"]["searchitem"] 74 tt = searchItem.split("Split") 75 searchSinger = tt[1] 76 searchSong = tt[-1] 77 resultSinger = searchResult["data"]["sounds"][0]["singer"] 78 resultSong = searchResult["data"]["sounds"][0]["title"] 79 if(searchSinger==resultSinger and searchSong==resultSong): 80 81 rightCount += 1 82 else: 83 84 errCount += 1 85 print(searchSinger, " ",resultSinger, " ",searchSong," ", resultSong) 86 except Exception: 87 errCount += 1 88 errorLog.write((r.text+"
").encode("latin-1").decode("unicode_escape")) 89 print(i) 90 executingLog.write(str(int(i/14))) 91 errorLog.close 92 executingLog.close 93 endTime = datetime.datetime.now 94 print("耗時: %d秒, 正確數: %d, 異常數: %d, 總數: %d, 通過率: %.2f%%" % ((endTime-startTime).seconds, rightCount, errCount, i, (rightCount)/i*100)) 95 96 voiceSearchInterface.exeRequests

喜歡這篇文章嗎？立刻分享出去讓更多人知道吧！