使用httpclient返回 header too long

最近些的爬蟲請求某代理網站時運行幾天就會持續報錯400,header too long，google無答案，於是看了下源碼，主要原因為cookie的累積導致（可以理解為你的瀏覽器很久沒有清理緩存），以下為排查過程，解決方案見文章最後。

httclient請求調用鏈路：

org.apache.http.impl.client.InternalHttpClient#doExecute

org.apache.http.impl.client.InternalHttpClient#setupContext

if (context.getAttribute(HttpClientContext.COOKIE_STORE) == null) {
context.setAttribute(HttpClientContext.COOKIE_STORE, this.cookieStore);
}

如果沒有顯示設置cooki_store，取this(httpclient)的成員變數cookieStore，而通常我們httpClient只有1個實例，那麼cookieStore也等同於是單例的。

org.apache.http.impl.execchain.RedirectExec#execute:108

org.apache.http.impl.execchain.RetryExec#execute:86

org.apache.http.impl.execchain.ProtocolExec#execute

org.apache.http.impl.execchain.MainClientExec#execute

org.apache.http.protocol.HttpRequestExecutor#execute

conn.sendRequestHeader(request);

org.apache.http.protocol.HttpRequestExecutor#doSendRequest

org.apache.http.impl.conn.CPoolProxy#sendRequestHeader

org.apache.http.impl.io.AbstractMessageWriter#write

Cookie的發送：

for (final HeaderIterator it = message.headerIterator; it.hasNext; ) {
final Header header = it.nextHeader;
this.sessionBuffer.writeLine
(lineFormatter.formatHeader(this.lineBuf, header));
}

Cookie的保存：

org.apache.http.impl.execchain.ProtocolExec#execute:200

org.apache.http.HttpResponseInterceptor#process

org.apache.http.client.protocol.ResponseProcessCookies#process

org.apache.http.client.protocol.ResponseProcessCookies#processCookies:114

cookieStore.addCookie(cookie);

/**
* Adds an {@link Cookie HTTP cookie}, replacing any existing equivalent cookies.
* If the given cookie has already expired it will not be added, but existing
* values will still be removed.
*
* @param cookie the {@link Cookie cookie} to be added
*
* @see #addCookies(Cookie[])
*
*/
public synchronized void addCookie(final Cookie cookie) {
if (cookie != null) {
// first remove any old cookie that is equivalent
cookies.remove(cookie);
if (!cookie.isExpired(new Date)) {
cookies.add(cookie);
}
}
}

注意到這裡添加一個cookie時會先移除，然後判斷cookie是否已經失效，沒有失效才會add，這樣看是不會出問題的，那問題到底出在哪裡？而通過調試發現我們的第三方網站的sessionID的cookie的name居然是會變的！導致老的cookie無法刪除，越積越多。

解決方案①：禁用cookie

CloseableHttpClient httpClient = HttpClientBuilder.create.setConnectionManager(connManager)
.setRetryHandler(retryHandler).setDefaultRequestConfig(config).disableCookieManagement.build;

disableCookieManagement方法會停止發送和接收cookie。

/**
* Disables state (cookie) management.
*

* Please note this value can be overridden by the {@link #setHttpProcessor(
* org.apache.http.protocol.HttpProcessor)} method.
*/
public final HttpClientBuilder disableCookieManagement {
this.cookieManagementDisabled = true;
return this;
}

啟用後org.apache.http.protocol.ImmutableHttpProcessor#responseInterceptors response攔截器里不再包含ResponseProcessCookies這一攔截器，不再執行存儲cookie操作，觀察後續請求，header里也不再包含cookie欄位。

攔截器註冊代碼：org.apache.http.impl.client.HttpClientBuilder#build:839

if (!cookieManagementDisabled) {
b.add(new RequestAddCookies);
}
if (!cookieManagementDisabled) {
b.add(new ResponseProcessCookies);
}

解放方案②：設置單獨的context

HttpClientContext context = HttpClientContext.create;
context.setCookieStore(new BasicCookieStore);
CloseableHttpResponse response = httpClient.execute(httpGet, context);

設置後因為context的cookieStore不為null，將不再默認取httpclient的成員變數cookiestore。

以上兩種方案，可根據自身情況進行選擇。

喜歡這篇文章嗎？立刻分享出去讓更多人知道吧！