Is your httpclient really used correctly?

When I used java to develop a toolkit like scrapy, I used httpclient to make a network request. I encountered the problem of requesting an infinite card death. I took out the solution today and avoided other people stepping on the pit.

Questions are as follows:
1
RequestConfig.custom().setSocketTimeout(SO_TIME_OUT).setConnectTimeout(CONNECTION_TIME_OUT).setConnectionRequestTimeout(CONNECTION_REQUEST_TIME_OUT)

In the regular timeout configuration such as socketTimeout, connectTimeout and connectionRequestTimeout, in the case of large concurrency, some requests will be stuck in the java.net.SocketInputStream.socketRead0 method from time to time. The information dumped is as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
"pool-2-thread-87" #202 prio=5 os_prio=0 tid=0x00007f52603a8000 nid=0x6672 runnable [0x00007f51888c6000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
at org.apache.http.impl.execchain.MainClientExec.createTunnelToTarget(MainClientExec.java:486)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:411)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
Check if you are stepping on the pit

You can run the command: jstack -l ${pid} | grep java.net.SocketInputStream.socketRead0
If multiple dump messages have the same thread stack, you can determine that this problem also exists for your service.

solution
  1. If it is https, and the version of httpclient is version 4.3.5, upgrade the httpclient version. This is a bug of httpclient. For details, see https://issues.apache.org/jira/browse/HTTPCLIENT-1589, this is the first possible solution.
  2. If the first one cannot be solved, it is basically this possibility. I am using httpclient proxy request, after numerous debugs, I found the proxy request in the TUNNEL_TARGET step, the connection used by the DefaultBHttpClientConnection binding, using the SocketConfig configuration, not RequestConfig, so without setting SocketConfig The socket’s ocketRead0 method is infinitely waiting, causing the thread to be stuck. Add the following configuration:
1
connectionManager.setSocketConfig(SocketConfig.custom().setSoTimeout(SO_TIME_OUT).build());

In addition, any use of socket, you may need to pay attention to whether the socket timeout is set, otherwise it will always appear socketRead0.

original text