linkerd-tcp connection reuse issue

[stevej]

Running a load test without connection reuse (this is part of my testing out the tacho OrderMap changes) and linkerd-tcp starts rejecting traffic after a thousand requests.

stevej@netty-test-8:~$ ./slow_cooker_linux_amd64 -noreuse -host 'default' -totalRequests 1000000 -qps 100 -concurrency 100 http://proxy-test-4d:7474/
# sending 10000 req/s with concurrency=100 to http://proxy-test-4d:7474/ ...
#                      good/b/f t   good%   min [p50 p95 p99  p999]  max change
2017-04-20T21:47:03Z  14105/0/0 100000  14% 10s   5 [ 44  47  48   54 ]   54 +
2017-04-20T21:47:13Z      0/0/0 100000   0% 10s   0 [  0   0   0    0 ]    0 -
2017-04-20T21:47:23Z   1039/0/0 100000   1% 10s  40 [ 54 15319 15327 15335 ] 15329 +++
2017-04-20T21:47:33Z    780/0/0 100000   0% 10s  40 [1060 3081 3083 3083 ] 3083
2017-04-20T21:47:43Z    659/0/0 100000   0% 10s  44 [1048 3073 3075 4073 ] 4073

CPU was at 1% and traffic was trickling through. Here's some example strace output

recvfrom(78, 0x7fea37c5f000, 8192, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(70, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 8192, 0, NULL, NULL) = 859
sendto(58, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 859, MSG_NOSIGNAL, NULL, 0) = 859
recvfrom(70, 0x7fea37c5f000, 8192, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(65, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 8192, 0, NULL, NULL) = 859
sendto(51, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 859, MSG_NOSIGNAL, NULL, 0) = 859
recvfrom(65, 0x7fea37c5f000, 8192, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(60, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 8192, 0, NULL, NULL) = 859
sendto(46, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 859, MSG_NOSIGNAL, NULL, 0) = 859
recvfrom(60, 0x7fea37c5f000, 8192, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(53, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 8192, 0, NULL, NULL) = 859
sendto(39, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 859, MSG_NOSIGNAL, NULL, 0) = 859
recvfrom(53, 0x7fea37c5f000, 8192, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(48, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 8192, 0, NULL, NULL) = 859
sendto(33, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 859, MSG_NOSIGNAL, NULL, 0) = 859
recvfrom(48, 0x7fea37c5f000, 8192, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(41, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 8192, 0, NULL, NULL) = 859
sendto(27, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 859, MSG_NOSIGNAL, NULL, 0) = 859
recvfrom(41, 0x7fea37c5f000, 8192, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(36, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 8192, 0, NULL, NULL) = 859
sendto(20, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 859, MSG_NOSIGNAL, NULL, 0) = 859
recvfrom(36, 0x7fea37c5f000, 8192, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(30, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 8192, 0, NULL, NULL) = 859
sendto(17, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 859, MSG_NOSIGNAL, NULL, 0) = 859
recvfrom(30, 0x7fea37c5f000, 8192, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
recvfrom(23, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 8192, 0, NULL, NULL) = 859
sendto(14, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 859, MSG_NOSIGNAL, NULL, 0) = 859
recvfrom(23, 0x7fea37c5f000, 8192, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)

and an lsof report

stevej@proxy-test-4d:~$ sudo lsof -p 14178
COMMAND     PID   USER   FD      TYPE    DEVICE SIZE/OFF      NODE NAME
linkerd-t 14178 stevej  cwd       DIR       8,1     4096   1812484 /home/stevej/src/linkerd-tcp
linkerd-t 14178 stevej  rtd       DIR       8,1     4096         2 /
linkerd-t 14178 stevej  txt       REG       8,1  9851408   1816829 /home/stevej/src/linkerd-tcp/target/release/
linkerd-tcp
linkerd-t 14178 stevej  mem       REG       8,1  1088952      1907 /lib/x86_64-linux-gnu/libm-2.24.so
linkerd-t 14178 stevej  mem       REG       8,1  1856752      1903 /lib/x86_64-linux-gnu/libc-2.24.so
linkerd-t 14178 stevej  mem       REG       8,1    92552      1931 /lib/x86_64-linux-gnu/libgcc_s.so.1
linkerd-t 14178 stevej  mem       REG       8,1   142400      1918 /lib/x86_64-linux-gnu/libpthread-2.24.so
linkerd-t 14178 stevej  mem       REG       8,1    31712      1920 /lib/x86_64-linux-gnu/librt-2.24.so
linkerd-t 14178 stevej  mem       REG       8,1    14608      1906 /lib/x86_64-linux-gnu/libdl-2.24.so
linkerd-t 14178 stevej  mem       REG       8,1   158512      1899 /lib/x86_64-linux-gnu/ld-2.24.so
linkerd-t 14178 stevej    0u      CHR     136,3      0t0         6 /dev/pts/3
linkerd-t 14178 stevej    1u      CHR     136,3      0t0         6 /dev/pts/3
linkerd-t 14178 stevej    2u      CHR     136,3      0t0         6 /dev/pts/3
linkerd-t 14178 stevej    3u  a_inode      0,11        0      8469 [eventpoll]
linkerd-t 14178 stevej    4r     FIFO      0,10      0t0 908911732 pipe
linkerd-t 14178 stevej    5w     FIFO      0,10      0t0 908911732 pipe
linkerd-t 14178 stevej    6u     IPv4 908911733      0t0       TCP *:7474 (LISTEN)
linkerd-t 14178 stevej    7u  a_inode      0,11        0      8469 [eventpoll]
linkerd-t 14178 stevej    8r     FIFO      0,10      0t0 908910932 pipe
linkerd-t 14178 stevej    9w     FIFO      0,10      0t0 908910932 pipe
linkerd-t 14178 stevej   10u     IPv4 908911215      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33709 (ESTABLISHED)
linkerd-t 14178 stevej   11u     IPv4 908911205      0t0       TCP proxy-test-4d:54924->perf-target-4.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej   12u     IPv4 908910933      0t0       TCP *:9989 (LISTEN)
linkerd-t 14178 stevej   13u     IPv4 908910934      0t0       TCP localhost:50620->localhost:4180 (ESTABLISHED
)
linkerd-t 14178 stevej   14u     IPv4 908910257      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33616 (ESTABLISHED)
linkerd-t 14178 stevej   15u     IPv4 908910258      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33604 (ESTABLISHED)
linkerd-t 14178 stevej   16u     IPv4 908910259      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33609 (ESTABLISHED)
linkerd-t 14178 stevej   17u     IPv4 908910260      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33622 (ESTABLISHED)
linkerd-t 14178 stevej   18u     IPv4 908910261      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33624 (ESTABLISHED)
linkerd-t 14178 stevej   19u     IPv4 908910262      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33620 (ESTABLISHED)
[cutting hundreds of other sockets that are open both upstream and downstream]
linkerd-t 14178 stevej  327u     IPv4 908911328      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33773 (ESTABLISHED)
linkerd-t 14178 stevej  328u     IPv4 908911331      0t0       TCP proxy-test-4d:42162->perf-target-5.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  329u     IPv4 908911332      0t0       TCP proxy-test-4d:7474->netty-test-8.c[124/1893]osted.internal:33770 (ESTABLISHED)
linkerd-t 14178 stevej  330u     IPv4 908911333      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33772 (ESTABLISHED)
linkerd-t 14178 stevej  331u     IPv4 908911334      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33775 (ESTABLISHED)
linkerd-t 14178 stevej  332u     IPv4 908911335      0t0       TCP proxy-test-4d:38944->perf-target-6.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  333u     IPv4 908911336      0t0       TCP proxy-test-4d:55042->perf-target-4.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  334u     IPv4 908911337      0t0       TCP proxy-test-4d:42168->perf-target-5.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  335u     IPv4 908911338      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33769 (ESTABLISHED)
linkerd-t 14178 stevej  336u     IPv4 908911339      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33771 (ESTABLISHED)
linkerd-t 14178 stevej  337u     IPv4 908911340      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33765 (ESTABLISHED)
linkerd-t 14178 stevej  338u     IPv4 908911343      0t0       TCP proxy-test-4d:38954->perf-target-6.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  339u     IPv4 908911344      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33777 (ESTABLISHED)
linkerd-t 14178 stevej  340u     IPv4 908911345      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33778 (ESTABLISHED)
linkerd-t 14178 stevej  341u     IPv4 908911346      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33776 (ESTABLISHED)
linkerd-t 14178 stevej  342u     IPv4 908911347      0t0       TCP proxy-test-4d:38956->perf-target-6.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  343u     IPv4 908911348      0t0       TCP proxy-test-4d:55054->perf-target-4.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  344u     IPv4 908911349      0t0       TCP proxy-test-4d:42180->perf-target-5.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  345u     IPv4 908911350      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33780 (ESTABLISHED)
linkerd-t 14178 stevej  346u     IPv4 908911351      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33782 (ESTABLISHED)
linkerd-t 14178 stevej  347u     IPv4 908911352      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33781 (ESTABLISHED)
linkerd-t 14178 stevej  348u     IPv4 908911353      0t0       TCP proxy-test-4d:42182->perf-target-5.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  349u     IPv4 908911354      0t0       TCP proxy-test-4d:55060->perf-target-4.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  350u     IPv4 908911355      0t0       TCP proxy-test-4d:38966->perf-target-6.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  351u     IPv4 908911356      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33779 (ESTABLISHED)
linkerd-t 14178 stevej  352u     IPv4 908911357      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33785 (ESTABLISHED)
linkerd-t 14178 stevej  353u     IPv4 908911358      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33783 (ESTABLISHED)
linkerd-t 14178 stevej  354u     IPv4 908911359      0t0       TCP proxy-test-4d:38968->perf-target-6.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  355u     IPv4 908911360      0t0       TCP proxy-test-4d:55066->perf-target-4.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  356u     IPv4 908911361      0t0       TCP proxy-test-4d:42192->perf-target-5.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  357u     IPv4 908911362      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33784 (ESTABLISHED)
linkerd-t 14178 stevej  358u     IPv4 908911363      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33790 (ESTABLISHED)
linkerd-t 14178 stevej  359u     IPv4 908911364      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33787 (ESTABLISHED)
linkerd-t 14178 stevej  360u     IPv4 908911365      0t0       TCP proxy-test-4d:55070->perf-target-4.[62/1893]-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  361u     IPv4 908911366      0t0       TCP proxy-test-4d:38976->perf-target-6.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  362u     IPv4 908911367      0t0       TCP proxy-test-4d:42198->perf-target-5.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  363u     IPv4 908911368      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33788 (ESTABLISHED)
linkerd-t 14178 stevej  364u     IPv4 908911369      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33789 (ESTABLISHED)
linkerd-t 14178 stevej  365u     IPv4 908911370      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33774 (ESTABLISHED)
linkerd-t 14178 stevej  366u     IPv4 908911371      0t0       TCP proxy-test-4d:55076->perf-target-4.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  367u     IPv4 908911372      0t0       TCP proxy-test-4d:38982->perf-target-6.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  368u     IPv4 908911373      0t0       TCP proxy-test-4d:42204->perf-target-5.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  369u     IPv4 908911374      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33793 (ESTABLISHED)
linkerd-t 14178 stevej  370u     IPv4 908911375      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33792 (ESTABLISHED)
linkerd-t 14178 stevej  371u     IPv4 908911376      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33786 (ESTABLISHED)
linkerd-t 14178 stevej  372u     IPv4 908911377      0t0       TCP proxy-test-4d:55082->perf-target-4.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  373u     IPv4 908911378      0t0       TCP proxy-test-4d:38988->perf-target-6.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  374u     IPv4 908911379      0t0       TCP proxy-test-4d:42210->perf-target-5.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  375u     IPv4 908911380      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33794 (ESTABLISHED)
linkerd-t 14178 stevej  376u     IPv4 908911381      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33797 (ESTABLISHED)
linkerd-t 14178 stevej  377u     IPv4 908911382      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33798 (ESTABLISHED)
linkerd-t 14178 stevej  378u     IPv4 908911383      0t0       TCP proxy-test-4d:42212->perf-target-5.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  379u     IPv4 908911384      0t0       TCP proxy-test-4d:55090->perf-target-4.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  380u     IPv4 908911385      0t0       TCP proxy-test-4d:38996->perf-target-6.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  381u     IPv4 908911386      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33801 (ESTABLISHED)
linkerd-t 14178 stevej  382u     IPv4 908911387      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33799 (ESTABLISHED)
linkerd-t 14178 stevej  383u     IPv4 908911388      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33795 (ESTABLISHED)
linkerd-t 14178 stevej  384u     IPv4 908911389      0t0       TCP proxy-test-4d:38998->perf-target-6.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  385u     IPv4 908911390      0t0       TCP proxy-test-4d:55096->perf-target-4.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  386u     IPv4 908911391      0t0       TCP proxy-test-4d:42222->perf-target-5.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  387u     IPv4 908911392      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33791 (ESTABLISHED)
linkerd-t 14178 stevej  388u     IPv4 908911393      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33796 (ESTABLISHED)
linkerd-t 14178 stevej  389u     IPv4 908911394      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-h
osted.internal:33800 (ESTABLISHED)
linkerd-t 14178 stevej  390u     IPv4 908911395      0t0       TCP proxy-test-4d:55100->perf-target-4.c.buoyant
-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  391u     IPv4 908911396      0t0       TCP proxy-test-4d:39006->perf-target-6.c.buoyant-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  392u     IPv4 908911397      0t0       TCP proxy-test-4d:42228->perf-target-5.c.buoyant-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  393u     IPv4 908911398      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-hosted.internal:33802 (ESTABLISHED)
linkerd-t 14178 stevej  394u     IPv4 908911399      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-hosted.internal:33807 (ESTABLISHED)
linkerd-t 14178 stevej  395u     IPv4 908911400      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-hosted.internal:33803 (ESTABLISHED)
linkerd-t 14178 stevej  396u     IPv4 908911401      0t0       TCP proxy-test-4d:55106->perf-target-4.c.buoyant-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  397u     IPv4 908911402      0t0       TCP proxy-test-4d:39012->perf-target-6.c.buoyant-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  398u     IPv4 908911403      0t0       TCP proxy-test-4d:42234->perf-target-5.c.buoyant-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  399u     IPv4 908911404      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-hosted.internal:33805 (ESTABLISHED)
linkerd-t 14178 stevej  400u     IPv4 908911405      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-hosted.internal:33806 (ESTABLISHED)
linkerd-t 14178 stevej  401u     IPv4 908911406      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-hosted.internal:33804 (ESTABLISHED)
linkerd-t 14178 stevej  402u     IPv4 908911407      0t0       TCP proxy-test-4d:55112->perf-target-4.c.buoyant-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  403u     IPv4 908911408      0t0       TCP proxy-test-4d:39018->perf-target-6.c.buoyant-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  404u     IPv4 908911409      0t0       TCP proxy-test-4d:42240->perf-target-5.c.buoyant-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  405u     IPv4 908911410      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-hosted.internal:33808 (ESTABLISHED)
linkerd-t 14178 stevej  406u     IPv4 909244034      0t0       TCP proxy-test-4d:44928->perf-target-5.c.buoyant-hosted.internal:http-alt (SYN_SENT)
linkerd-t 14178 stevej  407u     sock       0,8      0t0 909253650 protocol: TCP
linkerd-t 14178 stevej  408u     IPv4 908911413      0t0       TCP proxy-test-4d:39022->perf-target-6.c.buoyant-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  409u     IPv4 908911414      0t0       TCP proxy-test-4d:55120->perf-target-4.c.buoyant-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  410u     IPv4 908911415      0t0       TCP proxy-test-4d:42246->perf-target-5.c.buoyant-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  411u     IPv4 908911417      0t0       TCP proxy-test-4d:42248->perf-target-5.c.buoyant-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  412u     IPv4 908911418      0t0       TCP proxy-test-4d:55126->perf-target-4.c.buoyant-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  413u     IPv4 908911419      0t0       TCP proxy-test-4d:39032->perf-target-6.c.buoyant-hosted.internal:http-alt (ESTABLISHED)
linkerd-t 14178 stevej  414u     sock       0,8      0t0 909253652 protocol: TCP
linkerd-t 14178 stevej  415u     sock       0,8      0t0 909253654 protocol: TCP
linkerd-t 14178 stevej  416u     sock       0,8      0t0 909253637 protocol: TCP
linkerd-t 14178 stevej  417u  unknown                              /proc/14178/fd/417 (readlink: No such file or directory)
linkerd-t 14178 stevej  418u     sock       0,8      0t0 909253639 protocol: TCP
linkerd-t 14178 stevej  419u     sock       0,8      0t0 909252607 protocol: TCP
linkerd-t 14178 stevej  420u     sock       0,8      0t0 909253647 protocol: TCP
linkerd-t 14178 stevej  421u     sock       0,8      0t0 909253645 protocol: TCP
linkerd-t 14178 stevej  422u     sock       0,8      0t0 909253649 protocol: TCP
linkerd-t 14178 stevej  423u     IPv4 909235806      0t0       TCP proxy-test-4d:33498->perf-target-6.c.buoyant-hosted.internal:http-alt (SYN_SENT)
linkerd-t 14178 stevej  424u     sock       0,8      0t0 909253641 protocol: TCP
linkerd-t 14178 stevej  425u     sock       0,8      0t0 909253643 protocol: TCP
linkerd-t 14178 stevej  427u     sock       0,8      0t0 909253653 protocol: TCP
linkerd-t 14178 stevej  428u     IPv4 909253633      0t0       TCP proxy-test-4d:7474->netty-test-8.c.buoyant-hosted.internal:39840 (ESTABLISHED)
linkerd-t 14178 stevej  429u     sock       0,8      0t0 909253635 protocol: TCP

[stevej]

Good news: the same process will happily route requests after being in this state

stevej@netty-test-8:~$ ./slow_cooker_linux_amd64 -host 'default' -totalRequests 1000000 -qps 100 -concurrency 100 http://proxy-test-4d:7474/
# sending 10000 req/s with concurrency=100 to http://proxy-test-4d:7474/ ...
#                      good/b/f t   good%   min [p50 p95 p99  p999]  max change
2017-04-20T22:02:32Z  99667/0/0 100000  99% 10s   0 [  2   4   4   74 ]   74 +
2017-04-20T22:02:42Z 100000/0/0 100000 100% 10s   0 [  2   4   5   13 ]   13 +
2017-04-20T22:02:52Z 100002/0/0 100000 100% 10s   0 [  2   4   4   14 ]   14
2017-04-20T22:03:02Z 100000/0/0 100000 100% 10s   0 [  2   4   4    9 ]    9
2017-04-20T22:03:12Z 100000/0/0 100000 100% 10s   0 [  2   4   4   12 ]   12
2017-04-20T22:03:22Z 100000/0/0 100000 100% 10s   0 [  2   4   5   11 ]   11
2017-04-20T22:03:32Z 100000/0/0 100000 100% 10s   0 [  2   4   5   11 ]   11
2017-04-20T22:03:42Z 100002/0/0 100000 100% 10s   0 [  2   4   5    9 ]    9
2017-04-20T22:03:52Z 100044/0/0 100000 100% 10s   0 [  2   4   5    9 ]    9
2017-04-20T22:04:02Z  99975/0/0 100000  99% 10s   0 [  2   4   5    9 ]    9
2017-04-20T22:04:12Z 100010/0/0 100000 100% 10s   0 [  2   4   5    8 ]    8
FROM    TO #REQUESTS
   0     2 167367
   2     8 932083
   8    32 196
  32    64 41
  64   128 13
 128   256 0
 256   512 0
 512  1024 0
1024  4096 0
4096 16384 0