继续超短流水账一篇。
今天午饭时间，一个向来非常稳定的Flink on YARN任务忽然持续报警。查看TaskManager日志均没有问题，但JobManager日志内报出大量Connection reset by peer信息，其中更夹杂着奇怪的错误，如下图所示。
但是作业是一直正常运行的。根据日志，可以凭直觉推测JobManager的REST endpoint出了问题(遭到了RST攻击？)
登录到JM container所在的机器，通过netstat和lsof命令找出其PID和监听的端口，发现端口号是4347。然后用tcpdump命令抓包：
tcpdump -i eth0 tcp port 4347 -XX -vv >> dump.out抓出的一部分报文如下：
12:17:59.434870 IP (tos 0x0, ttl 61, id 34630, offset 0, flags [DF], proto TCP (6), length 60)172.16.200.34.36762 > ec-bigdata-flink-worker-040.lansurveyor: Flags [S], cksum 0xaa12 (correct), seq 611029849, win 64240, options [mss 1460,sackOK,TS val 285643676 ecr 0,nop,wscale 7], length 00x0000: 0016 3e34 f380 eeff ffff ffff 0800 4500 ..>4..........E.0x0010: 003c 8746 4000 3d06 1076 ac10 c822 0a00 .<.F@.=..v..."..0x0020: 27cd 8f9a 10fb 246b 9359 0000 0000 a002 '.....$k.Y......0x0030: faf0 aa12 0000 0204 05b4 0402 080a 1106 ................0x0040: 939c 0000 0000 0103 0307 ..........
12:17:59.434890 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)ec-bigdata-flink-worker-040.lansurveyor > 172.16.200.34.36762: Flags [S.], cksum 0xa62e (incorrect -> 0x3998), seq 2375752204, ack 611029850, win 28960, options [mss 1460,sackOK,TS val 3208549201 ecr 285643676,nop,wscale 9], length 00x0000: eeff ffff ffff 0016 3e34 f380 0800 4500 ........>4....E.0x0010: 003c 0000 4000 4006 94bc 0a00 27cd ac10 .<..@.@.....'...0x0020: c822 10fb 8f9a 8d9b 1a0c 246b 935a a012 ."........$k.Z..0x0030: 7120 a62e 0000 0204 05b4 0402 080a bf3e q..............>0x0040: 9351 1106 939c 0103 0309 .Q........
12:17:59.440022 IP (tos 0x0, ttl 61, id 34631, offset 0, flags [DF], proto TCP (6), length 52)172.16.200.34.36762 > ec-bigdata-flink-worker-040.lansurveyor: Flags [.], cksum 0xd78b (correct), seq 1, ack 1, win 502, options [nop,nop,TS val 285643681 ecr 3208549201], length 00x0000: 0016 3e34 f380 eeff ffff ffff 0800 4500 ..>4..........E.0x0010: 0034 8747 4000 3d06 107d ac10 c822 0a00 .4.G@.=..}..."..0x0020: 27cd 8f9a 10fb 246b 935a 8d9b 1a0d 8010 '.....$k.Z......0x0030: 01f6 d78b 0000 0101 080a 1106 93a1 bf3e ...............>0x0040: 9351 .Q
12:17:59.448889 IP (tos 0x0, ttl 61, id 34632, offset 0, flags [DF], proto TCP (6), length 431)172.16.200.34.36762 > ec-bigdata-flink-worker-040.lansurveyor: Flags [P.], cksum 0xa379 (correct), seq 1:380, ack 1, win 502, options [nop,nop,TS val 285643690 ecr 3208549201], length 3790x0000: 0016 3e34 f380 eeff ffff ffff 0800 4500 ..>4..........E.0x0010: 01af 8748 4000 3d06 0f01 ac10 c822 0a00 ...H@.=......"..0x0020: 27cd 8f9a 10fb 246b 935a 8d9b 1a0d 8018 '.....$k.Z......0x0030: 01f6 a379 0000 0101 080a 1106 93aa bf3e ...y...........>0x0040: 9351 4745 5420 2f63 6769 2d62 696e 2f63 .QGET./cgi-bin/c0x0050: 6f69 6e5f 696e 636c 7564 6573 2f63 6f6e oin_includes/con0x0060: 7374 616e 7473 2e70 6870 3f5f 4343 4647 stants.php?_CCFG0x0070: 5b5f 504b 475f 5041 5448 5f49 4e43 4c5d [_PKG_PATH_INCL]0x0080: 3d2f 6574 632f 7061 7373 7764 2530 3020 =/etc/passwd%00.0x0090: 4854 5450 2f31 2e31 0d0a 486f 7374 3a20 HTTP/1.1..Host:.0x00a0: 3130 2e30 2e33 392e 3230 353a 3433 3437 10.0.19.105:43470x00b0: 0d0a 4163 6365 7074 2d43 6861 7273 6574 ..Accept-Charset0x00c0: 3a20 6973 6f2d 3838 3539 2d31 2c75 7466 :.iso-8859-1,utf0x00d0: 2d38 3b71 3d30 2e39 2c2a 3b71 3d30 2e31 -8;q=0.9,*;q=0.10x00e0: 0d0a 4163 6365 7074 2d4c 616e 6775 6167 ..Accept-Languag0x00f0: 653a 2065 6e0d 0a43 6f6e 6e65 6374 696f e:.en..Connectio0x0100: 6e3a 204b 6565 702d 416c 6976 650d 0a55 n:.Keep-Alive..U0x0110: 7365 722d 4167 656e 743a 204d 6f7a 696c ser-Agent:.Mozil0x0120: 6c61 2f34 2e30 2028 636f 6d70 6174 6962 la/4.0.(compatib0x0130: 6c65 3b20 4d53 4945 2038 2e30 3b20 5769 le;.MSIE.8.0;.Wi0x0140: 6e64 6f77 7320 4e54 2035 2e31 3b20 5472 ndows.NT.5.1;.Tr0x0150: 6964 656e 742f 342e 3029 0d0a 5072 6167 ident/4.0)..Prag0x0160: 6d61 3a20 6e6f 2d63 6163 6865 0d0a 4163 ma:.no-cache..Ac0x0170: 6365 7074 3a20 696d 6167 652f 6769 662c cept:.image/gif,0x0180: 2069 6d61 6765 2f78 2d78 6269 746d 6170 .image/x-xbitmap0x0190: 2c20 696d 6167 652f 6a70 6567 2c20 696d ,.image/jpeg,.im0x01a0: 6167 652f 706a 7065 672c 2069 6d61 6765 age/pjpeg,.image0x01b0: 2f70 6e67 2c20 2a2f 2a0d 0a0d 0a /png,.*/*....
12:17:59.448904 IP (tos 0x0, ttl 64, id 35162, offset 0, flags [DF], proto TCP (6), length 52)ec-bigdata-flink-worker-040.lansurveyor > 172.16.200.34.36762: Flags [.], cksum 0xa626 (incorrect -> 0xd7b4), seq 1, ack 380, win 59, options [nop,nop,TS val 3208549215 ecr 285643690], length 00x0000: eeff ffff ffff 0016 3e34 f380 0800 4500 ........>4....E.0x0010: 0034 895a 4000 4006 0b6a 0a00 27cd ac10 .4.Z@.@..j..'...0x0020: c822 10fb 8f9a 8d9b 1a0d 246b 94d5 8010 ."........$k....0x0030: 003b a626 0000 0101 080a bf3e 935f 1106 .;.&.......>._..0x0040: 93aa ..
12:17:59.449337 IP (tos 0x0, ttl 64, id 35163, offset 0, flags [DF], proto TCP (6), length 251)ec-bigdata-flink-worker-040.lansurveyor > 172.16.200.34.36762: Flags [P.], cksum 0xa6ed (incorrect -> 0x438a), seq 1:200, ack 380, win 59, options [nop,nop,TS val 3208549216 ecr 285643690], length 1990x0000: eeff ffff ffff 0016 3e34 f380 0800 4500 ........>4....E.0x0010: 00fb 895b 4000 4006 0aa2 0a00 27cd ac10 ...[@.@.....'...0x0020: c822 10fb 8f9a 8d9b 1a0d 246b 94d5 8018 ."........$k....0x0030: 003b a6ed 0000 0101 080a bf3e 9360 1106 .;.........>.`..0x0040: 93aa 4854 5450 2f31 2e31 2034 3034 204e ..HTTP/1.1.404.N0x0050: 6f74 2046 6f75 6e64 0d0a 436f 6e74 656e ot.Found..Conten0x0060: 742d 5479 7065 3a20 6170 706c 6963 6174 t-Type:.applicat0x0070: 696f 6e2f 6a73 6f6e 3b20 6368 6172 7365 ion/json;.charse0x0080: 743d 5554 462d 380d 0a43 6f6e 6e65 6374 t=UTF-8..Connect0x0090: 696f 6e3a 206b 6565 702d 616c 6976 650d ion:.keep-alive.0x00a0: 0a63 6f6e 7465 6e74 2d6c 656e 6774 683a .content-length:0x00b0: 2038 320d 0a0d 0a7b 2265 7272 6f72 7322 .82....{"errors"0x00c0: 3a5b 2255 6e61 626c 6520 746f 206c 6f61 :["Unable.to.loa0x00d0: 6420 7265 7175 6573 7465 6420 6669 6c65 d.requested.file0x00e0: 202f 6367 692d 6269 6e2f 636f 696e 5f69 ./cgi-bin/coin_i0x00f0: 6e63 6c75 6465 732f 636f 6e73 7461 6e74 ncludes/constant0x0100: 732e 7068 702e 225d 7d s.php."]}
12:17:59.454755 IP (tos 0x0, ttl 61, id 34633, offset 0, flags [DF], proto TCP (6), length 52)172.16.200.34.36762 > ec-bigdata-flink-worker-040.lansurveyor: Flags [.], cksum 0xd52c (correct), seq 380, ack 200, win 501, options [nop,nop,TS val 285643696 ecr 3208549216], length 00x0000: 0016 3e34 f380 eeff ffff ffff 0800 4500 ..>4..........E.0x0010: 0034 8749 4000 3d06 107b ac10 c822 0a00 .4.I@.=..{..."..0x0020: 27cd 8f9a 10fb 246b 94d5 8d9b 1ad4 8010 '.....$k........0x0030: 01f5 d52c 0000 0101 080a 1106 93b0 bf3e ...,...........>0x0040: 9360 .`
12:17:59.454924 IP (tos 0x0, ttl 61, id 34634, offset 0, flags [DF], proto TCP (6), length 52)172.16.200.34.36762 > ec-bigdata-flink-worker-040.lansurveyor: Flags [R.], cksum 0xd528 (correct), seq 380, ack 200, win 501, options [nop,nop,TS val 285643696 ecr 3208549216], length 00x0000: 0016 3e34 f380 eeff ffff ffff 0800 4500 ..>4..........E.0x0010: 0034 874a 4000 3d06 107a ac10 c822 0a00 .4.J@.=..z..."..0x0020: 27cd 8f9a 10fb 246b 94d5 8d9b 1ad4 8014 '.....$k........0x0030: 01f5 d528 0000 0101 080a 1106 93b0 bf3e ...(...........>0x0040: 9360 .`以上是一个完整的循环，即“握手→互相Push消息→连接重置”(并且对方还试图请求非常敏感的数据，如/etc/passwd)，所以短时间内爆出大量"Connection reset by peer"也就是不足为奇了。后来兜兜转转找到安全部门，得知他们正在进行漏洞扫描，而4347恰好是LAN Surveyor工具默认使用的端口，虚惊一场 = =
为了彻底解决问题，我们可以指定高可用JobManager的端口范围，排除掉大部分的注册端口。在flink-conf.yaml中设置：
high-availability.jobmanager.port: 35000-49150今天帝都持续降雨，天气不好，还是早点回去吧。
民那晚安。