`admin.phacility.com` is receiving a huge volume of "leafweb" traffic
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	epriestley
	Oct 5 2017, 9:08 PM

Description

172.30.0.63 - leafweb [05/Oct/2017:21:08:19 +0000] [52.52.33.204] "POST /peernetwork/services/LeafNetsRegWebServiceV2 HTTP/1.1" 500 1624
172.30.0.63 - leafweb [05/Oct/2017:21:08:19 +0000] [52.52.33.204] "POST /peernetwork/services/LeafNetsRegWebServiceV2 HTTP/1.1" 500 1624
172.30.0.63 - leafweb [05/Oct/2017:21:08:19 +0000] [52.52.33.204] "POST /peernetwork/services/LeafNetsRegWebServiceV2 HTTP/1.1" 500 1624
172.30.0.63 - leafweb [05/Oct/2017:21:08:19 +0000] [52.52.33.204] "POST /peernetwork/services/LeafNetsRegWebServiceV2 HTTP/1.1" 500 1624
172.30.0.63 - leafweb [05/Oct/2017:21:08:19 +0000] [52.52.33.204] "POST /peernetwork/services/LeafNetsRegWebServiceV2 HTTP/1.1" 500 1624
172.30.0.63 - leafweb [05/Oct/2017:21:08:19 +0000] [52.52.33.204] "POST /peernetwork/services/LeafNetsRegWebServiceV2 HTTP/1.1" 500 1624
172.30.0.63 - leafweb [05/Oct/2017:21:08:19 +0000] [52.52.33.204] "POST /peernetwork/services/LeafNetsRegWebServiceV2 HTTP/1.1" 500 1624

Like 50 a second or something.

Revisions and Commits

	Restricted Differential Revision	Restricted Diffusion Commit
	Restricted Differential Revision	Restricted Diffusion Commit

Related Objects

Mentioned In: T13199: All "web" host rate limited ELB `/status/` requests simultaneously after 11 months without configuration changes
T12857: Temporary directory fullness can cause daemon issues?

Event Timeline

epriestley created this task.Oct 5 2017, 9:08 PM

Herald added a subscriber: eadler. · View Herald TranscriptOct 5 2017, 9:08 PM

There are a very large number of originating IP addresses so I'm going to filter this traffic by path rather than by IP address.

I early 500'd these requests in preamble.php and load seems better now:

Screen Shot 2017-10-05 at 2.22.06 PM.png (580×920 px, 65 KB)

I'm logging remote IPs so we can ban by IP later if there are merely a large number and not an enormous number, but it looked like there were at least several hundred.

I'll get something a little more durable into rCORE to replace my hacky patch on the live server.

Handful of these, too:

172.30.0.63 - DGND4000_9CD36D0A7C40 [05/Oct/2017:21:25:18 +0000] [52.52.33.204] "POST /peernetwork/services/LeafNetsConfigWebService HTTP/1.1" 500 24
172.30.0.63 - D6300_C404154EA80F [05/Oct/2017:21:25:17 +0000] [52.52.33.204] "POST /peernetwork/services/LeafNetsConfigWebService HTTP/1.1" 500 24

None of this stuff Googles for me.

There's some of this mixed in there too:

172.30.0.63 - - [05/Oct/2017:21:26:48 +0000] [phacility.com] "GET /wp-login.php HTTP/1.1" 302 -

...which is consistent with security scanners probing for Wordpress junk, so maybe this is just a garbage scanner, although they usually originate from some single "security researcher" IP address.

Seems like we're relatively stable now, in any case.

merely a large number and not an enormous number

In the last ~25K requests, we've seen 234 unique addresses and they've each made about 100 requests, so it looks like the attacker/"researcher" doesn't control all that many hosts.

This is still ongoing, although we're weathering it without any issues.

epriestley added a revision: Restricted Differential Revision.Oct 6 2017, 2:01 PM

epriestley added a commit: Restricted Diffusion Commit.Oct 6 2017, 3:50 PM

I've deployed the slightly more formal preamble. It no longer has the remote address logging, but here are the addresses captured up to now (first column: request count; second column: address):

 1311 58.111.114.128
 1536 115.64.14.214
 1702 109.150.121.33
 2235 121.214.27.120
 3145 110.174.82.30
 3676 108.209.47.160
 4020 121.44.199.62
 4117 118.211.31.189
 4344 123.211.66.83
 5294 220.253.121.53
 5998 203.206.181.18
 6181 92.6.209.142
 6288 151.67.211.55
 6610 86.203.36.178
 7263 60.228.234.109
 8250 151.40.175.196
 8634 101.163.8.173
 8688 106.69.70.148
 8953 14.201.51.217
10465 67.249.201.98
12402 206.188.150.224
12666 82.54.119.75
12956 146.241.144.82
13849 220.253.147.172
15482 27.33.88.209
16091 125.168.109.60
17622 27.99.35.240
17942 1.122.121.190
18917 14.203.245.63
19226 86.203.8.34
19960 79.79.149.185
20685 103.217.167.103
20837 220.233.106.53
20858 121.45.65.85
21226 110.147.128.226
21335 116.250.242.135
21387 216.21.207.133
21799 121.44.249.55
23204 118.210.232.221
23277 137.147.155.57
23571 218.214.246.94
23737 169.1.29.91
23770 109.152.224.107
23880 95.90.200.137
24018 105.228.174.160
24047 41.132.189.230
24108 14.200.131.245
24123 121.219.246.95
24172 80.41.147.237
24207 31.49.8.117
24298 2.238.141.251
24491 197.91.179.236
24719 94.29.213.54
24760 87.112.181.38
24907 60.241.157.252
24909 120.151.142.185
24934 51.7.250.205
24951 106.68.24.238
24999 115.187.154.24
25043 58.110.190.5
25071 178.20.16.59
25140 110.175.161.170
25145 58.111.106.161
25162 27.33.101.162
25182 58.7.112.191
25219 60.241.83.61
25222 120.144.17.195
25229 139.216.208.100
25230 218.215.80.33
25241 95.232.205.151
25246 59.102.117.63
25254 110.174.136.111
25275 27.32.205.174
25293 14.201.119.173
25295 14.200.87.108
25311 1.122.167.183
25316 121.217.133.57
25326 121.214.132.125
25327 220.253.242.247
25327 80.104.8.208
25330 110.175.63.252
25343 79.36.189.17
25351 110.175.9.236
25360 114.30.97.247
25362 49.207.59.196
25371 106.68.214.246
25381 118.208.97.206
25389 203.45.227.152
25391 110.174.123.240
25404 139.130.164.98
25405 125.209.188.241
25405 60.241.36.150
25409 60.240.241.220
25411 111.220.173.56
25413 220.240.188.79
25419 81.132.234.204
25421 144.138.90.215
25432 118.208.30.102
25435 123.2.107.186
25435 95.237.69.210
25437 203.220.72.246
25448 220.240.165.90
25452 60.242.208.5
25471 220.245.142.34
25474 138.130.34.251
25482 218.215.101.139
25486 122.148.121.208
25493 118.211.30.179
25500 122.148.91.179
25505 124.170.166.86
25509 87.100.231.141
25512 110.175.9.244
25514 202.161.76.82
25516 220.233.79.196
25521 151.65.188.132
25524 27.32.40.223
25525 58.165.237.118
25527 60.242.166.67
25533 151.40.58.164
25536 58.104.208.147
25538 203.219.244.224
25547 121.221.51.163
25548 120.29.54.148
25552 211.27.71.223
25554 111.220.48.104
25556 141.168.114.229
25556 220.233.183.238
25559 203.45.106.125
25566 87.8.141.38
25580 14.201.3.125
25582 115.70.30.228
25586 58.84.157.31
25589 124.171.236.21
25590 123.100.38.41
25606 58.165.218.50
25606 60.242.153.133
25606 60.242.81.19
25618 95.248.80.50
25622 176.250.220.69
25628 121.214.21.168
25631 155.143.52.197
25634 203.206.122.131
25636 118.210.175.188
25636 203.206.28.207
25638 110.175.224.57
25643 58.96.96.134
25646 124.169.26.125
25652 14.200.86.62
25654 110.21.153.101
25655 87.6.53.66
25656 88.109.192.20
25659 210.50.254.92
25663 123.3.146.60
25667 203.217.83.192
25671 79.21.226.215
25671 93.187.144.16
25673 122.151.163.218
25675 213.45.62.86
25675 87.4.42.179
25681 144.138.250.51
25681 84.13.93.221
25683 82.52.109.51
25684 14.200.191.184
25702 58.165.251.120
25704 106.68.82.95
25706 122.151.222.88
25708 121.44.238.203
25710 122.148.50.114
25713 122.148.217.138
25716 183.81.145.141
25717 121.200.10.110
25724 185.182.199.200
25729 121.222.162.230
25729 91.89.124.179
25730 101.160.22.85
25730 81.204.8.78
25732 122.148.221.151
25732 123.51.44.139
25734 155.143.202.7
25737 79.34.21.27
25740 77.53.69.240
25741 101.163.15.214
25755 137.147.140.68
25758 81.133.6.15
25762 124.149.186.147
25768 123.2.48.68
25770 43.255.45.65
25771 60.241.236.79
25775 110.143.84.243
25787 60.242.143.165
25790 60.241.195.70
25792 58.7.196.35
25798 101.175.40.53
25803 110.175.160.165
25810 93.44.86.29
25811 60.241.209.2
25821 122.108.97.149
25822 46.223.241.63
25837 86.129.51.58
25846 114.76.116.151
25849 114.72.103.220
25850 121.219.13.93
25865 86.175.131.82
25875 121.216.32.132
25877 2.229.135.189
25879 81.132.26.216
25897 115.70.166.97
25898 203.214.73.160
25925 92.25.119.49
25928 89.100.121.119
25944 47.72.164.46
25948 120.146.40.247
25953 81.152.22.12
25961 115.70.153.244
25962 68.209.176.21
25973 78.148.145.220
25985 86.178.71.14
25989 220.233.171.175
25995 2.96.103.185
25997 103.85.38.239
26003 121.75.56.155
26005 202.67.78.91
26011 82.83.195.165
26026 185.175.169.67
26051 121.222.40.72
26094 90.152.126.254
26131 123.203.66.19
26176 121.212.132.199
26340 122.58.135.63
26390 184.39.54.21
26419 71.181.234.6
26529 108.215.207.2
26572 142.0.97.2
26581 98.66.158.4
26697 68.121.18.143
26911 184.1.110.251
27130 71.223.137.59
27295 216.180.183.28
27383 47.41.252.115
52087 83.78.82.208

This is still ongoing but it isn't impacting us so I don't plan to do anything else here.

The traffic is coming directly to one of the external IP addresses for the alb ELB, so we could probably shed it by launching copying alb001 to alb002, rebinding DNS, then releasing alb001 once the DNS change propagated.

epriestley mentioned this in T12857: Temporary directory fullness can cause daemon issues?.Oct 12 2017, 3:37 PM

epriestley added a revision: Restricted Differential Revision.Oct 12 2017, 10:59 PM

epriestley added a commit: Restricted Diffusion Commit.Oct 13 2017, 8:13 PM

This traffic eventually stopped on October 20th.

epriestley mentioned this in T13199: All "web" host rate limited ELB `/status/` requests simultaneously after 11 months without configuration changes.Sep 10 2018, 8:25 PM

	F5212306: Screen Shot 2017-10-05 at 2.22.06 PM.png
	Oct 5 2017, 9:24 PM

`admin.phacility.com` is receiving a huge volume of "leafweb" trafficClosed, ResolvedPublicActions

Description

Revisions and Commits

Related Objects

Event Timeline

`admin.phacility.com` is receiving a huge volume of "leafweb" traffic
Closed, ResolvedPublic
Actions