Page MenuHomePhabricator
Create Task
Maniphest T11573

Phabricator goes down with minimal load due to asset delivery issue for some users
Closed, ResolvedPublic
Actions

Description

We're running a Phabricator instance that's been having a hard time staying up. Oddly, some users can load Phabricator just fine (with caching turned off) and some users are unable to load the main page (just hangs like it's loading the site, but it never comes up).

During a consistent load of pinging the healthcheck api, getting the feed query, and loading the website, users are reporting poor loading performance (5-10secs to load the main page) and some users report that the website does not load at all (waiting for assets to load).

We have since turned off our monitoring (/api/conduit.ping) and reduced the feed polling (e.g. /api/feed.query?api.token=<omitted>&limit=100&view=text&before=63....) to once a minute.

Our configuration is running Apache 2.4.18 with Maria DB 10.1.9. Both configurations are fairly vanilla with minimal changes.

Expected Behavior:
Phabricator is able to handle a minimal load (~10 requests/min for all users during peak time).

Actual Behavior:
Website becomes unresponsive inconsistently. Occasionally, all users experience an outage. Part of the time, only some users experience an outage. For example, one user has no problem using Phabricator. They log off and another user logs on the same machine and is unable to load Phabricator due to assets not loading.

Workaround:
kill all apache processes and restart server (graceful restart does not help) when server becomes unresponsive.

Reproduction Steps:
To reproduce, I used a library called locust.py to generate traffic. When I use this script to load the main page, I can easily take down Phabricator within about 5 minutes. If I just hit api endpoints mentioned earlier, Phabricator does not go down. If I load webpages with assets (no caching), Phabricator goes down.

Version
phabricator be84c051e5ba05296be4070bdfbf6f1fdd44355b (Fri, Aug 26)

Troubleshooting steps taken

Looked at Apache server status which revealed nearly all processes stuck in "W" status. This snapshot below was taken during normal operation and not during a load test.
Debug Apache logs were turned on, but does not show errors. The requests for assets come in, but they time out and are not recorded in the access logs.
The database has several hundred processes open which seems to be normal, but they are idle ('Sleep' status).
The server this is running on is a Mac Pro with 64GB of ram and a 2TB SSD so resources shouldn't be an issue.

Apache Server Status

Server Version: Apache/2.4.18 (Unix) PHP/5.6.24 LibreSSL/2.2.7
Server MPM: prefork
Server Built: Feb 20 2016 20:03:19
Parent Server Config. Generation: 1
Parent Server MPM Generation: 0
Server uptime: 1 hour 18 minutes 41 seconds
Server load: 1.61 1.92 1.88
Total accesses: 1123 - Total Traffic: 14.0 MB
CPU Usage: u108.07 s13.08 cu1.04 cs.49 - 2.6% CPU load
.238 requests/sec - 3110 B/second - 12.8 kB/request
125 requests currently being processed, 1 idle workers
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW_W..
................................................................
..........................................................
Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process
Srv	PID	Acc	M	CPU	SS	Req	Conn	Child	Slot	Client	VHost	Request
0-0	59015	0/31/31	W	3.32	4472	0	0.0	0.68	0.68	<omitted>	<omitted>	GET /res/phabricator/7148ae25/rsrc/css/phui/phui-curtain-view.c
1-0	59026	0/29/29	W	2.58	4472	0	0.0	0.32	0.32	<omitted>	<omitted>	GET /res/phabricator/027ba77e/rsrc/css/application/base/phui-th
2-0	59127	0/0/15	W	1.40	4470	0	0.0	0.00	0.31	<omitted>	<omitted>	GET /res/phabricator/36fb3a06/rsrc/css/phui/phui-two-column-vie
3-0	59128	0/0/14	W	1.58	4470	0	0.0	0.00	0.18	<omitted>	<omitted>	GET /res/phabricator/e5ef807d/differential.pkg.css HTTP/1.1
4-0	59088	0/5/5	W	0.58	4472	0	0.0	0.06	0.06	<omitted>	<omitted>	GET /res/phabricator/3010e992/rsrc/externals/javelin/core/init.
5-0	59126	0/0/0	W	0.00	4471	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/9196fb06/rsrc/js/phuix/PHUIXAutocomplete.j
6-0	59129	0/104/104	W	11.33	3485	0	0.0	0.87	0.87	<omitted>	<omitted>	GET /res/phabricator/6714ad26/core.pkg.css HTTP/1.1
7-0	59130	0/105/105	W	11.34	3485	0	0.0	1.02	1.02	<omitted>	<omitted>	GET /res/phabricator/591078d8/rsrc/css/phui/phui-fontkit.css HT
8-0	59131	0/100/100	W	11.13	3548	0	0.0	1.65	1.65	<omitted>	<omitted>	GET /res/phabricator/7148ae25/rsrc/css/phui/phui-curtain-view.c
9-0	59132	0/9/9	W	0.81	4359	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/defaultX/phabricator/ea681761/rsrc/js/application/aphl
10-0	61043	0/27/28	W	2.91	3548	0	0.0	0.57	0.57	<omitted>	<omitted>	GET /res/phabricator/9196fb06/rsrc/js/phuix/PHUIXAutocomplete.j
11-0	64639	0/2/3	W	0.13	3450	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/6714ad26/core.pkg.css HTTP/1.1
12-0	59135	0/94/94	W	10.78	3548	0	0.0	1.09	1.09	<omitted>	<omitted>	GET /res/phabricator/027ba77e/rsrc/css/application/base/phui-th
13-0	59136	0/96/96	W	11.05	3548	0	0.0	0.79	0.79	<omitted>	<omitted>	GET /res/phabricator/36fb3a06/rsrc/css/phui/phui-two-column-vie
14-0	59137	0/3/3	W	0.35	4359	0	0.0	0.01	0.01	<omitted>	<omitted>	GET /res/defaultX/phabricator/ea681761/rsrc/js/application/aphl
15-0	59138	0/94/94	W	10.90	3548	0	0.0	1.12	1.12	<omitted>	<omitted>	GET /res/phabricator/e5ef807d/differential.pkg.css HTTP/1.1
16-0	59139	0/92/92	W	10.22	3548	0	0.0	1.04	1.04	<omitted>	<omitted>	GET /res/phabricator/3010e992/rsrc/externals/javelin/core/init.
17-0	59140	0/6/6	W	0.85	4286	0	0.0	0.16	0.16	<omitted>	<omitted>	GET /res/phabricator/ea681761/rsrc/js/application/aphlict/behav
18-0	64621	0/0/0	W	0.00	3462	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/4b7d8f19/differential.pkg.js HTTP/1.1
19-0	64622	0/0/0	W	0.00	3462	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/bff6884b/rsrc/js/phuix/PHUIXIconView.js HT
20-0	64640	0/0/0	W	0.00	3461	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/027ba77e/rsrc/css/application/base/phui-th
21-0	64910	0/0/1	W	0.17	3448	0	0.0	0.00	0.02	<omitted>	<omitted>	GET /res/phabricator/8bdb2835/rsrc/css/font/font-aleo.css HTTP/
22-0	64642	0/0/0	W	0.00	3462	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/93fcdacd/core.pkg.js HTTP/1.1
23-0	64890	0/0/0	W	0.00	3449	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/591078d8/rsrc/css/phui/phui-fontkit.css HT
24-0	64911	0/0/0	W	0.00	3448	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/027ba77e/rsrc/css/application/base/phui-th
25-0	64912	0/0/0	W	0.00	3447	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/3010e992/rsrc/externals/javelin/core/init.
26-0	64913	0/0/0	W	0.00	3447	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/86668b8e/rsrc/js/phui/rdar.js HTTP/1.1
27-0	64914	0/25/25	W	1.31	2530	0	0.0	0.18	0.18	<omitted>	<omitted>	GET /res/defaultX/phabricator/8bdb2835/rsrc/css/font/font-aleo.
28-0	64915	0/8/8	W	0.80	3211	0	0.0	0.17	0.17	<omitted>	<omitted>	GET /res/defaultX/phabricator/8bdb2835/rsrc/css/font/font-aleo.
29-0	72319	0/0/13	W	1.28	2465	0	0.0	0.00	0.16	<omitted>	<omitted>	GET /res/phabricator/2bfa2836/rsrc/js/phui/behavior-phui-object
30-0	66750	0/9/9	W	0.63	2646	0	0.0	0.07	0.07	<omitted>	<omitted>	GET /res/defaultX/phabricator/8bdb2835/rsrc/css/font/font-aleo.
31-0	66751	0/14/14	W	1.00	2466	0	0.0	0.21	0.21	<omitted>	<omitted>	GET /res/phabricator/027ba77e/rsrc/css/application/base/phui-th
32-0	72085	0/3/3	W	0.69	2466	0	0.0	0.08	0.08	<omitted>	<omitted>	GET /res/phabricator/93fcdacd/core.pkg.js HTTP/1.1
33-0	72303	0/0/0	W	0.00	2466	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/bff6884b/rsrc/js/phuix/PHUIXIconView.js HT
34-0	72677	0/1/1	W	0.08	2422	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/93fcdacd/core.pkg.js HTTP/1.1
35-0	72678	0/0/0	W	0.00	2421	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/bff6884b/rsrc/js/phuix/PHUIXIconView.js HT
36-0	72679	0/0/7	W	0.54	2421	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/2bfa2836/rsrc/js/phui/behavior-phui-object
37-0	73304	0/1/2	W	0.09	2369	0	0.0	0.00	0.06	<omitted>	<omitted>	GET /res/phabricator/93fcdacd/core.pkg.js HTTP/1.1
38-0	72326	0/1/1	W	0.08	2423	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/027ba77e/rsrc/css/application/base/phui-th
39-0	73315	0/1/8	W	0.27	2368	0	0.0	0.02	0.02	<omitted>	<omitted>	GET /res/phabricator/027ba77e/rsrc/css/application/base/phui-th
40-0	72683	0/2/2	W	0.18	2370	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/027ba77e/rsrc/css/application/base/phui-th
41-0	73357	0/0/2	W	0.23	2366	0	0.0	0.00	0.08	<omitted>	<omitted>	GET /res/phabricator/93fcdacd/core.pkg.js HTTP/1.1
42-0	73334	0/0/0	W	0.00	2367	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/93fcdacd/core.pkg.js HTTP/1.1
43-0	73335	0/0/0	W	0.00	2367	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/bff6884b/rsrc/js/phuix/PHUIXIconView.js HT
44-0	73739	0/2/2	W	0.41	2317	0	0.0	0.10	0.10	<omitted>	<omitted>	GET /res/phabricator/91c5d3a6/diffusion.pkg.css HTTP/1.1
45-0	73337	0/0/0	W	0.00	2367	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/027ba77e/rsrc/css/application/base/phui-th
46-0	73358	0/0/0	W	0.00	2366	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/bff6884b/rsrc/js/phuix/PHUIXIconView.js HT
47-0	73833	0/0/0	W	0.00	2316	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/3a9a8bfa/diffusion.pkg.js HTTP/1.1
48-0	74471	0/0/2	W	0.19	2236	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/bff6884b/rsrc/js/phuix/PHUIXIconView.js HT
49-0	73844	0/3/3	W	0.33	2237	0	0.0	0.02	0.02	<omitted>	<omitted>	GET /res/phabricator/027ba77e/rsrc/css/application/base/phui-th
50-0	74453	0/1/1	W	0.08	2237	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/93fcdacd/core.pkg.js HTTP/1.1
51-0	73363	0/12/12	W	1.49	2331	0	0.0	0.37	0.37	<omitted>	<omitted>	GET /res/phabricator/3a9a8bfa/diffusion.pkg.js HTTP/1.1
52-0	73364	0/6/6	W	2.47	2331	0	0.0	0.32	0.32	<omitted>	<omitted>	GET /res/phabricator/91c5d3a6/diffusion.pkg.css HTTP/1.1
53-0	75262	0/3/4	W	0.22	2116	0	0.0	0.02	0.02	<omitted>	<omitted>	GET /res/defaultX/phabricator/8bdb2835/rsrc/css/font/font-aleo.
54-0	74493	0/0/0	W	0.00	2225	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/93fcdacd/core.pkg.js HTTP/1.1
55-0	74494	0/0/0	W	0.00	2225	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/027ba77e/rsrc/css/application/base/phui-th
56-0	74495	0/1/1	W	0.18	2225	0	0.0	0.02	0.02	<omitted>	<omitted>	GET /res/phabricator/bff6884b/rsrc/js/phuix/PHUIXIconView.js HT
57-0	75428	0/6/6	W	0.50	1732	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/e5ef807d/differential.pkg.css HTTP/1.1
58-0	74586	0/1/1	W	0.09	2131	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/defaultX/phabricator/8bdb2835/rsrc/css/font/font-aleo.
59-0	75429	0/1/1	W	0.33	2096	0	0.0	0.10	0.10	<omitted>	<omitted>	GET /res/phabricator/3a9a8bfa/diffusion.pkg.js HTTP/1.1
60-0	75430	0/0/0	W	0.00	2096	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/91c5d3a6/diffusion.pkg.css HTTP/1.1
61-0	75597	0/9/9	W	0.85	1732	0	0.0	0.10	0.10	<omitted>	<omitted>	GET /res/phabricator/36fb3a06/rsrc/css/phui/phui-two-column-vie
62-0	76781	0/5/5	W	0.47	1732	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/6714ad26/core.pkg.css HTTP/1.1
63-0	78329	0/0/0	W	0.00	1731	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/7148ae25/rsrc/css/phui/phui-curtain-view.c
64-0	78331	0/1/1	W	0.08	1730	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/591078d8/rsrc/css/phui/phui-fontkit.css HT
65-0	78766	0/0/0	W	0.00	1679	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/7148ae25/rsrc/css/phui/phui-curtain-view.c
66-0	78342	0/0/0	W	0.00	1729	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/8bdb2835/rsrc/css/font/font-aleo.css HTTP/
67-0	78343	0/1/1	W	0.10	1679	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/e5ef807d/differential.pkg.css HTTP/1.1
68-0	78344	0/1/1	W	0.10	1679	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/6714ad26/core.pkg.css HTTP/1.1
69-0	78345	0/1/1	W	0.25	1679	0	0.0	0.08	0.08	<omitted>	<omitted>	GET /res/phabricator/36fb3a06/rsrc/css/phui/phui-two-column-vie
70-0	81205	0/0/0	W	0.00	1356	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/7148ae25/rsrc/css/phui/phui-curtain-view.c
71-0	78803	0/0/0	W	0.00	1678	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/591078d8/rsrc/css/phui/phui-fontkit.css HT
72-0	78816	0/0/0	W	0.00	1677	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/8bdb2835/rsrc/css/font/font-aleo.css HTTP/
73-0	78817	0/5/5	W	0.31	1357	0	0.0	0.05	0.05	<omitted>	<omitted>	GET /res/phabricator/e5ef807d/differential.pkg.css HTTP/1.1
74-0	78818	0/4/4	W	0.49	1357	0	0.0	0.08	0.08	<omitted>	<omitted>	GET /res/phabricator/36fb3a06/rsrc/css/phui/phui-two-column-vie
75-0	78819	0/5/5	W	0.44	1357	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/6714ad26/core.pkg.css HTTP/1.1
76-0	81215	0/1/1	W	0.09	1355	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/591078d8/rsrc/css/phui/phui-fontkit.css HT
77-0	83518	0/5/5	W	0.67	918	0	0.0	0.25	0.25	<omitted>	<omitted>	GET /res/phabricator/36fb3a06/rsrc/css/phui/phui-two-column-vie
78-0	81227	0/0/0	W	0.00	1354	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/8bdb2835/rsrc/css/font/font-aleo.css HTTP/
79-0	81228	0/18/18	W	1.82	918	0	0.0	0.04	0.04	<omitted>	<omitted>	GET /res/phabricator/6714ad26/core.pkg.css HTTP/1.1
80-0	81229	0/18/18	W	2.15	918	0	0.0	0.22	0.22	<omitted>	<omitted>	GET /res/phabricator/e5ef807d/differential.pkg.css HTTP/1.1
81-0	81230	0/18/18	W	2.12	918	0	0.0	0.21	0.21	<omitted>	<omitted>	GET /res/phabricator/7148ae25/rsrc/css/phui/phui-curtain-view.c
82-0	85119	0/1/1	W	0.23	898	0	0.0	0.02	0.02	<omitted>	<omitted>	GET /res/phabricator/6714ad26/core.pkg.css HTTP/1.1
83-0	85254	0/12/12	W	0.96	522	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/6714ad26/core.pkg.css HTTP/1.1
84-0	88705	0/0/10	W	0.86	520	0	0.0	0.00	0.02	<omitted>	<omitted>	GET /res/phabricator/8bdb2835/rsrc/css/font/font-aleo.css HTTP/
85-0	88677	0/0/0	W	0.00	521	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/591078d8/rsrc/css/phui/phui-fontkit.css HT
86-0	88706	0/0/0	W	0.00	520	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/027ba77e/rsrc/css/application/base/phui-th
87-0	88749	0/0/0	W	0.00	519	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/3010e992/rsrc/externals/javelin/core/init.
88-0	88750	0/0/0	W	0.00	519	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/86668b8e/rsrc/js/phui/rdar.js HTTP/1.1
89-0	88751	0/1/1	W	0.09	504	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/6714ad26/core.pkg.css HTTP/1.1
90-0	88899	0/0/1	W	0.17	502	0	0.0	0.00	0.02	<omitted>	<omitted>	GET /res/phabricator/3010e992/rsrc/externals/javelin/core/init.
91-0	88841	0/0/0	W	0.00	504	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/591078d8/rsrc/css/phui/phui-fontkit.css HT
92-0	88867	0/0/0	W	0.00	503	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/8bdb2835/rsrc/css/font/font-aleo.css HTTP/
93-0	88868	0/0/0	W	0.00	503	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/027ba77e/rsrc/css/application/base/phui-th
94-0	88900	0/0/0	W	0.00	502	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/86668b8e/rsrc/js/phui/rdar.js HTTP/1.1
95-0	89737	0/0/1	W	0.18	480	0	0.0	0.00	0.02	<omitted>	<omitted>	GET /res/phabricator/8bdb2835/rsrc/css/font/font-aleo.css HTTP/
96-0	88902	0/0/0	W	0.00	483	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/6714ad26/core.pkg.css HTTP/1.1
97-0	89705	0/0/0	W	0.00	481	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/591078d8/rsrc/css/phui/phui-fontkit.css HT
98-0	89738	0/0/0	W	0.00	480	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/027ba77e/rsrc/css/application/base/phui-th
99-0	89749	0/0/0	W	0.00	479	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/3010e992/rsrc/externals/javelin/core/init.
100-0	89750	0/0/0	W	0.00	479	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/86668b8e/rsrc/js/phui/rdar.js HTTP/1.1
101-0	89751	0/2/2	W	0.17	455	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/6714ad26/core.pkg.css HTTP/1.1
102-0	90250	0/0/2	W	0.23	453	0	0.0	0.00	0.02	<omitted>	<omitted>	GET /res/phabricator/8bdb2835/rsrc/css/font/font-aleo.css HTTP/
103-0	90243	0/0/0	W	0.00	455	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/591078d8/rsrc/css/phui/phui-fontkit.css HT
104-0	90251	0/0/0	W	0.00	453	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/027ba77e/rsrc/css/application/base/phui-th
105-0	90270	0/0/0	W	0.00	452	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/3010e992/rsrc/externals/javelin/core/init.
106-0	90271	0/0/0	W	0.00	452	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/86668b8e/rsrc/js/phui/rdar.js HTTP/1.1
107-0	90272	0/1/1	W	0.09	437	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/6714ad26/core.pkg.css HTTP/1.1
108-0	90755	0/0/1	W	0.18	435	0	0.0	0.00	0.02	<omitted>	<omitted>	GET /res/phabricator/8bdb2835/rsrc/css/font/font-aleo.css HTTP/
109-0	90719	0/0/0	W	0.00	437	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/591078d8/rsrc/css/phui/phui-fontkit.css HT
110-0	90756	0/0/0	W	0.00	435	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/027ba77e/rsrc/css/application/base/phui-th
111-0	90770	0/9/9	W	1.55	318	0	0.0	0.21	0.21	<omitted>	<omitted>	GET /res/phabricator/91c5d3a6/diffusion.pkg.css HTTP/1.1
112-0	90771	0/0/0	W	0.00	411	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/2bfa2836/rsrc/js/phui/behavior-phui-object
113-0	90772	0/1/1	W	0.14	392	0	0.0	0.02	0.02	<omitted>	<omitted>	GET /res/defaultX/phabricator/8bdb2835/rsrc/css/font/font-aleo.
114-0	90773	0/0/0	W	0.00	411	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/7bc725c4/rsrc/css/application/conduit/cond
115-0	91565	0/5/5	W	0.44	336	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/91c5d3a6/diffusion.pkg.css HTTP/1.1
116-0	91914	0/2/2	W	0.15	336	0	0.0	0.05	0.05	<omitted>	<omitted>	GET /res/phabricator/bdaf4d04/rsrc/js/application/diffusion/beh
117-0	92216	0/0/0	W	0.00	318	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/bdaf4d04/rsrc/js/application/diffusion/beh
118-0	92334	0/0/2	W	0.27	307	0	0.0	0.00	0.02	<omitted>	<omitted>	GET /res/phabricator/3010e992/rsrc/externals/javelin/core/init.
119-0	92236	0/1/1	W	0.00	308	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/591078d8/rsrc/css/phui/phui-fontkit.css HT
120-0	92313	0/0/0	W	0.00	309	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/6714ad26/core.pkg.css HTTP/1.1
121-0	92331	0/0/0	W	0.00	308	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/8bdb2835/rsrc/css/font/font-aleo.css HTTP/
122-0	92332	0/0/0	W	0.00	308	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/027ba77e/rsrc/css/application/base/phui-th
123-0	92335	0/0/0	W	0.00	307	0	0.0	0.00	0.00	<omitted>	<omitted>	GET /res/phabricator/86668b8e/rsrc/js/phui/rdar.js HTTP/1.1
124-0	92336	0/9/9	_	1.04	13	44	0.0	0.32	0.32	<omitted>	<omitted>	POST /login/refresh/ HTTP/1.1
125-0	92337	0/7/7	W	1.09	0	0	0.0	0.32	0.32	<omitted>	<omitted>	GET /server-status HTTP/1.1
Srv	Child Server number - generation
PID	OS process ID
Acc	Number of accesses this connection / this child / this slot
M	Mode of operation
CPU	CPU usage, number of seconds
SS	Seconds since beginning of most recent request
Req	Milliseconds required to process most recent request
Conn	Kilobytes transferred this connection
Child	Megabytes transferred this child
Slot	Total megabytes transferred this slot

Related Objects

Event Timeline

jessjohnson created this task.Sep 1 2016, 6:12 PM
epriestley added a subscriber: epriestley.Sep 1 2016, 6:20 PM

GET /res/phabricator/86668b8e/rsrc/js/phui/rdar.js HTTP/1.1

This isn't a resource we ship with. Are you running any custom code?

epriestley added a comment.Sep 1 2016, 6:23 PM

Also, can you show me what these pages look like?

/multimeter/
/multimeter/?type=1&group=label
/multimeter/?type=0&group=label
jessjohnson added a comment.Sep 1 2016, 6:24 PM

I've added a simple javascript file on the installation I grabbed this report from. As part of the troubleshooting process, I installed a vanilla installation of Phabricator on a separate machine with no modifications (version referenced in the bug report) and was able to reproduce the problem.

epriestley added a comment.Sep 1 2016, 6:29 PM

Are you running APC (Config → Cache Status shows APC as the active "Data Cache")?

If so, can you reproduce it if you disable APC (apc.enabled=0 in php.ini), restart apache, verify that no "Data Cache" is available any more in Config → Cache Status, and then use the takedown script?

jessjohnson added a comment.Sep 1 2016, 6:33 PM

Here's the output. I removed "events" from some of them to improve formatting.

/multimeter/

Samples (Wed, Aug 31, 11:24 AM - Thu, Sep 1, 11:24 AM)	
Samples	
ID	Request	Viewer	Context	Host	Type	Label	Avg	Cost	Rate	Epoch
15	(All)	(All)	(All)	(All)	Web Request	(All)	69,532 us	1,042,976,000 us	-	Wed, Aug 31, 12:50 PM
12839	(All)	(All)	(All)	(All)	Subprocesses	(All)	6,848 us	6,848,000 us	1000	Wed, Aug 31, 2:18 PM
10	(All)	(All)	(All)	(All)	Static Resource	(All)	1 Req	10,000 Req	-	Wed, Aug 31, 5:36 PM

/multimeter/?type=1&group=label

Samples (Wed, Aug 31, 11:26 AM - Thu, Sep 1, 11:26 AM)	
Samples	
ID	Request	Viewer	Context	Host	Type	Label	Avg	Cost	Rate	Epoch
12853	(All)	(All)	(All)	(All)	Web Request	web.PhabricatorHomeMainController	202,943 us	202,943,000 us	1000	Wed, Aug 31, 5:36 PM
12838	(All)	(All)	(All)	(All)	Web Request	web.DifferentialChangesetViewController	150,361 us	150,361,000 us	1000	Wed, Aug 31, 2:18 PM
12836	(All)	(All)	(All)	(All)	Web Request	web.PhabricatorNotificationPanelController	139,054 us	139,054,000 us	1000	Wed, Aug 31, 1:40 PM
3	(All)	(All)	(All)	(All)	Web Request	api.conduit.ping	39,069 us	117,208,000 us	-	Wed, Aug 31, 12:50 PM
3	(All)	(All)	(All)	(All)	Web Request	web.CelerityPhabricatorResourceController	33,335 us	100,005,000 us	-	Wed, Aug 31, 12:52 PM
12856	(All)	(All)	(All)	(All)	Web Request	web.PhabricatorNotificationIndividualController	77,789 us	77,789,000 us	1000	Thu, Sep 1, 10:55 AM
12855	(All)	(All)	(All)	(All)	Web Request	web.DifferentialCommentPreviewController	63,349 us	63,349,000 us	1000	Thu, Sep 1, 10:38 AM
12854	(All)	(All)	(All)	(All)	Web Request	api.diffusion.looksoon	56,272 us	56,272,000 us	1000	Thu, Sep 1, 9:08 AM
12840	(All)	(All)	(All)	(All)	Web Request	api.feed.query	51,612 us	51,612,000 us	1000	Wed, Aug 31, 3:30 PM
12837	(All)	(All)	(All)	(All)	Web Request	api.conduit.connect	49,064 us	49,064,000 us	1000	Wed, Aug 31, 2:10 PM
12857	(All)	(All)	(All)	(All)	Web Request	api.conduit.getcapabilities	35,319 us	35,319,000 us	1000	Thu, Sep 1, 11:08 AM

/multimeter/?type=0&group=label

Samples (Wed, Aug 31, 11:27 AM - Thu, Sep 1, 11:27 AM)	
Samples	
ID	Request	Viewer	Context	Host	Type	Label	Avg	Cost	Rate	Epoch
12852	(All)	(All)	(All)	(All)	Static Resource	rsrc.rsrc/js/phuix/PHUIXIconView.js	1 Req	1,000 Req	1000	Wed, Aug 31, 5:36 PM
12851	(All)	(All)	(All)	(All)	Static Resource	rsrc.differential.pkg.js	1 Req	1,000 Req	1000	Wed, Aug 31, 5:36 PM
12850	(All)	(All)	(All)	(All)	Static Resource	rsrc.core.pkg.js	1 Req	1,000 Req	1000	Wed, Aug 31, 5:36 PM
12848	(All)	(All)	(All)	(All)	Static Resource	rsrc.rsrc/externals/javelin/core/init.js	1 Req	1,000 Req	1000	Wed, Aug 31, 5:36 PM
12847	(All)	(All)	(All)	(All)	Static Resource	rsrc.rsrc/css/application/base/phui-theme.css	1 Req	1,000 Req	1000	Wed, Aug 31, 5:36 PM
12846	(All)	(All)	(All)	(All)	Static Resource	rsrc.rsrc/css/font/font-aleo.css	1 Req	1,000 Req	1000	Wed, Aug 31, 5:36 PM
12845	(All)	(All)	(All)	(All)	Static Resource	rsrc.rsrc/css/phui/phui-fontkit.css	1 Req	1,000 Req	1000	Wed, Aug 31, 5:36 PM
12844	(All)	(All)	(All)	(All)	Static Resource	rsrc.rsrc/css/font/font-lato.css	1 Req	1,000 Req	1000	Wed, Aug 31, 5:36 PM
12843	(All)	(All)	(All)	(All)	Static Resource	rsrc.core.pkg.css	1 Req	1,000 Req	1000	Wed, Aug 31, 5:36 PM
jessjohnson added a comment.Sep 1 2016, 6:36 PM
In T11573#192996, @epriestley wrote:

Are you running APC (Config → Cache Status shows APC as the active "Data Cache")?

If so, can you reproduce it if you disable APC (apc.enabled=0 in php.ini), restart apache, verify that no "Data Cache" is available any more in Config → Cache Status, and then use the takedown script?

Yes, we are running APC.

Cache                 APCu Enabled
Enabled               Version  4.0.11
Memory Usage          0.8% of 63 MB
Cache Entries         17

I'll disable it and run a load test.

epriestley added a comment.Sep 1 2016, 6:40 PM

I don't see anything interesting in Multimeter, it just shows that the host doesn't see much traffic and that we probably need to update some package definitions.

jessjohnson added a comment.Sep 1 2016, 6:59 PM

I've started the load tests without APC. One odd item I didn't notice before that may or may not be related is that Phabricator is having a hard time knowing that daemons are running (e.g. you have 2 unresolved setup issues instead of just 1 because APC is turned off). I checked the daemon logs and see they are flooded with this with one entry occurring a lot. I can't track down the source because I don't know what address it's trying to reach out to (is this a repository pull or a herald rule?). Again, this may be completely unrelated and a different issue all together.

404 Not Found at [<phutil>/src/future/http/BaseHTTPFuture.php:339]
[01-Sep-2016 11:51:09 America/Los_Angeles] arcanist(head=stable, ref.master=9e82ef979e81, ref.stable=10e519475290), phabricator(head=stable, ref.master=86231a9d6db5, ref.stable=be84c051e5ba), phutil(head=stable, ref.master=491ebc74d816, ref.stable=d6818e59c176)
[01-Sep-2016 11:51:09 America/Los_Angeles]   #0 <#2> BaseHTTPFuture::parseRawHTTPResponse(string) called at [<phutil>/src/future/http/HTTPSFuture.php:418]
[01-Sep-2016 11:51:09 America/Los_Angeles]   #1 <#2> HTTPSFuture::isReady() called at [<phutil>/src/future/Future.php:37]
[01-Sep-2016 11:51:09 America/Los_Angeles]   #2 <#2> Future::resolve() called at [<phutil>/src/future/http/BaseHTTPFuture.php:279]
[01-Sep-2016 11:51:09 America/Los_Angeles]   #3 <#2> BaseHTTPFuture::resolvex() called at [<phabricator>/src/applications/feed/worker/FeedPublisherHTTPWorker.php:32]
[01-Sep-2016 11:51:09 America/Los_Angeles]   #4 <#2> FeedPublisherHTTPWorker::doWork() called at [<phabricator>/src/infrastructure/daemon/workers/PhabricatorWorker.php:122]
[01-Sep-2016 11:51:09 America/Los_Angeles]   #5 <#2> PhabricatorWorker::executeTask() called at [<phabricator>/src/infrastructure/daemon/workers/storage/PhabricatorWorkerActiveTask.php:171]
[01-Sep-2016 11:51:09 America/Los_Angeles]   #6 <#2> PhabricatorWorkerActiveTask::executeTask() called at [<phabricator>/src/infrastructure/daemon/workers/PhabricatorTaskmasterDaemon.php:22]
[01-Sep-2016 11:51:09 America/Los_Angeles]   #7 PhabricatorTaskmasterDaemon::run() called at [<phutil>/src/daemon/PhutilDaemon.php:184]
[01-Sep-2016 11:51:09 America/Los_Angeles]   #8 PhutilDaemon::execute() called at [<phutil>/scripts/daemon/exec/exec_daemon.php:127]
[01-Sep-2016 11:51:10 America/Los_Angeles] [2016-09-01 11:51:10] EXCEPTION: (PhutilProxyException) Error while executing Task ID 2726730. {>} (HTTPFutureHTTPResponseStatus) [HTTP/404] Not Found

The task looks like it's an infinitely repeated with 43699 failures so far. Going to /daemon/ shows that there are 6 daemons running, yet Phabricator indicates that I need to start the daemons.

epriestley added a comment.Sep 1 2016, 7:05 PM

That's probably a webhook URI configured in feed.http-hooks (which may no longer be configured).

You can bin/worker cancel --id <id> to cancel the task. We try webhooks (and most other tasks) forever by default, under the assumption that any finite number of failures are caused by temporary problems on the far end and that you'd like to get your callbacks as soon as the service is back up.

epriestley added a comment.Sep 1 2016, 7:08 PM

I'm not immediately sure about the daemon confusion, although I suspect it's not likely to be a hint at the root issue. Caching for setup issues falls back to some pretty whacky shenanigans with APC/APCu disabled.

jessjohnson added a comment.Sep 1 2016, 8:36 PM

I've been running load tests for the past 2-3 hours and for some reason, I can't bring it down anymore. You can probably mark this bug as 'unable to reproduce'. I'll continue looking into the issue and see if there's a definitive way to reproduce.

If I do get into a situation where Phabricator stops responding again, is there any useful troubleshooting tips I could try to help narrow down the root cause while it's in that state?

epriestley added a comment.Sep 1 2016, 8:42 PM

If you turn APCu back on, does it freeze up again?

(I'm suspicious this is a locking bug in APCu.)

jessjohnson added a comment.Sep 1 2016, 9:32 PM

I turned it back on and ran additional load tests but Apache was able to handle everything. To be honest, I was surprised how well the site held up with APCu turned off. The numbers are about the same.

If this comes up again, I'll turn off APCu and try to reproduce the issue. Alternately, I could update APCu and see if that helps.

jessjohnson added a comment.Sep 1 2016, 9:35 PM

Just some quick googling revealed this issue with a prior version of APCu.

https://bugs.launchpad.net/ubuntu/+source/php-apcu/+bug/1422484

The symptoms reported in this bug report are nearly identical to my issue.

epriestley added a comment.Sep 1 2016, 9:41 PM

We don't really use APCu for very much, so if you are able to isolate it as the issue (or are just suspicious, e.g. based on that launchpad bug looking pretty similar) it's probably OK to turn it off. I'd expect you to lose something like 5% performance with it off, not 50%+.

(We primarily use it because it has gets reset when you restart the webserver, which is a desirable property which we can't otherwise easily detect. We also use it as a fast key-value cache, but since we can't dirty it in multi-web-server setups we can only really put a few kinds of mostly-immutable data into it.)

We may eventually ship some other kind of cache with similar characteristics, particularly after figuring out the cache story for PHP7 (T9640), but you're likely OK without APCu until then.

Or this could be totally unrelated. I don't really have any particularly strong reason to suspect APCu, but super weird locks/hangs like this have been APC/APCu more than other things in the past.

epriestley closed this task as Resolved.Sep 2 2016, 11:38 AM
epriestley claimed this task.

I'm going to close this for now since there's no more upstream action available given that the issue either no longer reproduces or disabling APCu appears to fix it and be a reasonable workaround.

If APCu truly is the issue we could try to detect bad APCu and warn about it, but don't have enough data yet (e.g., it's not truly clear that APCu was the issue here, I can't reproduce the issue myself, and we haven't seen other similar reports that point a finger at particular APCu versions or incompatibilities). If we see more reports in the future, we can cross-reference them with this report to try to figure out what we should be looking for and warning about.

Feel free to follow up or file a new issue if anything else comes up.

Phabricator · Built by Phacility