I'm currently trying to troubleshoot some performance issues that we are experiencing with our Phabricator setup. I have a few things that I want to check, but I would appreciate any tips on getting to the bottom of this.
We recently moved our Phabricator install onto a decidated host. We now have two hosts dedicated to Phabricator, one running daemons and the other running the application itself (plus aphlict). Our install roughly consists of the following:
- Ubuntu 12.04
- PHP v5.3
- `php-fpm` running in dynamic mode.
- Aphlict is running on NodeJS v0.10.
- Syntax highlighting with `pygmentize` v0.6.
- Both hosts are running on c1.xl EC2 instances.
Our new host has been up for 8 days now and it seemed to be really struggling today. I noticed that all 8 cores were running at basically 100% and there were occasional 502s occurring. At the time, we had around 200 concurrent users (according to `/notificaton/status/`.
At one stage, the load from `php-fpm` dropped down momentarily, but there was two `pygmentize` processes running at 100%.
I need to do some more investigating, but I'm suspecting one of the following issues:
- `pygmentize` is being called excessively and on large files (is there any cache in front of syntax highlighting?)
- Aphlict leaking resources perhaps (we have had issues with NodeJS previously)
- Misconfiguration of PHP/nginx.