Investigate high CPU usage
Closed, InvalidPublic
Actions

Assigned To

Authored By

	joshuaspence
	May 12 2015, 8:02 AM

Description

It's certainly possible that this is unrelated, but I've been noticing some performance issues on our Phabricator install. Looking at the following graphs from CloudWatch, you can see that the CPU usage seems noticably higher after the 4th of May.

Furthermore, on a couple of occasions I have SSHed into the instance and seen the CPU usage pegged at 100% across all four cores (it is a c3.xlarge). Interestingly, the CPU usage is caused by php-fpm rather than pygmentize. I have added the following script to help me diagnose the issue:

P1780 preamble.php

1	<?php
2
3	if (extension_loaded('setproctitle') && isset($_GET['__path__'])) {
4	setproctitle('php-fpm: '.$_GET['__path__']);
5	}

I haven't found anything yet, but I've been out of office for a while.

Related Objects

Mentioned Here: P1780 preamble.php

Event Timeline

joshuaspence created this task.May 12 2015, 8:02 AM

joshuaspence raised the priority of this task from to Needs Triage.

joshuaspence updated the task description. (Show Details)

joshuaspence added a project: Infrastructure.

joshuaspence added a subscriber: joshuaspence.

joshuaspence claimed this task.May 12 2015, 9:35 AM

In theory, the new "Multimeter" tool may be helpful in understanding this, although we probably need to at least let you put an explicit date window on queries.

I don't know that the requests are slow, just that they are CPU intensive (or maybe there are a lot of them?)

joshuaspence edited projects, added Performance; removed Infrastructure.May 18 2015, 12:01 AM

I took a screencast watching htop over a ~1 minute period (filtered to show only PHP processes). It looks like /conpherence/panel/ and /notification/panel/ are quite CPU intensive.

htop.ogv16 MBDownload

I turned on XHPROF sampling and grabbed some data:

Interestingly, these requests take more than 2 seconds and a significant amount of time is spent in the PhutilBootloader::getLibraryMap() method.

Is /config/cache/ showing significant cache fullness?

michel-slm added a subscriber: michel-slm.Jun 30 2015, 5:12 AM

I haven't really had any issues in a while.

	F465635: cache.png
	Jun 2 2015, 12:02 PM

	F444144: htop.ogv
	May 28 2015, 8:24 AM

Investigate high CPU usageClosed, InvalidPublicActions

Description

Related Objects

Event Timeline

Investigate high CPU usage
Closed, InvalidPublic
Actions