Page MenuHomePhabricator
Create Task
Maniphest T11011

Can't connect to MySQL server
Open, Needs TriagePublic
Actions

Description

I just updated our Phabricator install today to rP3d3fff4991ab921853fd0d1fc96f8b1e65b962d7. Previously we were probably somewhere around rP45718268a96ef8290ec1e5bed0ab5b978ca9f1a2. Today we have had two occasions in which users have received "Can't connect to MySQL server" error messages. The first time that it happened I figured that it was some AWS issue, but the second time makes me suspect that there are performance issues with the latest update.

The following screenshot shows the CloudWatch metrics for our RDS database over the past two weeks:

cloudwatch.png (437×1 px, 137 KB)

None of the metrics shown in the above screenshot seem worrying to me, but I did notice that memory usage was quite high around the time of the second downtime. I'm not sure if I am onto the right track, but I noticed that some queries on the home page are doing full table scans... I will post some screenshots from DarkConsole in a few minutes.

Related Objects

Event Timeline

joshuaspence created this task.May 23 2016, 5:27 AM
joshuaspence added a comment.May 23 2016, 5:31 AM

darkconsole.png (4×1 px, 2 MB)

The following queries are full table scans:

  • SELECT commit`.* FROM repository_commit commit LEFT JOIN repository_auditrequest status ON commit.phid = status.commitPHID WHERE (commit.authorPHID IN ('PHID-USER-abzwhc667osh5mwd4gk7')) AND (status.auditStatus = 'concerned') GROUP BY commit.phid ORDER BY commit.id DESC LIMIT 100`
  • SELECT commit`.* FROM repository_commit commit LEFT JOIN repository_auditrequest status ON commit.phid = status.commitPHID JOIN repository_auditrequest needs ON commit.phid = needs.commitPHID AND needs.auditStatus IN ('requested', 'audit-required') WHERE (needs.auditorPHID IN ('PHID-USER-abzwhc667osh5mwd4gk7', 'PHID-OPKG-3s3imoazwnidjqv2gqla', 'PHID-OPKG-eij6lsth6ztqbkbqmokp', 'PHID-OPKG-22mrmc67kvih7syz7y23', 'PHID-PROJ-z4lum22ekior7wxtn7ub', 'PHID-PROJ-uixvrftqv4cgqutudbbs', 'PHID-PROJ-5mjg4fuuo3uordsz7br3', 'PHID-PROJ-cfebmxv6jagr5lxsazmr', 'PHID-PROJ-j5rqbo274ix26g7nm4fy', 'PHID-PROJ-ucboqvdt2w2pwd5plogu', 'PHID-PROJ-cbz6xwpsmzwa4oqodzqg', 'PHID-PROJ-mvlbajc23cfsdvszg4rx', 'PHID-PROJ-iapge4dqkstwge7gsju7', 'PHID-PROJ-2gbj5ztrrj22gglwd44w', 'PHID-PROJ-54gko6yxulpfmpco5rdy', 'PHID-PROJ-kaskqzsc6cvt7m53muul', 'PHID-PROJ-5ts4nmbgwdhofbmt3njy', 'PHID-PROJ-h7aqgwfkq32anogk3aw3', 'PHID-PROJ-kdaiyxcy5d34kkbfvmuv', 'PHID-PROJ-hpwfysxuxvewi7pbbjo3', 'PHID-PROJ-h7ek3xueii23nmr4adlj', 'PHID-PROJ-7r7hdzn46vt722ej37q5', 'PHID-PROJ-e4vyvdjt3viwdfemffmu', 'PHID-PROJ-pakyjeg6msgvy74j5ako', 'PHID-PROJ-nbtcpwy6jyqurn4donug', 'PHID-PROJ-fpucoegg437ttgxyfciu', 'PHID-PROJ-ldc27jcuq4z4oihiyt37', 'PHID-PROJ-xqtz2uzzxkw7hekywkyu', 'PHID-PROJ-ytmn33u2eoqzvqrrkx4p', 'PHID-PROJ-3ecog2unqfikccogkaeb', 'PHID-PROJ-mbjb3j3s26b4cw4e7yy7', 'PHID-PROJ-7rb6kfxg3ig2g43tg3rx', 'PHID-PROJ-ziza6s3hkygbe2u5qfvl')) AND (status.auditStatus in ('audit-required', 'requested', 'concerned')) GROUP BY commit.phid ORDER BY commit.id DESC LIMIT 100`

This query is "whole table" but I'm not quite sure what that is:

  • SELECT p`.* FROM owners_package p JOIN owners_owner o ON o.packageID = p.id WHERE (o.userPHID IN ('PHID-USER-abzwhc667osh5mwd4gk7', 'PHID-PROJ-z4lum22ekior7wxtn7ub', 'PHID-PROJ-uixvrftqv4cgqutudbbs', 'PHID-PROJ-5mjg4fuuo3uordsz7br3', 'PHID-PROJ-cfebmxv6jagr5lxsazmr', 'PHID-PROJ-j5rqbo274ix26g7nm4fy', 'PHID-PROJ-ucboqvdt2w2pwd5plogu', 'PHID-PROJ-cbz6xwpsmzwa4oqodzqg', 'PHID-PROJ-mvlbajc23cfsdvszg4rx', 'PHID-PROJ-iapge4dqkstwge7gsju7', 'PHID-PROJ-2gbj5ztrrj22gglwd44w', 'PHID-PROJ-54gko6yxulpfmpco5rdy', 'PHID-PROJ-kaskqzsc6cvt7m53muul', 'PHID-PROJ-5ts4nmbgwdhofbmt3njy', 'PHID-PROJ-h7aqgwfkq32anogk3aw3', 'PHID-PROJ-kdaiyxcy5d34kkbfvmuv', 'PHID-PROJ-hpwfysxuxvewi7pbbjo3', 'PHID-PROJ-h7ek3xueii23nmr4adlj', 'PHID-PROJ-7r7hdzn46vt722ej37q5', 'PHID-PROJ-e4vyvdjt3viwdfemffmu', 'PHID-PROJ-pakyjeg6msgvy74j5ako', 'PHID-PROJ-nbtcpwy6jyqurn4donug', 'PHID-PROJ-fpucoegg437ttgxyfciu', 'PHID-PROJ-ldc27jcuq4z4oihiyt37', 'PHID-PROJ-xqtz2uzzxkw7hekywkyu', 'PHID-PROJ-ytmn33u2eoqzvqrrkx4p', 'PHID-PROJ-3ecog2unqfikccogkaeb', 'PHID-PROJ-mbjb3j3s26b4cw4e7yy7', 'PHID-PROJ-7rb6kfxg3ig2g43tg3rx', 'PHID-PROJ-ziza6s3hkygbe2u5qfvl')) GROUP BY p.phid ORDER BY p.id DESC`
Phabricator · Built by Phacility