Page MenuHomePhabricator

Attempt to connect to phabricator@p:127.0.0.1 failed with error #1040: Too many connections.
Closed, ResolvedPublic

Description

We run a Phabricator server via Chef, updated from the repo every 30 minutes. Yesterday, around midnight GMT, the server started having trouble due to the server failing to connect to mysql. Any commits around that time that may have caused problems of this sort?

This is still running on an old Ubuntu Precise setup, so very possible that it's just some artifact of our setup, but I though it best to check in just in case.

Event Timeline

When rebooting the sql server, we get an error on:

/usr/sbin/mysqld: File '/opt/phabricator/phabricator/resources/sql/stopwords.txt' not found (Errcode: 13)

Touching that file to recreate it allows phab to start up again, but still with some "too many connections" problems in various parts of the dashboard.

We made one change recently (T11672) but that should cause us to use fewer connections.

What is max_connections set to? You can figure it out like this:

mysql> select @@max_connections;
+-------------------+
| @@max_connections |
+-------------------+
|              1750 |
+-------------------+
1 row in set (0.00 sec)

I can confirm that this bug also happen on a local install with a few users (around 15)

max connections is set to the following

MariaDB [(none)]> select @@max_connections;
+-------------------+
| @@max_connections |
+-------------------+
|               151 |
+-------------------+
1 row in set (0.00 sec)

Does raising max_connections in /etc/my.cnf (or similar) and restarting MySQL resolve this?

(Try raising it to at least 1000.)

max connections was set to 250 (default?). We bumped it up to 1000, which seems to have fixed the issue for now.

We have seen exactly the same error on our instance today, since updating to master. MySQL max_connections was set to the default of 151.

151 seems to be the common default.

After setting max_connections=1500 , the problem still happens some hours after running the instance.

Actually, the problem seems to be in the system configuration. There a hard limit on open file descriptor ( which limits mysqld to 214 connections.

The following guide explains how to fix this: http://blog.endpoint.com/2013/12/increasing-mysql-55-maxconnections-on.html

So the real fix would beto set the following:

[mysqld]
open_files_limit = 8192
max_connections = 1000