Database connection lost while upgrading Phabricator
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	adityar7
	May 17 2014, 9:02 AM

Description

I was upgrading Phabricator using the update_phabricator.sh script and this happened:

Applying patch 'phabricator:20140323.harbor.2.message.sql'...
Applying patch 'phabricator:20140325.push.1.event.sql'...
Applying patch 'phabricator:20140325.push.2.eventphid.sql'...
Applying patch 'phabricator:20140325.push.3.groups.php'...
Adding transaction log event groups...
Done.
Applying patch 'phabricator:20140325.push.4.prune.sql'...
[2014-05-17 08:48:06] PHLOG: Retrying (2) after AphrontQueryConnectionException: Attempt to connect to root@rdsmachine.com failed with error #2003: Can't connect to MySQL server on 'rdsmachine.com' (111). at [/home/ubuntu/phabricator/libphutil/src/aphront/storage/connection/mysql/AphrontMySQLDatabaseConnectionBase.php:107]
[2014-05-17 08:48:06] PHLOG: Retrying (1) after AphrontQueryConnectionException: Attempt to connect to root@rdsmachine.com failed with error #2003: Can't connect to MySQL server on 'rdsmachine.com' (111). at [/home/ubuntu/phabricator/libphutil/src/aphront/storage/connection/mysql/AphrontMySQLDatabaseConnectionBase.php:107]
[2014-05-17 08:48:06] EXCEPTION: (AphrontQueryConnectionException) Attempt to connect to root@rdsmachine.com failed with error #2003: Can't connect to MySQL server on 'rdsmachine.com' (111). at [/home/ubuntu/phabricator/libphutil/src/aphront/storage/connection/mysql/AphrontMySQLiDatabaseConnection.php:67]
  #0 AphrontMySQLiDatabaseConnection::connect() called at [/home/ubuntu/phabricator/libphutil/src/aphront/storage/connection/mysql/AphrontMySQLDatabaseConnectionBase.php:100]
  #1 AphrontMySQLDatabaseConnectionBase::establishConnection() called at [/home/ubuntu/phabricator/libphutil/src/aphront/storage/connection/mysql/AphrontMySQLDatabaseConnectionBase.php:123]
  #2 AphrontMySQLDatabaseConnectionBase::requireConnection() called at [/home/ubuntu/phabricator/libphutil/src/aphront/storage/connection/mysql/AphrontMySQLDatabaseConnectionBase.php:159]
  #3 AphrontMySQLDatabaseConnectionBase::executeRawQuery(ALTER TABLE phabricator_repository.repository_pushlog
  DROP remoteAddress) called at [/home/ubuntu/phabricator/libphutil/src/xsprintf/queryfx.php:9]
  #4 queryfx(Object AphrontMySQLiDatabaseConnection, %Q, ALTER TABLE phabricator_repository.repository_pushlog
  DROP remoteAddress) called at [/home/ubuntu/phabricator/phabricator/src/infrastructure/storage/management/PhabricatorStorageManagementAPI.php:190]
  #5 PhabricatorStorageManagementAPI::applyPatchSQL(/home/ubuntu/phabricator/phabricator/resources/sql/autopatches/20140325.push.4.prune.sql) called at [/home/ubuntu/phabricator/phabricator/src/infrastructure/storage/management/PhabricatorStorageManagementAPI.php:168]
  #6 PhabricatorStorageManagementAPI::applyPatch(Object PhabricatorStoragePatch) called at [/home/ubuntu/phabricator/phabricator/src/infrastructure/storage/management/workflow/PhabricatorStorageManagementUpgradeWorkflow.php:168]
  #7 PhabricatorStorageManagementUpgradeWorkflow::execute(Object PhutilArgumentParser) called at [/home/ubuntu/phabricator/libphutil/src/parser/argument/PhutilArgumentParser.php:396]
  #8 PhutilArgumentParser::parseWorkflowsFull(Array of size 7 starting with: { PhabricatorStorageManagementDatabasesWorkflow => Object PhabricatorStorageManagementDatabasesWorkflow }) called at [/home/ubuntu/phabricator/libphutil/src/parser/argument/PhutilArgumentParser.php:292]
  #9 PhutilArgumentParser::parseWorkflows(Array of size 7 starting with: { PhabricatorStorageManagementDatabasesWorkflow => Object PhabricatorStorageManagementDatabasesWorkflow }) called at [/home/ubuntu/phabricator/phabricator/scripts/sql/manage_storage.php:115]

The server is EC2 and the database is an RDS MySQL instance. It may have been a transient connection error.

Running the update script again gives:

Applying patch 'phabricator:20140325.push.4.prune.sql'...
[2014-05-17 08:56:32] EXCEPTION: (AphrontQueryException) #1050: Table 'phabricator_repository/#sql-ib414' already exists at [/home/ubuntu/phabricator/libphutil/src/aphront/storage/connection/mysql/AphrontMySQLDatabaseConnectionBase.php:311]
  #0 AphrontMySQLDatabaseConnectionBase::throwQueryCodeException(1050, Table 'phabricator_repository/#sql-ib414' already exists) called at [/home/ubuntu/phabricator/libphutil/src/aphront/storage/connection/mysql/AphrontMySQLDatabaseConnectionBase.php:278]
  #1 AphrontMySQLDatabaseConnectionBase::throwQueryException(Object mysqli) called at [/home/ubuntu/phabricator/libphutil/src/aphront/storage/connection/mysql/AphrontMySQLDatabaseConnectionBase.php:184]
  #2 AphrontMySQLDatabaseConnectionBase::executeRawQuery(ALTER TABLE phabricator_repository.repository_pushlog
  DROP remoteAddress) called at [/home/ubuntu/phabricator/libphutil/src/xsprintf/queryfx.php:9]
  #3 queryfx(Object AphrontMySQLiDatabaseConnection, %Q, ALTER TABLE phabricator_repository.repository_pushlog
  DROP remoteAddress) called at [/home/ubuntu/phabricator/phabricator/src/infrastructure/storage/management/PhabricatorStorageManagementAPI.php:190]
  #4 PhabricatorStorageManagementAPI::applyPatchSQL(/home/ubuntu/phabricator/phabricator/resources/sql/autopatches/20140325.push.4.prune.sql) called at [/home/ubuntu/phabricator/phabricator/src/infrastructure/storage/management/PhabricatorStorageManagementAPI.php:168]
  #5 PhabricatorStorageManagementAPI::applyPatch(Object PhabricatorStoragePatch) called at [/home/ubuntu/phabricator/phabricator/src/infrastructure/storage/management/workflow/PhabricatorStorageManagementUpgradeWorkflow.php:168]
  #6 PhabricatorStorageManagementUpgradeWorkflow::execute(Object PhutilArgumentParser) called at [/home/ubuntu/phabricator/libphutil/src/parser/argument/PhutilArgumentParser.php:396]
  #7 PhutilArgumentParser::parseWorkflowsFull(Array of size 7 starting with: { PhabricatorStorageManagementDatabasesWorkflow => Object PhabricatorStorageManagementDatabasesWorkflow }) called at [/home/ubuntu/phabricator/libphutil/src/parser/argument/PhutilArgumentParser.php:292]
  #8 PhutilArgumentParser::parseWorkflows(Array of size 7 starting with: { PhabricatorStorageManagementDatabasesWorkflow => Object PhabricatorStorageManagementDatabasesWorkflow }) called at [/home/ubuntu/phabricator/phabricator/scripts/sql/manage_storage.php:115]

How might I recover from this?

Event Timeline

adityar7 created this task.May 17 2014, 9:02 AM

adityar7 raised the priority of this task from to Needs Triage.

adityar7 updated the task description. (Show Details)

adityar7 added a subscriber: adityar7.

That error is a little odd, but try this first:

Open resources/sql/autopatches/20140325.push.4.prune.sql.
Comment out the first statement.
Re-run bin/storage upgrade.
If you still get an error, comment out the first and second statements and run again.
If you still get an error, comment out the first, second, and third statements.
Keep doing this until the patch succeeds. (There are only 5 statements.)
Once the upgrade continues, you can undo your comments.

You could also examine the table to see which statements have been applied, in order to figure out how many to comment out.

You can comment out a statement in SQL like this:

/* ALTER TABLE {$NAMESPACE}_repository.repository_pushlog
    DROP remoteAddress; */

Removing all lines worked, thanks!

Database connection lost while upgrading PhabricatorClosed, ResolvedPublicActions

Description

Event Timeline

Database connection lost while upgrading Phabricator
Closed, ResolvedPublic
Actions