Page MenuHomePhabricator

When returning a writable connection as a "r" connection, label it so it can be reused as a "w" connection
Open, LowPublic

Description

When applications request a "write" connection, we must return a writable connection.

When the application requests a "read" connection, we're free to return a readable or a writable connection.

If the application requests a "read" connection, we currently return a writable connection if we already have one, and establish and return a readable connection otherwise.

It's desirable that the connection we return is labeled as "read" so we can throw if the application runs a write over it, like an INSERT. However, if an existing "w" connection exists we could reasonably return it labeled as an "r" connection. This would also improve "r" vs "w" safety.

That is, these are current bad behaviors:

(A) Allowing Writes on a Read-After-Write Connection
$conn_w = $dao->establishConnection('w');
$conn_r = $dao->establishConnection('r');

// BAD: This works because $conn_r silently "upgraded" and returned
// the existing writable connection.
// Instead, it should throw: no writes allowed on a read connection.
queryfx($conn_r, 'INSERT ...');
(B) Too Many Connections
$conn_r = $dao->establishConnection('r');

// BAD: Always establishes a new connection, but could sometimes return
// the existing connection. Should return existing connection if the
// database on the other end is writable.
$conn_w = $dao->establishConnection('w');

The impact of both is likely quite small. Case (A) is just minor application bugs around 'r' vs 'w' that will manifest once we start sending traffic to read replicas more regularly (see T11056). Case (B) mostly impacts the daemons. These would be good to clean up when T11908 happens, though.

PHI916 notes a likely-adjacent issue with enabling connection persistence and getting connection mode errors:

Exception: Attempting to issue a write query on a read-only connection (to database "phabricator_conduit")!

Event Timeline

I am seeing a similar issue on our install:

[2018-12-13 12:20:12] EXCEPTION: (PhabricatorClusterImproperWriteException) Unable to establish a write-mode connection (to application database "phabricator_repository") because Phabricator is in read-only mode. Whatever you are trying to do does not function correctly in read-only mode. at [<phabricator>/src/infrastructure/storage/lisk/PhabricatorLiskDAO.php:119] arcanist(head=stable, ref.master=d9a4293ae734, ref.stable=45a8d22c74a6), phabricator(head=stable, ref.master=2951694c2737, ref.stable=237a2a190984), phlab(head=master, ref.master=564c60d09ff4), phutil(head=stable, ref.master=dd136d1c3712, ref.stable=414a4c6abb1b)
  #0 PhabricatorLiskDAO::raiseImproperWrite(string) called at [<phabricator>/src/infrastructure/storage/lisk/PhabricatorLiskDAO.php:60]
  #1 PhabricatorLiskDAO::establishLiveConnection(string) called at [<phabricator>/src/infrastructre/storage/lisk/LiskDAO.php:1011]
  #2 LiskDAO::establishConnection(string) called at [<phabricator>/src/applications/repository/stora... (619 more bytes) ... at [<phutil>/src/future/exec/ExecFuture.php:380]
[13-Dec-2018 12:20:12 Etc/UTC] arcanist(head=stable, ref.master=d9a4293ae734, ref.stable=45a8d22c74a6), phabricator(head=stable, ref.master=2951694c2737, ref.stable=237a2a190984), phlab(head=master, ref.master=564c60d09ff4), phutil(head=stable, ref.master=dd136d1c3712, ref.stable=414a4c6abb1b)
[13-Dec-2018 12:20:12 Etc/UTC]   #0 <#3> ExecFuture::resolvex() called at [<phabricator>/src/applications/repository/daemon/PhabricatorRepositoryPullLocalDaemon.php:446]
[13-Dec-2018 12:20:12 Etc/UTC]   #1 phlog(PhutilProxyException) called at [<phabricator>/src/applications/repository/daemon/PhabricatorRepositoryPullLocalDaemon.php:453]
[13-Dec-2018 12:20:12 Etc/UTC]   #2 PhabricatorRepositoryPullLocalDaemon::resolveUpdateFuture(PhabricatorRepository, ExecFuture, integer) called at [<phabricator>/src/applications/repository/daemon/PhabricatorRepositoryPullLocalDaemon.php:222]
[13-Dec-2018 12:20:12 Etc/UTC]   #3 PhabricatorRepositoryPullLocalDaemon::run() called at [<phutil>/src/daemon/PhutilDaemon.php:219]
[13-Dec-2018 12:20:12 Etc/UTC]   #4 PhutilDaemon::execute() called at [<phutil>/scripts/daemon/exec/exec_daemon.php:131]

I am seeing a similar issue on our install:

[2018-12-13 12:20:12] EXCEPTION: (PhabricatorClusterImproperWriteException) Unable to establish a write-mode connection (to application database "phabricator_repository") because Phabricator is in read-only mode. Whatever you are trying to do does not function correctly in read-only mode. at [<phabricator>/src/infrastructure/storage/lisk/PhabricatorLiskDAO.php:119] arcanist(head=stable, ref.master=d9a4293ae734, ref.stable=45a8d22c74a6), phabricator(head=stable, ref.master=2951694c2737, ref.stable=237a2a190984), phlab(head=master, ref.master=564c60d09ff4), phutil(head=stable, ref.master=dd136d1c3712, ref.stable=414a4c6abb1b)
  #0 PhabricatorLiskDAO::raiseImproperWrite(string) called at [<phabricator>/src/infrastructure/storage/lisk/PhabricatorLiskDAO.php:60]
  #1 PhabricatorLiskDAO::establishLiveConnection(string) called at [<phabricator>/src/infrastructre/storage/lisk/LiskDAO.php:1011]
  #2 LiskDAO::establishConnection(string) called at [<phabricator>/src/applications/repository/stora... (619 more bytes) ... at [<phutil>/src/future/exec/ExecFuture.php:380]
[13-Dec-2018 12:20:12 Etc/UTC] arcanist(head=stable, ref.master=d9a4293ae734, ref.stable=45a8d22c74a6), phabricator(head=stable, ref.master=2951694c2737, ref.stable=237a2a190984), phlab(head=master, ref.master=564c60d09ff4), phutil(head=stable, ref.master=dd136d1c3712, ref.stable=414a4c6abb1b)
[13-Dec-2018 12:20:12 Etc/UTC]   #0 <#3> ExecFuture::resolvex() called at [<phabricator>/src/applications/repository/daemon/PhabricatorRepositoryPullLocalDaemon.php:446]
[13-Dec-2018 12:20:12 Etc/UTC]   #1 phlog(PhutilProxyException) called at [<phabricator>/src/applications/repository/daemon/PhabricatorRepositoryPullLocalDaemon.php:453]
[13-Dec-2018 12:20:12 Etc/UTC]   #2 PhabricatorRepositoryPullLocalDaemon::resolveUpdateFuture(PhabricatorRepository, ExecFuture, integer) called at [<phabricator>/src/applications/repository/daemon/PhabricatorRepositoryPullLocalDaemon.php:222]
[13-Dec-2018 12:20:12 Etc/UTC]   #3 PhabricatorRepositoryPullLocalDaemon::run() called at [<phutil>/src/daemon/PhutilDaemon.php:219]
[13-Dec-2018 12:20:12 Etc/UTC]   #4 PhutilDaemon::execute() called at [<phutil>/scripts/daemon/exec/exec_daemon.php:131]

Actually nevermind, I think this is different.