Since observed repositories version differently today, this strategy won't work -- but I can't come up with any valid reason to ever put a repository into a "write maintenance" mode anyway. I do imagine making observed repositories "replay" fetches into the push log (as though they were pushes) in the future, but that still won't make "write maintenance" on an observed repository meaningful, so it seems fine to just prevent putting non-hosted repositories into this mode.

A minor issue on the way to this is that calling synchronizeWorkingCopyBeforeWrite() with an omnipotent viewer will write to the WorkingCopyVersion table with a null userPHID, which shows as "Unknown Object" in the UI.

A useful maintenance operation for staging area repositories is to remove out-of-date staging refs: old diffs which have already landed. This is of some particular importance for large installs, since Git has a significant per-ref overhead for many operations until protocol v2: by the time a repository has ~50K refs, interacting with it in basically any way has become slow and cumbersome.

• This was originally implemented in D19357.
• This was broken by a refactoring change in D20775.
• I made problem observable in D21575 fixed it in D21576.

Also remaining is to extend this behavior to the HTTP pathway (and to Mercurial/SVN, eventually).

• if we have already retried 3 times, do not retry;

we'll reduce silly client-visible behavior where you request /tourtle.git instead of /turtle.git and the server seems confused...

Not necessarily applicable in the general case, but see also T13393.

I assume this is being done already in the Phacility cluster on some level when repositories get really large, but I'm not particularly sure how to perform this migration.

See PHI1015 for a slightly meatier explanation of this issue.

Yeah, that's T10769.

In T13219#242865, @joshuaspence wrote:

I am seeing a similar issue on our install:

[2018-12-13 12:20:12] EXCEPTION: (PhabricatorClusterImproperWriteException) Unable to establish a write-mode connection (to application database "phabricator_repository") because Phabricator is in read-only mode. Whatever you are trying to do does not function correctly in read-only mode. at [<phabricator>/src/infrastructure/storage/lisk/PhabricatorLiskDAO.php:119] arcanist(head=stable, ref.master=d9a4293ae734, ref.stable=45a8d22c74a6), phabricator(head=stable, ref.master=2951694c2737, ref.stable=237a2a190984), phlab(head=master, ref.master=564c60d09ff4), phutil(head=stable, ref.master=dd136d1c3712, ref.stable=414a4c6abb1b)
  #0 PhabricatorLiskDAO::raiseImproperWrite(string) called at [<phabricator>/src/infrastructure/storage/lisk/PhabricatorLiskDAO.php:60]
  #1 PhabricatorLiskDAO::establishLiveConnection(string) called at [<phabricator>/src/infrastructre/storage/lisk/LiskDAO.php:1011]
  #2 LiskDAO::establishConnection(string) called at [<phabricator>/src/applications/repository/stora... (619 more bytes) ... at [<phutil>/src/future/exec/ExecFuture.php:380]
[13-Dec-2018 12:20:12 Etc/UTC] arcanist(head=stable, ref.master=d9a4293ae734, ref.stable=45a8d22c74a6), phabricator(head=stable, ref.master=2951694c2737, ref.stable=237a2a190984), phlab(head=master, ref.master=564c60d09ff4), phutil(head=stable, ref.master=dd136d1c3712, ref.stable=414a4c6abb1b)
[13-Dec-2018 12:20:12 Etc/UTC]   #0 <#3> ExecFuture::resolvex() called at [<phabricator>/src/applications/repository/daemon/PhabricatorRepositoryPullLocalDaemon.php:446]
[13-Dec-2018 12:20:12 Etc/UTC]   #1 phlog(PhutilProxyException) called at [<phabricator>/src/applications/repository/daemon/PhabricatorRepositoryPullLocalDaemon.php:453]
[13-Dec-2018 12:20:12 Etc/UTC]   #2 PhabricatorRepositoryPullLocalDaemon::resolveUpdateFuture(PhabricatorRepository, ExecFuture, integer) called at [<phabricator>/src/applications/repository/daemon/PhabricatorRepositoryPullLocalDaemon.php:222]
[13-Dec-2018 12:20:12 Etc/UTC]   #3 PhabricatorRepositoryPullLocalDaemon::run() called at [<phutil>/src/daemon/PhutilDaemon.php:219]
[13-Dec-2018 12:20:12 Etc/UTC]   #4 PhutilDaemon::execute() called at [<phutil>/scripts/daemon/exec/exec_daemon.php:131]

I am seeing a similar issue on our install:

See PHI860 and T13111. In the future, repository nodes may automatically gc/prune/repack. If they do, it may make sense to sort them to the bottom of the list so traffic is sent to them only if no other nodes are available, in order to minimize the impact that gc/prune/repack have on other activity.

Whoops this is probably my bad :-/

See PHI14 for priority.

Cool, that all sounds like it's roughly what I expected. Thanks!

Skipping questions where you have the right answer:

Synchronously in the request. I'll look at adding logging.

If you go through the process manually with the web UI, are the versions OK after initial creation? After adding an observed URI?

So I'm not immediately able to reproduce this. Some possible questions:

I used this parameter bundle to create a repository:

I'm attempting to reproduce this from the web UI first. I created a new repository, and get 0 versions as expected:

I'm going to put D17497 + D17498 into this release even though they don't directly tackle the issue here and I think they're slightly risky changes (mostly because git ls-remote may have odd behaviors in some cases, and we don't currently use it in other workflows). But they may help with T12296 and general cluster load issues, and the followup changes will generally be more complicated to reason about (more locking/concurrency stuff), so I think getting these in earlier spreads risk out somewhat even though they're something to watch out for in this release.

Probably bump the version unconditionally now

(For example, if the goal was to very aggressively optimize for minimizing network traffic, we could read the entire repository history to figure out which refs were ancestors of other refs first? Then we could drop those and only ask for descendant refs. But this seems crazy, since it's saying that ~20 bytes of network traffic is more costly than like one hundred million disk I/O operations?)

it has to compare "What do I have" vs. "What do you have"

Do git ls-remote (which, curiously, seems to be significantly faster than git fetch even when git fetch is a no-op).

I think T12393 and this probably have a similar set of root causes.

This has been running cleanly in production for roughly two weeks, and appears stable. secure001 stopped writing Files data at F1943597 and we're now at F2078289 on secure003. We saw a couple of minor setup issues (mostly: exception messages not being tailored enough) but no fundamental issues.

This seems to be working now. I'm going to let it sit in production for a while and see if any issues crop up before considering it resolved, but it seems like everything is working smoothly.

I configured 003 to replicate to 004:

Okay, we're headed back into readonly mode shortly to set up replication. I'm going to verify D16916 along the way so there may be some "partitions disagree about life" errors.

Here's what I've done so far: