Since observed repositories version differently today, this strategy won't work -- but I can't come up with any valid reason to ever put a repository into a "write maintenance" mode anyway. I do imagine making observed repositories "replay" fetches into the push log (as though they were pushes) in the future, but that still won't make "write maintenance" on an observed repository meaningful, so it seems fine to just prevent putting non-hosted repositories into this mode.

A minor issue on the way to this is that calling synchronizeWorkingCopyBeforeWrite() with an omnipotent viewer will write to the WorkingCopyVersion table with a null userPHID, which shows as "Unknown Object" in the UI.

A useful maintenance operation for staging area repositories is to remove out-of-date staging refs: old diffs which have already landed. This is of some particular importance for large installs, since Git has a significant per-ref overhead for many operations until protocol v2: by the time a repository has ~50K refs, interacting with it in basically any way has become slow and cumbersome.

• This was originally implemented in D19357.
• This was broken by a refactoring change in D20775.
• I made problem observable in D21575 fixed it in D21576.

Also remaining is to extend this behavior to the HTTP pathway (and to Mercurial/SVN, eventually).

• if we have already retried 3 times, do not retry;

we'll reduce silly client-visible behavior where you request /tourtle.git instead of /turtle.git and the server seems confused...

Not necessarily applicable in the general case, but see also T13393.

I assume this is being done already in the Phacility cluster on some level when repositories get really large, but I'm not particularly sure how to perform this migration.

See PHI1015 for a slightly meatier explanation of this issue.

Yeah, that's T10769.

In T13219#242865, @joshuaspence wrote:

I am seeing a similar issue on our install:

[2018-12-13 12:20:12] EXCEPTION: (PhabricatorClusterImproperWriteException) Unable to establish a write-mode connection (to application database "phabricator_repository") because Phabricator is in read-only mode. Whatever you are trying to do does not function correctly in read-only mode. at [<phabricator>/src/infrastructure/storage/lisk/PhabricatorLiskDAO.php:119] arcanist(head=stable, ref.master=d9a4293ae734, ref.stable=45a8d22c74a6), phabricator(head=stable, ref.master=2951694c2737, ref.stable=237a2a190984), phlab(head=master, ref.master=564c60d09ff4), phutil(head=stable, ref.master=dd136d1c3712, ref.stable=414a4c6abb1b)
  #0 PhabricatorLiskDAO::raiseImproperWrite(string) called at [<phabricator>/src/infrastructure/storage/lisk/PhabricatorLiskDAO.php:60]
  #1 PhabricatorLiskDAO::establishLiveConnection(string) called at [<phabricator>/src/infrastructre/storage/lisk/LiskDAO.php:1011]
  #2 LiskDAO::establishConnection(string) called at [<phabricator>/src/applications/repository/stora... (619 more bytes) ... at [<phutil>/src/future/exec/ExecFuture.php:380]
[13-Dec-2018 12:20:12 Etc/UTC] arcanist(head=stable, ref.master=d9a4293ae734, ref.stable=45a8d22c74a6), phabricator(head=stable, ref.master=2951694c2737, ref.stable=237a2a190984), phlab(head=master, ref.master=564c60d09ff4), phutil(head=stable, ref.master=dd136d1c3712, ref.stable=414a4c6abb1b)
[13-Dec-2018 12:20:12 Etc/UTC]   #0 <#3> ExecFuture::resolvex() called at [<phabricator>/src/applications/repository/daemon/PhabricatorRepositoryPullLocalDaemon.php:446]
[13-Dec-2018 12:20:12 Etc/UTC]   #1 phlog(PhutilProxyException) called at [<phabricator>/src/applications/repository/daemon/PhabricatorRepositoryPullLocalDaemon.php:453]
[13-Dec-2018 12:20:12 Etc/UTC]   #2 PhabricatorRepositoryPullLocalDaemon::resolveUpdateFuture(PhabricatorRepository, ExecFuture, integer) called at [<phabricator>/src/applications/repository/daemon/PhabricatorRepositoryPullLocalDaemon.php:222]
[13-Dec-2018 12:20:12 Etc/UTC]   #3 PhabricatorRepositoryPullLocalDaemon::run() called at [<phutil>/src/daemon/PhutilDaemon.php:219]
[13-Dec-2018 12:20:12 Etc/UTC]   #4 PhutilDaemon::execute() called at [<phutil>/scripts/daemon/exec/exec_daemon.php:131]

I am seeing a similar issue on our install:

See PHI860 and T13111. In the future, repository nodes may automatically gc/prune/repack. If they do, it may make sense to sort them to the bottom of the list so traffic is sent to them only if no other nodes are available, in order to minimize the impact that gc/prune/repack have on other activity.

Whoops this is probably my bad :-/

See PHI14 for priority.

Cool, that all sounds like it's roughly what I expected. Thanks!

Skipping questions where you have the right answer:

Synchronously in the request. I'll look at adding logging.

If you go through the process manually with the web UI, are the versions OK after initial creation? After adding an observed URI?

So I'm not immediately able to reproduce this. Some possible questions:

I used this parameter bundle to create a repository:

I'm attempting to reproduce this from the web UI first. I created a new repository, and get 0 versions as expected:

try to get harbormaster to build (push to staging?)

• actually, acutally utilize the health monitoring...

• Improved the status monitoring UI in config/cluster/search/
• Actually utilize the health monitoring cache to avoid connecting to downed servers.

@epriestley sweet, I'll land this as soon as I see that you've merged to stable.

I don't see any noteworthy issues here.

Haha, thanks. Let me finish testing one other change and then I'll give this another look.

@epriestley: I think this is ready to land but I want to give you one more chance to change your mind.

• Created diviner documentation: Cluster: Search
• removed stray phlog

• Fix searching relationships which I had inadvertantly broken.
• Better elasticsearch 2.x and 5.x support
• more optimized query

Fix method signature un-final PhabricatorElasticFulltextStorageEngine

Ok I think I've eliminated the problematic parts like indexing project slugs.