The migration in PHI1403 seems to have gone through cleanly. This workflow can continue to improve, but it's in relatively good shape now.

A generally cleaner version of this would also fix the parallelization TODO (just above the patch in the previous comment). This probably needs FutureIterator to be better at managing in-flight changes to the working set.

Anecdata: locally, using 2 subprocesses went twice as fast (~85s -> ~42s). 4 subprocesses chopped another ~20% of the time off (~42s > ~35s). It stopped getting faster at 4. However, the largest table took 24s, so even if this was completely parallelizable we wouldn't expect it to drop lower than that.

Anecdotally from the last time around, gzipping the tarball didn't really do much. Possibly, this might more broadly imply that we'd be better off not compressing repository backups.

I believe I've moved "core/" from "instances.queryinstances" and sequenced all the followup changes properly, now, and that the only remaining piece is glue.

bin/host restart does not start no-daemon services.

daemon behaviors

Provisioning was once close-ish to automated. Is this close enough to automate?

I think this leaves us with:

Adjacent is the older instances.queryinstances API method. This is still used by service synchronization.

It is also used to cache InstancesManageCapability::CAPABILITY but this can easily just be cached in the request cache instead.

The operation in PHI1329 (against a ~8GB export) went through cleanly. Remaining work here is:

The "Instance" almanac service type can be destroyed.

Can we get rid of the instance-specific services completely now, after changes connected to T11413?

Mostly-promising answers on much of the rest of this:

For posterity, bleugh:

I have a patch for this, but I'm not thrilled about the retry model. Maybe better would be for the caller retry the actual upload operation (which will automatically resume) and bail out while retaining the temporary file. Even if we retry on 504, we lose a lot of progress if there's a service interruption for longer than we're willing to sit in a retry loop.

I'm hoping this is a reasonable excuse to find a way forward [on >2GB downloads] here.

This is something of an aside, but it would be nice to formalize PhutilConsoleProgressBar into a generic progress sink. A lot of bin/storage dump-related stuff could use this and bin/host download could obviously use it, and we likely have some use cases for reporting progress to the web via the API, but PhutilConsoleProgressBar lacks an indirection layer to really make this work cleanly.

My tentative plan is to add methods for sending the output to disk to HTTPSFuture, then go down the new parser pathway only if we're writing to disk. This should limit the amount of surface area exposed on the new parser.

Is the 2GB HTTP stuff in T12907 realistic to fix?

A possible issue is that letting cURL pick a protocol might lead to it selecting HTTP/1.0 in some cases (how/when could it possibly do this?

According to curl/symbols-in-versions (this is a text file in the repository):

A possible issue is that letting cURL pick a protocol might lead to it selecting HTTP/1.0 in some cases (how/when could it possibly do this? Only by hard-coding known-broken hostnames, I think?), and forcing it to use HTTP/1.1 could break those cases, so maybe I'll go spelunking here. I also can't immediately find a date of introduction for CURL_HTTP_VERSION_1_1 from the documentation.

An adjacent issue is that PhabricatorMarkupCache is not currently marked as having cache persistence

loading a dump with missing data into a replica and then starting replication really causes issues

We also need --enable-zip to get the zip extension, to get the ZipArchive class, so "Export to Excel" works. See upcoming change on T13342.

An adjacent issue is that PhabricatorMarkupCache is not currently marked as having cache persistence (PhabricatorConfigTableSchema::PERSISTENCE_CACHE), so the data dumps even when we do not intend to dump data for readthrough caches.

The stalled production backup process completed successfully after deploying the change.

I'm going to try to sneak this out to the db tier to resolve things before the west coast wakes up, at least.

PHP Fatal error: Out of memory (allocated 311164928) (tried to allocate 105988097 bytes) in /core/lib/libphutil/src/future/exec/ExecFuture.php on line 246

The power is in my hands, now.

The change above seems to have worked around things, I'll leave this open until we can confirm that PHP 7.3.6 or some later version fixes the underlying issue.

The changelog for PHP 7.3.6 mentions "Fixed possible crashes, because of inconsistent PCRE cache and opcache SHM reset." which might be relevant, but no bug link so who knows.

I built my PHP from source in T13232 so this is presumably not a Homebrew issue.

(This seems stable now, and there's no specific action here.)

This seems to have quieted down, now.