There are only 23 occurrences of the string "GROUP BY" in the codebase, and, from inspection, many obviously do not conflict with ONLY_FULL_GROUP_BY.

Also remaining is to extend this behavior to the HTTP pathway (and to Mercurial/SVN, eventually).

• if we have already retried 3 times, do not retry;

we'll reduce silly client-visible behavior where you request /tourtle.git instead of /turtle.git and the server seems confused...

See also https://discourse.phabricator-community.org/t/diffusion-search-how-to-do-case-insensitive-search/2854/.

The migration in PHI1403 seems to have gone through cleanly. This workflow can continue to improve, but it's in relatively good shape now.

A generally cleaner version of this would also fix the parallelization TODO (just above the patch in the previous comment). This probably needs FutureIterator to be better at managing in-flight changes to the working set.

Everything here appears to have made it to production cleanly.

Anecdata: locally, using 2 subprocesses went twice as fast (~85s -> ~42s). 4 subprocesses chopped another ~20% of the time off (~42s > ~35s). It stopped getting faster at 4. However, the largest table took 24s, so even if this was completely parallelizable we wouldn't expect it to drop lower than that.

Anecdotally from the last time around, gzipping the tarball didn't really do much. Possibly, this might more broadly imply that we'd be better off not compressing repository backups.

I believe I've moved "core/" from "instances.queryinstances" and sequenced all the followup changes properly, now, and that the only remaining piece is glue.

SyncWorkflow also depends on creatorPHID to synchronize the initial administrator account.

The primary Week 34 / Week 35 deployment has completed without apparent issues. I'm going to deploy some followup changes for T13393 later, but it looks like we're out of the woods on the bulk of outbound changes.

Clicking "Pay Now" from landing page fatals in "PhortuneCartCheckoutController.php:104"; "Call to undefined method PhortuneCartCheckoutController::buildCartContentTable()"
Pacts have a bad URI for billing accounts on the "Billing" tab.
Instances also have a bad URI for billing accounts on the "Billing " tab. Maybe the handle is using the wrong URI?
The sync worker is failing on new instance launch for lack of credentials.

The sync worker is failing on new instance launch for lack of credentials.

bin/host restart does not start no-daemon services.

daemon behaviors

Provisioning was once close-ish to automated. Is this close enough to automate?

I think this leaves us with:

Adjacent is the older instances.queryinstances API method. This is still used by service synchronization.

It is also used to cache InstancesManageCapability::CAPABILITY but this can easily just be cached in the request cache instead.

The operation in PHI1329 (against a ~8GB export) went through cleanly. Remaining work here is:

The "Instance" almanac service type can be destroyed.

Can we get rid of the instance-specific services completely now, after changes connected to T11413?

Mostly-promising answers on much of the rest of this:

For posterity, bleugh:

I have a patch for this, but I'm not thrilled about the retry model. Maybe better would be for the caller retry the actual upload operation (which will automatically resume) and bail out while retaining the temporary file. Even if we retry on 504, we lose a lot of progress if there's a service interruption for longer than we're willing to sit in a retry loop.

I'm hoping this is a reasonable excuse to find a way forward [on >2GB downloads] here.

This is something of an aside, but it would be nice to formalize PhutilConsoleProgressBar into a generic progress sink. A lot of bin/storage dump-related stuff could use this and bin/host download could obviously use it, and we likely have some use cases for reporting progress to the web via the API, but PhutilConsoleProgressBar lacks an indirection layer to really make this work cleanly.

My tentative plan is to add methods for sending the output to disk to HTTPSFuture, then go down the new parser pathway only if we're writing to disk. This should limit the amount of surface area exposed on the new parser.

This task covers a lot of ground and many of the issues have been resolved. There are two remaining issues which are more narrowly covered by these followups:

Is the 2GB HTTP stuff in T12907 realistic to fix?

Not necessarily applicable in the general case, but see also T13393.

A possible issue is that letting cURL pick a protocol might lead to it selecting HTTP/1.0 in some cases (how/when could it possibly do this?