Progress so far:

I'm probably done for the night, here's a quick update:

We need this for T4571 for the cluster status console so it doesn't take 10000 years if replicas are down.

We haven't seen further reports of this in more than two months, so I don't plan to mitigate it at the application level.

I think it's very unlikely that we'll ever write some kind of script to just fix this stuff. I suspect such a script is impossible to write (i.e., the "correct" value for most fields is ambiguous) and that maintaining it is very difficult.

I think that would be reasonable.

Why not change Phobject::__isset and Phobject::__unset to throw an exception in the same way that Phobject::__get and Phobject::__set do?

D15358 should fix this by declaring the property. Nice catch on finding this!

I'm just going to close this because it doesn't identify any specific problems and I don't want to build these things to deal with hypothetical problems.

I think it's good that destruction happens in the foreground, since I think this should be a very rare operation and errors should always be reported to the user. Users who want to background long destruction operations for some reason can use tools like screen.

I don't believe so, https://secure.phabricator.com/T4209#146794 clearly states that until that patch lands, daemons and repos must run on the same host.

I thought that it should still work because I'm running the repository pull daemon on the worker host?

Right, try stopping the pull daemons on the other tiers and leave only one running on the repository machine and see if that resolves your issue.

Sure, I get that... But the import is failing even on the instance running without the --no-discovery flag.

It means you don't need to run the taskmaster daemons on the same machines as the repositories because it can just make a Conduit call instead. You no longer need to run separate daemon tiers or use the --no-discovery flag.

I don't think that would fix the deadlock issue, which seems to be the root cause?

You probably need to apply D11874, which fixes this issue.

Seeing this on the repository host:

The number of direct invocations of Editors in the codebase is decreasing over time, reducing the value of building this.

I don't know how to reproduce this, either. This may fix itself as T9132 evolves.

Fixed in HEAD (D14571).

Sorry about the delay... here's the trace from running /bin/storage upgrade --namespace testT9773_$RANDOM --force --no-adjust --trace:

No one else can reproduce this. storage upgrade --trace ... might provide some insight, or add --xprofile to that or whatever. We can look at it in more detail if you can narrow it down more.

Maybe adding --trace will reveal what's going on?