I kicked a couple things out and moved the rest to T13164 to sort of force things back on some kind of normal-looking track with a plausible connection to realtime.

I'm descheduling this out of T13151 since none of this is too burning and it's grown kind of big. Probably makes more sense after T13076 as we move toward private clusters, since these UIs will need updates anyway.

I've marked D19541 as resolving this. The change there isn't quite what was asked for here, but overlaps significantly.

See also PHI786 for an example of a general issue: voiding invoices is currently inconvenient: there's no actual "void" state for invoices so you have to go hunt them down manually and DELETE them. There should be a formal "Void" state they can be kicked into.

I can't get anything to break at all now. 💁

I made a change like this to split responses into small blocks and give each a random chance of failing (I tried 4096 and 4027, a prime near 4KB, as miniblock sizes):

A recent, probably more concrete issue that I've seen is that wget of a large file can succeed (wget automatically resumes) but we can end up with a file that has different content than the original, which is disastrously bad.

I cherry-picked D19536, deployed it to aux001, and restarted the file data copy, which worked this time. I'm going to restart the export step and hopefully we're off to the races this time.

403 instead of a 404

We could probably clean this up (it does look a lot like an error) but DarkConsole isn't exactly a shining example of perfect visual design anyway and I'll probably remember or catch this when it next gets a cleanup pass, some time in 3050.

It seems like that went through cleanly. I just did Stop + Start + bin/remote deploy on the affected hosts. I then launched a test instance with placement allocations on two of the affected services; it came up cleanly.

Beginning the stop/start stuff now.

I'm planning to stop/start these instances during the maintenance window today since getting the rebalance into production in the next five days seems wildly optimistic.

I made it as far as locala.phacility.com working right so maybe I got everything?

I think in D19196 the engine always generated uppercase text, but that got changed in D19200, perhaps?

how did this ever escape us

See email. An instance got an invite into an awkward state by cancelling the invite after the user had accepted it but before they registered an account.

The GRANT syntax also appears to have changed. We currently do this (in Phacility-specific code):

Also ignore the untamed-wilds branch, I'm just pulling it out of the wreckage of my explosion-prone laptop (see also T13168).

ignore me

(Yell if I'm assuming too much here.)

The fix I'd propose is:

This isn't meaningful, but we can raise a more specific error.

When you "Analyze", we include a stack trace of where the query was issued from so you can hunt it down if you aren't sure. Does that explain the behavior?

Via PHI773, filter_input_array(INPUT_ENV, FILTER_UNSAFE_RAW); does not appear to work if variables_order excludes E.

Plans, yes. Timing is trickier. Currently, we only have one outstanding request for this feature (PHI760, which mostly wants project.search). So it's in queue somewhere, but currently beyond the forecasting horizon.

Conduit API methods define a getRequiredScope() method, which tells you the scope they require.

Build just the extension from source?

I'm going to look at improving the push logs a bit to show which rule blocked a push to address the actual use case in PHI765, which makes this more of a distance nice-to-have.

T1022 is possibly somewhat-vaguely-adjacent on symlink stuff.

As a special case of this, if you commit an empty a.py file, then add content to it and also add a new empty b.py file in a commit on top of it, the new empty b.py will be detected as a copy of a.py based on the previous (empty) content of the file. I think Git is being pretty reasonable/consistent here, but this is potentially also expectation-defying:

Trying to reproduce this locally just hits the ft_doc.epochModified issue. I'm not immediately sure why the behavior differs between my local install and secure, but that issue probably needs to be fixed first.

Actually, I'm not entirely right in merging that task -- T13163 isn't quite the same as the other two issues here. I think they're similar, but the query text is relevant in the case of T13163. Notably: