On SMS MFA, Reddit got bitten recently by SMS having poor security properties:

The workaround does not work for me on Debian 9.5 with OpenSSH 7.4p1. Actually I am surprised this worked for anyone, since the issue appears to be that the buffer of the pipe between ssh-auth's stdout and the parent's stdin is being filled, hence ssh-auth blocking at some point. It should not matter how fast these buffers are being filled...

We have a more reliable break for this now, in the form:

Looks like CircleCI will stop supporting the V1 API, which we use, in about 5 weeks: https://circleci.com/sunset1-0/

I kicked a couple things out and moved the rest to T13164 to sort of force things back on some kind of normal-looking track with a plausible connection to realtime.

I'm descheduling this out of T13151 since none of this is too burning and it's grown kind of big. Probably makes more sense after T13076 as we move toward private clusters, since these UIs will need updates anyway.

I've marked D19541 as resolving this. The change there isn't quite what was asked for here, but overlaps significantly.

See also PHI786 for an example of a general issue: voiding invoices is currently inconvenient: there's no actual "void" state for invoices so you have to go hunt them down manually and DELETE them. There should be a formal "Void" state they can be kicked into.

I can't get anything to break at all now. 💁

I made a change like this to split responses into small blocks and give each a random chance of failing (I tried 4096 and 4027, a prime near 4KB, as miniblock sizes):

A recent, probably more concrete issue that I've seen is that wget of a large file can succeed (wget automatically resumes) but we can end up with a file that has different content than the original, which is disastrously bad.

I cherry-picked D19536, deployed it to aux001, and restarted the file data copy, which worked this time. I'm going to restart the export step and hopefully we're off to the races this time.

403 instead of a 404

We could probably clean this up (it does look a lot like an error) but DarkConsole isn't exactly a shining example of perfect visual design anyway and I'll probably remember or catch this when it next gets a cleanup pass, some time in 3050.

Yeah it does, and I just didn't recall ever seeing this behavior and assumed "stack trace" meant "error".

It seems like that went through cleanly. I just did Stop + Start + bin/remote deploy on the affected hosts. I then launched a test instance with placement allocations on two of the affected services; it came up cleanly.

Beginning the stop/start stuff now.

I'm planning to stop/start these instances during the maintenance window today since getting the rebalance into production in the next five days seems wildly optimistic.

I made it as far as locala.phacility.com working right so maybe I got everything?

I think in D19196 the engine always generated uppercase text, but that got changed in D19200, perhaps?