Some recent change has disrupted this...

There's currently a bug (likely related to D20652 or D20654?) where normal edits aren't respecting card order correctly during the redraw. I suspect this is just an order parameter getting lost somewhere.

Everything has made it to master now so I suspect we're in good shape here.

Adjusting log_warnings = 2 in production (to get connection aborts into the error log) is also possibly desirable, although the background level of connection abortions (general network flakiness, server restarts during deploy, wait_timeout on very long-running demons?) may be high enough that this is more noise than signal.

To reproduce this easily:

I'm likely going to propose some variation of this change, but focus it on max_allowed_packet and on emphasizing that there are two different copies of this setting with different error/failure behavior.

We could also consider these things:

So actual actionable stuff here is:

Bumping max_allowed_packet to 1G in the server config resolved things. The export process then spent a long time doing a bin/files migration (which could use a progress bar, maybe) and is now doing a dump (which could too, although I'm less sure of how we'd build one).

I adjusted innodb_log_file_size to 1GB and attempted the import again, but ran into the same issue.

Let me hold this until I'm more confident I'm on the right track.

The "age of the last checkpoint" error appears to primarily implicate innodb_log_file_size, which is currently set to the default value (5MB):

I'll also double check wait_timeout and interactive_timeout...

Aha! The MySQL error log actually appears to have something useful:

Run it with source ...;

Unzip the dump before running it.

Look at the unzipped dump and see if line 13935 is bad in some obvious way.

(Whatever the resolution is here might also motivate tailoring our restore/import instructions, since this error is pretty opaque and the next steps aren't obvious.)

(Internally, see also PHI1329.)

Not all of this has landed yet, but once it does: