In D20616#260712, @epriestley wrote:

The disk can be full, or read-only. Or become full later, including while the daemon is running.

Any of these policy/disk state problems can pass any effort we make to test them early, then fail after the process daemonizes, making it difficult to report the failure to the user.

Jun 25 2019, 8:29 PM

epriestley requested review of D20619: Bump the remarkup cache version after JIRA/Asana rule changes.

Jun 25 2019, 7:56 PM

epriestley added a revision to T13291: Implement a "{src ...}" rule in Remarkup for referencing source files in Diffusion: D20619: Bump the remarkup cache version after JIRA/Asana rule changes.

Jun 25 2019, 7:55 PM · Remarkup, Diffusion

epriestley added a comment to D20616: Make "PhutilProcessQuery" detection of overseer processes more robust.

Fix typo.
Fix out-of-date comment/logic around overseer tests.

Jun 25 2019, 7:45 PM

epriestley added a comment to D20616: Make "PhutilProcessQuery" detection of overseer processes more robust.

This whole series of changes is motivated by "a PID file was out of date because of a reboot, and we tried to kill some other random process", right? Was there anything else that PID file tracking wasn't doing correctly?

Jun 25 2019, 7:43 PM

amckinley added a comment to D20616: Make "PhutilProcessQuery" detection of overseer processes more robust.

This whole series of changes is motivated by "a PID file was out of date because of a reboot, and we tried to kill some other random process", right? Was there anything else that PID file tracking wasn't doing correctly?

Jun 25 2019, 6:59 PM

epriestley added a revision to T13320: Unannounced/unlogged AWS reboots are apparently routine procedure, not cosmic rays / ghosts: Unknown Object (Differential Revision).

Jun 25 2019, 5:50 PM · Phacility

epriestley added a comment to T13320: Unannounced/unlogged AWS reboots are apparently routine procedure, not cosmic rays / ghosts.

This incantation seems to work as an upstart job:

Jun 25 2019, 5:46 PM · Phacility

epriestley added a comment to T13320: Unannounced/unlogged AWS reboots are apparently routine procedure, not cosmic rays / ghosts.

I'm going to install an upstart task on secure004 and kick it a few times, some stuff might be sketchy until I figure that out.

Jun 25 2019, 5:17 PM · Phacility

epriestley added a comment to T5955: Refactor Conduit auth to be stateless, token-based, and support wire encodings.

Support multiple request encodings (likely BSON, protobuf, or messagepack). Leave JSON as the default, but in cases where messages can not be represented in JSON this gives us a plausible way forward.

Jun 25 2019, 4:55 PM · Restricted Project, Auth, Conduit

epriestley added a comment to T13328: Database dumps can buffer (?) too much data to hold in memory.

The stalled production backup process completed successfully after deploying the change.

Jun 25 2019, 3:54 PM · Restricted Project, Infrastructure, Phacility

epriestley added a comment to T13290: Clean up remaining "Autoclose" behaviors related to "One Revision, Many Commits".

See also PHI1318. The new behavior here has tightened our rules about which users we act as.

Jun 25 2019, 2:38 PM · Diffusion, Differential

epriestley committed rPccf207f20928: (stable) Limit the read buffer size in `bin/storage dump` (authored by epriestley).

(stable) Limit the read buffer size in `bin/storage dump`

Jun 25 2019, 2:17 PM

epriestley added a comment to T13328: Database dumps can buffer (?) too much data to hold in memory.

I'm going to try to sneak this out to the db tier to resolve things before the west coast wakes up, at least.

Jun 25 2019, 2:17 PM · Restricted Project, Infrastructure, Phacility

epriestley committed rPeaa60334ec16: Limit the read buffer size in `bin/storage dump` (authored by epriestley).

Limit the read buffer size in `bin/storage dump`

Jun 25 2019, 2:17 PM

epriestley closed D20617: Limit the read buffer size in `bin/storage dump`.

Jun 25 2019, 2:17 PM

D20617: Limit the read buffer size in `bin/storage dump` is now accepted and ready to land.

Jun 25 2019, 1:53 PM

epriestley requested review of D20617: Limit the read buffer size in `bin/storage dump`.

Jun 25 2019, 12:27 PM

epriestley added a revision to T13328: Database dumps can buffer (?) too much data to hold in memory: D20617: Limit the read buffer size in `bin/storage dump`.

Jun 25 2019, 12:25 PM · Restricted Project, Infrastructure, Phacility

epriestley added a comment to T13328: Database dumps can buffer (?) too much data to hold in memory.

PHP Fatal error: Out of memory (allocated 311164928) (tried to allocate 105988097 bytes) in /core/lib/libphutil/src/future/exec/ExecFuture.php on line 246

Jun 25 2019, 12:18 PM · Restricted Project, Infrastructure, Phacility

epriestley created T13328: Database dumps can buffer (?) too much data to hold in memory.

Jun 25 2019, 12:06 PM · Restricted Project, Infrastructure, Phacility

epriestley requested review of D20616: Make "PhutilProcessQuery" detection of overseer processes more robust.

Jun 25 2019, 12:52 AM

epriestley added a revision to T13321: "bin/phd" may try to kill processes it does not own if the daemon PID logfiles on disk are out of date: D20616: Make "PhutilProcessQuery" detection of overseer processes more robust.

Jun 25 2019, 12:51 AM · Daemons, Phacility