Page MenuHomePhabricator
Create Task
Maniphest T11818

Phacility `notify` tier is pooled, but tries to load database configuration on restart
Closed, ResolvedPublic
Actions

Description

We pool notifications on the notify tier, but bin/aphlict attempts to connect to the configured database when it restarts. No database is available on the tier.

The script doesn't need to read database configuration anyway, so it's sufficient to just not try this connection. However, this should probably be formalized (e.g., with a --do-not-read-database-config flag) or maybe we should just put a dummy database on the host to keep it happy, since that might be easier in the long run than dealing with a weird, special cased, one-off flag. However, there's at least some value in the general case in isolating these hosts from other cluster hosts, and supporting such a flag isn't necessarily unreasonable.

Revisions and Commits

rP Phabricator
D16855rPfc71a7e92dc2 (stable) Discard stdout/stderr from the aphlict subprocess when running in…
D16854rP8a3d6071a026 (stable) Allow bin/aphlict to start without a valid database connection
D16855rP7165e4da90a1 Discard stdout/stderr from the aphlict subprocess when running in daemon…
D16854rP19c6ccb27987 Allow bin/aphlict to start without a valid database connection

Event Timeline

epriestley created this task.Nov 3 2016, 9:04 PM
Herald added subscribers: chad, eadler. · View Herald TranscriptNov 3 2016, 9:04 PM
epriestley added a comment.Nov 14 2016, 12:24 AM

Vaguely related: aphlict exited on notify001 with this error:

[13-Nov-2016 20:24:30 UTC] PHP Fatal error: String size overflow in /core/lib/libphutil/src/future/exec/ExecFuture.php on line 742

That's here:

if ($max_stdout_read_bytes > 0) {
  $this->stdout .= $this->readAndDiscard(
    $stdout,
    $this->getStdoutSizeLimit() - strlen($this->stdout),
    'stdout',
    $max_stdout_read_bytes);
}

We're probably missing a discardBuffers() call somewhere, I'll see if I can hunt it down.

epriestley added a revision: D16854: Allow bin/aphlict to start without a valid database connection.Nov 14 2016, 12:31 AM
epriestley added a revision: D16855: Discard stdout/stderr from the aphlict subprocess when running in daemon (normal) mode.Nov 14 2016, 12:40 AM
epriestley closed this task as Resolved by committing rP7165e4da90a1: Discard stdout/stderr from the aphlict subprocess when running in daemon….Nov 14 2016, 12:43 AM
epriestley added a commit: rP19c6ccb27987: Allow bin/aphlict to start without a valid database connection.
epriestley added a commit: rP7165e4da90a1: Discard stdout/stderr from the aphlict subprocess when running in daemon….
epriestley added a commit: rP8a3d6071a026: (stable) Allow bin/aphlict to start without a valid database connection.
epriestley added a commit: rPfc71a7e92dc2: (stable) Discard stdout/stderr from the aphlict subprocess when running in….
epriestley added a comment.Nov 14 2016, 12:45 AM

I deployed this, seems to have worked properly in production.

Phabricator · Built by Phacility