Page MenuHomePhabricator
Create Task
Maniphest T13253

Plans: Daemon Status Reporting
Open, NormalPublic
Actions

Description

See PHI1063. See T6768.

  • Daemon lease names are meaningless/unwieldy.
  • There's no way easy way to connect a particular log message to a particular daemon.
  • There's no easy way to find logs for a given task.

See PHI1062. This is T13121. The -o LogLevel=quiet option to ssh actively impedes debugging and even if the remedy is "filter stderr" we'd be in a better position for stopping the bleeding.

Other adjacent changes:

Revisions and Commits

rP Phabricator
D20246rP9b0b50fbf469 Give "bin/worker" flags to repeat and retry tasks
D20199rPc10b283b9270 Remove some ancient daemon log code
D20175rP4b10bc2b643d Correct schema irregularities (including weird keys) with worker task tables
D20177rP454a76256228 Queue search indexing tasks at a new PRIORITY_INDEX, not PRIORITY_IMPORT

Related Objects

Event Timeline

epriestley triaged this task as Normal priority.Feb 15 2019, 12:19 PM
epriestley created this task.
Herald added subscribers: amckinley, faulconbridge. · View Herald TranscriptFeb 15 2019, 12:19 PM
epriestley updated the task description. (Show Details)Feb 15 2019, 12:23 PM
epriestley added a revision: D20175: Correct schema irregularities (including weird keys) with worker task tables.Feb 15 2019, 12:54 PM
epriestley added a revision: D20177: Queue search indexing tasks at a new PRIORITY_INDEX, not PRIORITY_IMPORT.Feb 15 2019, 1:09 PM
epriestley added a commit: rP454a76256228: Queue search indexing tasks at a new PRIORITY_INDEX, not PRIORITY_IMPORT.Feb 15 2019, 10:16 PM
epriestley added a commit: rP4b10bc2b643d: Correct schema irregularities (including weird keys) with worker task tables.Feb 16 2019, 3:17 AM
epriestley added a revision: D20199: Remove some ancient daemon log code.Feb 20 2019, 5:34 PM
epriestley added a commit: rPc10b283b9270: Remove some ancient daemon log code.Feb 20 2019, 5:52 PM
epriestley added a comment.Feb 20 2019, 6:05 PM

I'm looking at these storage/backend changes:

  • Rename daemon_log.pid to daemon_log.overseerPID. This column already records the overseer PID, and a given daemon "slot" may have multiple PIDs over its lifetime so this can't be "fixed" by recording a different PID.
  • Add pid to daemon_logevent, and record the PID of the daemon which emitted the event.
  • Add taskID to daemon_logevent, and record the task ID the daemon was working on. Not all logs will have a value (for example, the Trigger daemon does not work on tasks), but the task queue is the source of most log messages.
  • Add daemonID to the activetask and archivetask tables and record the (most recent) daemon to work on the task.
  • Add archivedEpoch or similar to archivetask, for recording total time in queue (T5401).
  • Change lease values to purely random values (T6768).
  • Add generation to daemon_logevent and have bin/phd start and similar launch daemon groups with an incrementing generation ID (T10867).
epriestley added a comment.Mar 2 2019, 2:04 PM

See https://discourse.phabricator-community.org/t/sql-error-during-ferret-migration-of-ponder/2471.

Some old migrations call PhabricatorSearchWorker::queueDocumentForIndexing(). This no longer works after D20200 because the PHP implementation expects a dateCreated column to exist, but it won't exist until 20190220.daemon_worker.completed.02.sql runs.

This kind of migration is inherently fragile and we haven't added any since 2017. It was also largely obsoleted by T11932.

I'm just going to no-op all these migrations. Installs that haven't upgraded since 2017 will get a rebuild activity after upgrade anyway, so they don't really do anything.

The index can also always be rebuilt with bin/search index ....

epriestley mentioned this in D20243: No-op old search indexing migrations which no longer run and have been obsoleted by upgrade "activities".Mar 2 2019, 2:08 PM
epriestley added a revision: D20246: Give "bin/worker" flags to repeat and retry tasks.Mar 5 2019, 1:36 AM
epriestley added a commit: rP9b0b50fbf469: Give "bin/worker" flags to repeat and retry tasks.Mar 5 2019, 7:30 PM
epriestley mentioned this in rPee2bc07c9025: No-op old search indexing migrations which no longer run and have been….
• pasik added a subscriber: • pasik.Mar 24 2019, 9:23 PM
joshuaspence added a subscriber: joshuaspence.Apr 20 2019, 12:18 AM
epriestley mentioned this in T13321: "bin/phd" may try to kill processes it does not own if the daemon PID logfiles on disk are out of date.Jun 20 2019, 7:32 PM
Phabricator · Built by Phacility