Page MenuHomePhabricator

DaemonsProject
ActivePublic

Watchers (1)

  • This project does not have any watchers.
  • View All

Details

Description

Phabricator daemon infrastructure.

Recent Activity

Mar 10 2021

epriestley closed T13625: Daemons may make commit-related API calls as users without privileges as Resolved.

Although I didn't look particularly hard, I can't immediately find any more evidence that this is occurring in production.

Mar 10 2021, 6:26 PM · Diffusion, Daemons

Mar 2 2021

epriestley updated the task description for T13625: Daemons may make commit-related API calls as users without privileges.
Mar 2 2021, 11:29 PM · Diffusion, Daemons

Mar 1 2021

epriestley added a comment to T13625: Daemons may make commit-related API calls as users without privileges.

When we are importing a commit, we often don't have any legitimate user to act as: the commit may come from an observed repository (so we never authenticated a pusher), and the "Author" and "Committer" strings are arbitrary and untrusted in Git and Mercurial.

Mar 1 2021, 7:09 PM · Diffusion, Daemons
epriestley added a revision to T13625: Daemons may make commit-related API calls as users without privileges: D21582: When updating revisions in responset to commits, use the omnipotent viewer to pull diffs.
Mar 1 2021, 7:00 PM · Diffusion, Daemons
epriestley added a comment to T13625: Daemons may make commit-related API calls as users without privileges.

All three of these are almost certainly diffusion.rawdiffquery, I think the first one just predated D21463 reaching production so it didn't show the method.

Mar 1 2021, 6:15 PM · Diffusion, Daemons

Feb 27 2021

epriestley updated the task description for T13625: Daemons may make commit-related API calls as users without privileges.
Feb 27 2021, 7:41 PM · Diffusion, Daemons
epriestley triaged T13625: Daemons may make commit-related API calls as users without privileges as Normal priority.
Feb 27 2021, 7:38 PM · Diffusion, Daemons

Feb 24 2021

epriestley triaged T13619: Provide an administrative tool to adjust TriggerClock configuration as Low priority.
Feb 24 2021, 10:13 PM · Daemons, Phacility

Feb 2 2021

epriestley closed T13591: Importing repositories may incorrectly queue some tasks at "PRIORITY_COMMIT" as Resolved.

Update the daemon UI to break tasks down by object/container.

Feb 2 2021, 10:37 PM · Daemons, Diffusion
epriestley added a comment to T13561: Allow the daemons to generate tasks dynamically at runtime.

See T13591. Tasks now (may) have a containerPHID, and whatever daemon console changes are made in service of dynamic generation should also expose container objects.

Feb 2 2021, 10:35 PM · Daemons
epriestley added a revision to T13591: Importing repositories may incorrectly queue some tasks at "PRIORITY_COMMIT": D21536: Add basic documentation for using "bin/worker" to manage imports of large repositories.
Feb 2 2021, 9:38 PM · Daemons, Diffusion
epriestley added a revision to T13591: Importing repositories may incorrectly queue some tasks at "PRIORITY_COMMIT": D21535: Add "bin/worker delay" and "bin/worker priority" utilities for managing task queues.
Feb 2 2021, 12:28 AM · Daemons, Diffusion

Feb 1 2021

epriestley added a revision to T13591: Importing repositories may incorrectly queue some tasks at "PRIORITY_COMMIT": D21534: Add more selectors to existing "bin/worker" commands.
Feb 1 2021, 9:53 PM · Daemons, Diffusion
epriestley added a revision to T13591: Importing repositories may incorrectly queue some tasks at "PRIORITY_COMMIT": D21533: Internally, align commit processing tasks around PHIDs, not IDs.
Feb 1 2021, 8:07 PM · Daemons, Diffusion
epriestley added a revision to T13591: Importing repositories may incorrectly queue some tasks at "PRIORITY_COMMIT": D21532: Improve consistency in use of "via", "objectPHID", and "containerPHID" parameters in repository workers.
Feb 1 2021, 7:44 PM · Daemons, Diffusion

Jan 29 2021

epriestley added a revision to T13591: Importing repositories may incorrectly queue some tasks at "PRIORITY_COMMIT": D21531: Add support for a "containerPHID" in the worker queue.
Jan 29 2021, 12:33 AM · Daemons, Diffusion
epriestley added a revision to T13591: Importing repositories may incorrectly queue some tasks at "PRIORITY_COMMIT": D21529: Introduce storage patch "phases" to allow index-rebuilding patches to execute after worker queue schema changes.
Jan 29 2021, 12:23 AM · Daemons, Diffusion

Jan 23 2021

epriestley added a comment to T13591: Importing repositories may incorrectly queue some tasks at "PRIORITY_COMMIT".

After D21518:

Jan 23 2021, 3:51 AM · Daemons, Diffusion
epriestley added a revision to T13591: Importing repositories may incorrectly queue some tasks at "PRIORITY_COMMIT": D21518: Correct some issues around IMPORTED_PERMANENT in RefEngine.
Jan 23 2021, 12:38 AM · Daemons, Diffusion
epriestley added a revision to T13591: Importing repositories may incorrectly queue some tasks at "PRIORITY_COMMIT": D21517: Support an "--active" flag for selecting active tasks.
Jan 23 2021, 12:10 AM · Daemons, Diffusion

Jan 22 2021

epriestley added a comment to T13591: Importing repositories may incorrectly queue some tasks at "PRIORITY_COMMIT".

These parts seem likely resolved once I convince myself the patches so far actually work:

Jan 22 2021, 11:57 PM · Daemons, Diffusion
epriestley added a comment to T13591: Importing repositories may incorrectly queue some tasks at "PRIORITY_COMMIT".

I have a change to add containerPHID locally, but it ends up having relatively high complexity because several other patches (including 20190909.herald.01.rebuild.php) call PhabricatorRebuildIndexesWorker::rebuildObjectsWithQuery(...), which does not work if executed in sequence prior to a worker queue schema change.

Jan 22 2021, 11:47 PM · Daemons, Diffusion
epriestley added a revision to T13591: Importing repositories may incorrectly queue some tasks at "PRIORITY_COMMIT": D21516: Lift logic for queueing commit import tasks into RepositoryEngine.
Jan 22 2021, 10:50 PM · Daemons, Diffusion
epriestley added a revision to T13591: Importing repositories may incorrectly queue some tasks at "PRIORITY_COMMIT": D21515: Separate "shouldPublishRef()" from "isPermanentRef()" and set "IMPORTED_PERMANENT" more narrowly.
Jan 22 2021, 10:09 PM · Daemons, Diffusion
epriestley added a revision to T13591: Importing repositories may incorrectly queue some tasks at "PRIORITY_COMMIT": D21514: Rename "IMPORTED_CLOSEABLE" to "IMPORTED_PERMANENT" to clarify the meaning of the flag.
Jan 22 2021, 9:41 PM · Daemons, Diffusion
epriestley added a comment to T13591: Importing repositories may incorrectly queue some tasks at "PRIORITY_COMMIT".

This also relates slightly to T13580, but I believe the two issues are addressable independently.

Jan 22 2021, 9:24 PM · Daemons, Diffusion
epriestley added a comment to T13591: Importing repositories may incorrectly queue some tasks at "PRIORITY_COMMIT".

However, I'd like to have a better understanding of how we're reaching this state, and I'm not satisfied that these repositories are going down the "natural" pathway (of changing ref definitions after the import starts) and suspect there is some more complicated interaction at play here.

Jan 22 2021, 9:18 PM · Daemons, Diffusion
epriestley added a revision to T13591: Importing repositories may incorrectly queue some tasks at "PRIORITY_COMMIT": D21513: Correct the behavior of "bin/repository discover --repair".
Jan 22 2021, 8:25 PM · Daemons, Diffusion
epriestley added a comment to T13591: Importing repositories may incorrectly queue some tasks at "PRIORITY_COMMIT".

I'm hoping to land at least a narrow fix for this today to support an import in PHI1979 tomorrow. However, I'd like to have a better understanding of how we're reaching this state, and I'm not satisfied that these repositories are going down the "natural" pathway (of changing ref definitions after the import starts) and suspect there is some more complicated interaction at play here.

Jan 22 2021, 8:23 PM · Daemons, Diffusion
epriestley updated the task description for T13591: Importing repositories may incorrectly queue some tasks at "PRIORITY_COMMIT".
Jan 22 2021, 6:47 PM · Daemons, Diffusion
epriestley updated the task description for T13591: Importing repositories may incorrectly queue some tasks at "PRIORITY_COMMIT".
Jan 22 2021, 6:45 PM · Daemons, Diffusion
epriestley added a project to T13591: Importing repositories may incorrectly queue some tasks at "PRIORITY_COMMIT": Daemons.
Jan 22 2021, 6:05 PM · Daemons, Diffusion

Oct 26 2020

sshannin added a comment to T4627: PhabricatorTaskmasterDaemon 100% CPU load.

I know it's an old thread, but I am encountering what is, as far as I can tell, exactly the same issue. I'm on the latest version of trunk (~Oct 26) with php 5.5.9 on Linux 3.13.0. I also observed the issue on the May 30 version.

Oct 26 2020, 11:17 PM · Daemons

Oct 16 2020

epriestley added a revision to T13572: Daemon overseers may have a slow memory leak, possibly in FuturePool: D21478: Fix a typo of "previously" in FutureIterator.
Oct 16 2020, 7:31 PM · Daemons

Sep 17 2020

epriestley added a comment to T13572: Daemon overseers may have a slow memory leak, possibly in FuturePool.

This synthetic script demonstrates the conceptual problem with FutureIterator:

Sep 17 2020, 7:49 PM · Daemons

Sep 16 2020

epriestley closed T13569: When running worker tasks in-process, subtasks which fail incorrectly cause the parent tasks to fail as Resolved by committing rP737e7c85412c: When an in-process worker subtask fails permanently, don't fatal the whole….
Sep 16 2020, 12:37 AM · Daemons

Sep 8 2020

epriestley added a revision to T13569: When running worker tasks in-process, subtasks which fail incorrectly cause the parent tasks to fail: D21459: When an in-process worker subtask fails permanently, don't fatal the whole process.
Sep 8 2020, 8:15 PM · Daemons

Sep 4 2020

epriestley added a revision to T13579: Overseer may fatal when trying to write to "pid" property in "annihilateProcessGroup()": D21452: Remove obsolete write to "pid" property in "annihilateProcessGroup()" in Daemon Overseer.
Sep 4 2020, 11:39 PM · Daemons
epriestley triaged T13579: Overseer may fatal when trying to write to "pid" property in "annihilateProcessGroup()" as Normal priority.
Sep 4 2020, 11:38 PM · Daemons

Aug 17 2020

epriestley triaged T13572: Daemon overseers may have a slow memory leak, possibly in FuturePool as Low priority.
Aug 17 2020, 3:17 PM · Daemons

Aug 12 2020

epriestley triaged T13569: When running worker tasks in-process, subtasks which fail incorrectly cause the parent tasks to fail as Wishlist priority.
Aug 12 2020, 8:32 PM · Daemons

Aug 11 2020

epriestley updated the task description for T13561: Allow the daemons to generate tasks dynamically at runtime.
Aug 11 2020, 3:26 PM · Daemons
epriestley triaged T13561: Allow the daemons to generate tasks dynamically at runtime as Normal priority.
Aug 11 2020, 3:15 PM · Daemons

Jul 24 2020

epriestley added a comment to T13556: Setting "phd.trace" may cause TTY testing with "posix_isatty()" against closed pipes.

In D21426, I removed phd.trace and phd.verbose.

Jul 24 2020, 12:18 AM · Daemons

Jul 23 2020

epriestley added a revision to T13556: Setting "phd.trace" may cause TTY testing with "posix_isatty()" against closed pipes: D21426: Remove ancient "phd.trace" and "phd.verbose" configuration options.
Jul 23 2020, 7:22 PM · Daemons
epriestley claimed T13556: Setting "phd.trace" may cause TTY testing with "posix_isatty()" against closed pipes.
Jul 23 2020, 6:05 PM · Daemons
epriestley triaged T13556: Setting "phd.trace" may cause TTY testing with "posix_isatty()" against closed pipes as Wishlist priority.
Jul 23 2020, 6:05 PM · Daemons
epriestley added a revision to T13555: In "ExecFuture", "proc_get_status()" may be passed NULL, possibly leading to a daemon stall: D21425: Streamline handling of Futures and PIDs in daemons.
Jul 23 2020, 5:40 PM · Daemons
epriestley added a comment to T13555: In "ExecFuture", "proc_get_status()" may be passed NULL, possibly leading to a daemon stall.

The call sequence here is approximately:

Jul 23 2020, 5:11 PM · Daemons
epriestley added a revision to T13555: In "ExecFuture", "proc_get_status()" may be passed NULL, possibly leading to a daemon stall: D21424: Manage PIDs more carefully in DaemonHandle.
Jul 23 2020, 5:09 PM · Daemons