HomePhabricator

Add an intracluster synchronization log for cluster repositories

Description

Add an intracluster synchronization log for cluster repositories

Summary:
Depends on D19778. Ref T13216. See PHI943, PHI889, et al.

We currently have a push log and a pull log, but do not separately log intracluster synchronization events. We've encountered several specific cases where having this kind of log would be helpful:

  • In PHI943, an install was accidentally aborting locks early. Having timing information in the sync log would let us identify this more quickly.
  • In PHI889, an install hit an issue with MaxStartups configuration in sshd. A log would let us identify when this is an issue.
  • In PHI889, I floated a "push the linux kernel + fetch timeout" theory. A sync log would let us see sync/fetch timeouts to confirm this being a problem in practice.
  • A sync log will help us assess, develop, test, and monitor intracluster routing sync changes (likely those in T13211) in the future.

Some of these events are present in the pull log already, but only if they make it as far as running a git upload-pack subprocess (not the case with MaxStartups problems) -- and they can't record end-to-end timing.

No UI yet, I'll add that in a future change.

Test Plan:

  • Forced all operations to synchronize by adding || true to the version check.
  • Pulled, got a sync log in the database.

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13216

Differential Revision: https://secure.phabricator.com/D19779

Details

Provenance
epriestleyAuthored on Nov 6 2018, 10:30 PM
epriestleyPushed on Nov 8 2018, 2:24 AM
Reviewer
amckinley
Differential Revision
D19779: Add an intracluster synchronization log for cluster repositories
Parents
rPe09d29fb1a3e: Clean up the workflow for some post-push logging code
Branches
Unknown
Tags
Unknown
Tasks
T13216: 2018 Week 45-47 Bonus Content
Build Status
Buildable 21113
Build 28700: Run Core Tests