Page MenuHomePhabricator

PhacilityOrganization
ActivePublic

Details

Description

Phabricator SaaS

Recent Activity

Tue, Jan 21

epriestley added a comment to T13111: Periodically run `git prune` on Git working copies.

The logic here appears to be that gc.auto is set to some value (by default: 6,700). If the number of loose objects exceeds this threshold (technically, if the number of loose objects in objects/17/ is more than 1/256th of this value), it triggers a repack (in a comment, git repack -d -l).

Tue, Jan 21, 6:00 PM · Phacility, Diffusion
epriestley added a comment to T13111: Periodically run `git prune` on Git working copies.

See PHI1613, where an install hit this warning (and resolved it by running git prune):

Tue, Jan 21, 5:13 PM · Phacility, Diffusion

Wed, Jan 15

epriestley closed T13477: (2020 Week 3) Restart AWS host db025 (i-05bc80634586ef7a0) as Resolved.

This went through cleanly.

Wed, Jan 15, 4:19 PM · Ops, Phacility
epriestley triaged T13477: (2020 Week 3) Restart AWS host db025 (i-05bc80634586ef7a0) as Low priority.
Wed, Jan 15, 2:36 AM · Ops, Phacility

Nov 26 2019

epriestley triaged T13467: Expand the "ddata003" volume as Low priority.
Nov 26 2019, 6:00 PM · Phacility
epriestley added a comment to T13466: AWS instance termination may fail/hang indefinitely.

there is no way to bin/host query against the set of instances using a particular repository shard service

Nov 26 2019, 12:14 AM · Phacility, Ops
epriestley added a revision to T13466: AWS instance termination may fail/hang indefinitely: Restricted Differential Revision.
Nov 26 2019, 12:09 AM · Phacility, Ops

Nov 25 2019

epriestley added a revision to T13466: AWS instance termination may fail/hang indefinitely: Restricted Differential Revision.
Nov 25 2019, 11:57 PM · Phacility, Ops
epriestley added a revision to T13466: AWS instance termination may fail/hang indefinitely: Restricted Differential Revision.
Nov 25 2019, 10:46 PM · Phacility, Ops
epriestley added a comment to T13466: AWS instance termination may fail/hang indefinitely.

Update Almanac definitions for all instances not on the paired db023 shard.

Nov 25 2019, 5:01 PM · Phacility, Ops
epriestley added a comment to T13466: AWS instance termination may fail/hang indefinitely.

PHI1566 is resolved narrowly. These cleanup steps still need to happen.

Nov 25 2019, 4:42 PM · Phacility, Ops
epriestley added a comment to T13466: AWS instance termination may fail/hang indefinitely.

(Updating addresses with bin/host query leaves the service address cache dirty (the "mutable structure cache" via PhabricatorRepository->getAlmanacServiceRefs()) so it should be followed with bin/cache purge --caches general.)

Nov 25 2019, 4:29 PM · Phacility, Ops
epriestley added a comment to T13466: AWS instance termination may fail/hang indefinitely.

I'll flesh this out more later, but the move away from db123 = repo123 shard pairing, plus bin/host query using mysql makes it difficult to directly query instances using a particular repository service.

Nov 25 2019, 4:15 PM · Phacility, Ops
epriestley added a comment to T13466: AWS instance termination may fail/hang indefinitely.

Minor issue that should be looked at during service sync arising from improved validation elsewhere:

Nov 25 2019, 4:06 PM · Phacility, Ops
epriestley added a comment to T13466: AWS instance termination may fail/hang indefinitely.

I'm deploying the new host now. We just crossed a release so I'm going to manually restore it to 72f82abe07 once it comes up (see also T13359). Then, I'll resynchronize instance services for active instances.

Nov 25 2019, 3:57 PM · Phacility, Ops
epriestley added a comment to T13466: AWS instance termination may fail/hang indefinitely.

Instance termination completed after about 20 minutes and all the volumes detached. Since the original instance can be recycled, I'm going to reattach and restart it, and throw away the replacement host.

Nov 25 2019, 3:51 PM · Phacility, Ops
epriestley added a project to T13466: AWS instance termination may fail/hang indefinitely: Phacility.
Nov 25 2019, 3:48 PM · Phacility, Ops

Nov 8 2019

epriestley triaged T13450: Update "Renaming Instances" in Phacility documentation as Wishlist priority.
Nov 8 2019, 2:27 AM · Phacility

Oct 29 2019

epriestley added a comment to T13436: Improve performance of "ssh-auth".

I deployed the --sshd-key %k stuff to secure and it looks like that shaved ~1,000-2000ms off the total cost. The ssh-auth cost has dropped to about 200ms:

Oct 29 2019, 1:47 AM · Diffusion, Phacility, Infrastructure
epriestley added a revision to T13436: Improve performance of "ssh-auth": D20875: Don't use "phutil_hashes_are_identical()" to compare public keys.
Oct 29 2019, 1:31 AM · Diffusion, Phacility, Infrastructure
epriestley added a revision to T13436: Improve performance of "ssh-auth": D20874: Add an optional "--sshd-key" argument to "bin/ssh-auth" for reading "%k" from modern sshd.
Oct 29 2019, 12:49 AM · Diffusion, Phacility, Infrastructure
epriestley added a revision to T13436: Improve performance of "ssh-auth": D20873: Remove the "ssh-auth-key" script.
Oct 29 2019, 12:36 AM · Diffusion, Phacility, Infrastructure
epriestley added a comment to T13436: Improve performance of "ssh-auth".

I saw things hang during deploy and the OpenSSH bug should have been fixed years ago.

Oct 29 2019, 12:20 AM · Diffusion, Phacility, Infrastructure
epriestley added a comment to T13436: Improve performance of "ssh-auth".

I configured ControlMaster on secure.

Oct 29 2019, 12:14 AM · Diffusion, Phacility, Infrastructure

Oct 28 2019

epriestley triaged T13436: Improve performance of "ssh-auth" as Normal priority.
Oct 28 2019, 11:00 PM · Diffusion, Phacility, Infrastructure

Sep 28 2019

epriestley edited the content of Migrating Repository Shards.
Sep 28 2019, 2:59 PM · Phacility

Sep 23 2019

epriestley closed T9456: Evaluate upstream support for third-party build systems, a subtask of T7869: Support CircleCI webhooks for Test results (so that one can run unit tests asynchronously), as Resolved.
Sep 23 2019, 4:28 PM · Phacility

Sep 19 2019

epriestley updated the task description for T9515: Build a "quorum" mechanism and associated UI.
Sep 19 2019, 8:02 PM · Restricted Project, Notifications, Phacility, Auth

Sep 18 2019

epriestley updated the task description for T9515: Build a "quorum" mechanism and associated UI.
Sep 18 2019, 5:18 PM · Restricted Project, Notifications, Phacility, Auth

Sep 4 2019

epriestley added a revision to T13366: Update Phortune to work better with "enterprise" billing/accounts departments: D20783: Update a straggling "getAuthorities()" call in Fund.
Sep 4 2019, 2:11 PM · Phortune, Phacility

Sep 2 2019

epriestley closed T13393: Improve repository shard migration pathway in the shared cluster as Resolved.

The migration in PHI1403 seems to have gone through cleanly. This workflow can continue to improve, but it's in relatively good shape now.

Sep 2 2019, 3:02 PM · Phacility, Infrastructure
epriestley closed T13401: Phacility 2019 Week 35 Deployment Log as Resolved.

Everything here appears to have made it to production cleanly.

Sep 2 2019, 1:07 PM · Phacility
epriestley updated the task description for T13401: Phacility 2019 Week 35 Deployment Log.
Sep 2 2019, 1:05 PM · Phacility
epriestley updated the task description for T13367: Phortune "enterprise" update errata.
Sep 2 2019, 1:04 PM · Phortune, Phacility
epriestley added a revision to T13401: Phacility 2019 Week 35 Deployment Log: Restricted Differential Revision.
Sep 2 2019, 1:04 PM · Phacility
epriestley added a revision to T13393: Improve repository shard migration pathway in the shared cluster: Restricted Differential Revision.
Sep 2 2019, 12:57 PM · Phacility, Infrastructure

Sep 1 2019

epriestley added a revision to T13393: Improve repository shard migration pathway in the shared cluster: Restricted Differential Revision.
Sep 1 2019, 8:12 PM · Phacility, Infrastructure
epriestley added a revision to T13393: Improve repository shard migration pathway in the shared cluster: Restricted Differential Revision.
Sep 1 2019, 8:08 PM · Phacility, Infrastructure
epriestley added a comment to T13393: Improve repository shard migration pathway in the shared cluster.

Anecdotally from the last time around, gzipping the tarball didn't really do much. Possibly, this might more broadly imply that we'd be better off not compressing repository backups.

Sep 1 2019, 7:26 PM · Phacility, Infrastructure
epriestley added a comment to T13393: Improve repository shard migration pathway in the shared cluster.

I believe I've moved "core/" from "instances.queryinstances" and sequenced all the followup changes properly, now, and that the only remaining piece is glue.

Sep 1 2019, 5:48 PM · Phacility, Infrastructure
epriestley added a revision to T13393: Improve repository shard migration pathway in the shared cluster: Restricted Differential Revision.
Sep 1 2019, 5:31 PM · Phacility, Infrastructure
epriestley added a revision to T13393: Improve repository shard migration pathway in the shared cluster: Restricted Differential Revision.
Sep 1 2019, 5:29 PM · Phacility, Infrastructure
epriestley added a revision to T13393: Improve repository shard migration pathway in the shared cluster: Restricted Differential Revision.
Sep 1 2019, 5:19 PM · Phacility, Infrastructure
epriestley added a revision to T13393: Improve repository shard migration pathway in the shared cluster: Restricted Differential Revision.
Sep 1 2019, 5:12 PM · Phacility, Infrastructure
epriestley added a revision to T13393: Improve repository shard migration pathway in the shared cluster: Restricted Differential Revision.
Sep 1 2019, 5:09 PM · Phacility, Infrastructure
epriestley added a revision to T13393: Improve repository shard migration pathway in the shared cluster: Restricted Differential Revision.
Sep 1 2019, 4:51 PM · Phacility, Infrastructure
epriestley added a comment to T13399: Remove "instances.queryinstances" Conduit API method.

SyncWorkflow also depends on creatorPHID to synchronize the initial administrator account.

Sep 1 2019, 4:48 PM · Phacility
epriestley added a revision to T13393: Improve repository shard migration pathway in the shared cluster: Restricted Differential Revision.
Sep 1 2019, 4:44 PM · Phacility, Infrastructure
epriestley added a revision to T13393: Improve repository shard migration pathway in the shared cluster: Restricted Differential Revision.
Sep 1 2019, 4:41 PM · Phacility, Infrastructure
epriestley added a revision to T13393: Improve repository shard migration pathway in the shared cluster: Restricted Differential Revision.
Sep 1 2019, 12:35 AM · Phacility, Infrastructure