Page MenuHomePhabricator

PhacilityOrganization
ActivePublic

Details

Description

Phabricator SaaS

Recent Activity

May 28 2020

epriestley added a comment to T13542: Rebalance Phacility instances into a private subnet.

Piledriver would also benefit from having some functional equivalent of destroying an Almanac resource. This can be implemented as a piledriver.destroyed property, but a formal disabled state would be cleaner. PHI1331 is vaguely related.

May 28 2020, 5:53 PM · Phacility
epriestley added a comment to T13542: Rebalance Phacility instances into a private subnet.
  • When Piledriver destroys a resource pile, it's helpful if it can read the entire authoritative state from sources by using only a pile ID.
    • EC2 can do this with "DescribeTags".
    • Almanac currently can not. Almanac types should support searching by property value.
      • This could be directly on almanac.*.search.
      • Or this could be generic, via T12799.
May 28 2020, 3:23 PM · Phacility

May 26 2020

epriestley closed T13467: Expand the "ddata003" volume as Resolved.

Effectively mooted by T13542.

May 26 2020, 8:32 PM · Phacility
epriestley added a comment to T13542: Rebalance Phacility instances into a private subnet.

I've written some Terraform-class tooling which can likely automate all the actual hardware allocations. This needs some more work, but I believe the tricky stuff (mostly: representing resources and allowing templating to reference resources which haven't been built yet) is at least working.

May 26 2020, 8:31 PM · Phacility
epriestley added a comment to T13542: Rebalance Phacility instances into a private subnet.

Subnet/NAT issues in T12816.

May 26 2020, 8:18 PM · Phacility
epriestley closed T13076: Plans: Phacility cluster caching, renaming, and rebalance/compaction as Resolved.

Continued in T13542.

May 26 2020, 8:13 PM · Plans, Ops, Infrastructure, Phacility
epriestley closed T12801: Simplify Almanac services in the Phacility production cluster as Resolved.

The major offender here (services per instance) was fixed by updating caching, and I destroyed all the old services. This is perhaps spiritually continued in T13542.

May 26 2020, 8:11 PM · Almanac, Ops, Phacility
epriestley updated the task description for T13542: Rebalance Phacility instances into a private subnet.
May 26 2020, 8:10 PM · Phacility
epriestley closed T12999: Replace cluster magnetic volumes with SSD volumes as Resolved.

Continued in T13542.

May 26 2020, 8:10 PM · Phacility, Ops
epriestley closed T12856: Evaluate various "infrastructure-as-code" products as Resolved.

Continued in T13542. I wrote a Terraform/CloudFormation-style service in PHP over the last couple of days.

May 26 2020, 8:07 PM · Ops, Phacility
epriestley closed T12816: Setup NAT for the primary Phacility cluster as Resolved.

Continued in T13542.

May 26 2020, 8:03 PM · Ops, Phacility
epriestley updated the task description for T13542: Rebalance Phacility instances into a private subnet.
May 26 2020, 8:03 PM · Phacility
epriestley triaged T13542: Rebalance Phacility instances into a private subnet as Low priority.
May 26 2020, 8:02 PM · Phacility

May 22 2020

epriestley added a comment to T13111: Periodically run `git prune` on Git working copies.

Since many of these options probably don't have "right answers", I'm trying this reasonable-seeming variation on some repositories which seem like they'll benefit from a repack:

May 22 2020, 3:03 AM · Phacility, Diffusion

May 21 2020

epriestley added a comment to T11760: Phacility Cluster: Ad-Hoc Query Log.

See PHI1748. I ran a query against a subset of instances to determine how widespread usage of "Dark Mode" is, to help inform a decision to either implement the mode properly (see T12311) or remove the mode. The query was of this form:

May 21 2020, 8:08 PM · Phacility

May 18 2020

epriestley added a comment to T7149: Allow users to import data into a new Phacility instance.

See T13537 for a subtle issue where digestWithNamedKey() keys were cached in APCu on the web tier. Importing instance data may require restarting the web tier until the import process can either dump these caches or version them (versioning may be easier).

May 18 2020, 5:38 PM · Phacility
epriestley added a comment to T13537: Support local port forwarding through Phacility cluster bastion hosts.

So I'm going to turn web off and on again and see if that fixes things; my expectation is that it will.

May 18 2020, 5:36 PM · Phacility
epriestley added a comment to T13537: Support local port forwarding through Phacility cluster bastion hosts.

The specific issue I'm trying to debug is fairly bizarre.

May 18 2020, 5:35 PM · Phacility
epriestley updated the task description for T13537: Support local port forwarding through Phacility cluster bastion hosts.
May 18 2020, 4:59 PM · Phacility
epriestley updated the task description for T13537: Support local port forwarding through Phacility cluster bastion hosts.
May 18 2020, 4:56 PM · Phacility
epriestley updated the task description for T13537: Support local port forwarding through Phacility cluster bastion hosts.
May 18 2020, 4:47 PM · Phacility
epriestley updated the task description for T13537: Support local port forwarding through Phacility cluster bastion hosts.
May 18 2020, 4:37 PM · Phacility
epriestley triaged T13537: Support local port forwarding through Phacility cluster bastion hosts as Low priority.
May 18 2020, 4:33 PM · Phacility

May 15 2020

epriestley added a revision to T13529: Phacility domain rename errata and SSH user identity: D21259: Remove code which overrides "diffusion.ssh-username" when instanced.
May 15 2020, 1:31 PM · Phacility

May 13 2020

epriestley updated the task description for T13229: On Third-Party Integrations.
May 13 2020, 12:01 AM · Phacility

May 12 2020

epriestley added a comment to T13529: Phacility domain rename errata and SSH user identity.

The SSH username change wasn't sufficient because there's a hard-coded piece of logic to select the username by instance name:

May 12 2020, 6:20 PM · Phacility

May 8 2020

epriestley closed T13531: Phacility cluster account/service sync issues (account identifiers, instance refs) as Resolved.
May 8 2020, 6:49 PM · Phacility

May 5 2020

epriestley added a comment to T13531: Phacility cluster account/service sync issues (account identifiers, instance refs).

There was a related AccountIdentifier issue with InstancesShadowUserQuery: we loaded shadow users based on accountID, but this is no longer consistently populated after T13493.

May 5 2020, 6:32 PM · Phacility
epriestley added a comment to T13531: Phacility cluster account/service sync issues (account identifiers, instance refs).

AccountIdentifiers do not sync during setup after T13493.

May 5 2020, 4:18 PM · Phacility
epriestley added a comment to T13531: Phacility cluster account/service sync issues (account identifiers, instance refs).

bin/services sync --instance X exits with no error if X does not exist.

May 5 2020, 4:15 PM · Phacility
epriestley triaged T13531: Phacility cluster account/service sync issues (account identifiers, instance refs) as Normal priority.
May 5 2020, 3:10 PM · Phacility

May 1 2020

epriestley renamed T13529: Phacility domain rename errata and SSH user identity from Phacility domain errata to Phacility domain rename errata and SSH user identity.
May 1 2020, 5:25 PM · Phacility
epriestley updated the task description for T13529: Phacility domain rename errata and SSH user identity.
May 1 2020, 4:39 PM · Phacility
epriestley triaged T13529: Phacility domain rename errata and SSH user identity as Wishlist priority.
May 1 2020, 4:34 PM · Phacility

Apr 17 2020

epriestley updated the task description for T13178: Plans: Support Pact Competitive Update.
Apr 17 2020, 7:04 PM · Plans, Support, Phacility

Mar 2 2020

epriestley added a comment to T13111: Periodically run `git prune` on Git working copies.

Does using --max-pack-size to reduce the maximum packfile size really let Git "checkpoint" after each packfile, so the process is effectively resumable?

Mar 2 2020, 4:36 PM · Phacility, Diffusion
epriestley added a comment to T13111: Periodically run `git prune` on Git working copies.

PHI1655 identifies a specific case where enormous packfiles may create problems:

Mar 2 2020, 3:56 PM · Phacility, Diffusion
epriestley updated the task description for T13156: Plans: Improve Phacility UI for managing instance managers and cards.
Mar 2 2020, 2:30 PM · Plans, Phacility

Feb 3 2020

epriestley closed T13483: (2020 Week 5) Restart AWS host db001 (i-3a9b99f2) as Resolved.

Both of these hosts restarted cleanly.

Feb 3 2020, 2:19 PM · Phacility, Ops
epriestley added a comment to T13483: (2020 Week 5) Restart AWS host db001 (i-3a9b99f2).

AWS is also rebooting web007.

Feb 3 2020, 1:22 PM · Phacility, Ops

Jan 30 2020

epriestley closed T13450: Update "Renaming Instances" in Phacility documentation as Resolved by committing Restricted Diffusion Commit.
Jan 30 2020, 6:37 PM · Phacility
epriestley triaged T13483: (2020 Week 5) Restart AWS host db001 (i-3a9b99f2) as Normal priority.
Jan 30 2020, 6:36 PM · Phacility, Ops
epriestley added a revision to T13450: Update "Renaming Instances" in Phacility documentation: Restricted Differential Revision.
Jan 30 2020, 5:51 PM · Phacility

Jan 21 2020

epriestley added a comment to T13111: Periodically run `git prune` on Git working copies.

The logic here appears to be that gc.auto is set to some value (by default: 6,700). If the number of loose objects exceeds this threshold (technically, if the number of loose objects in objects/17/ is more than 1/256th of this value), it triggers a repack (in a comment, git repack -d -l).

Jan 21 2020, 6:00 PM · Phacility, Diffusion
epriestley added a comment to T13111: Periodically run `git prune` on Git working copies.

See PHI1613, where an install hit this warning (and resolved it by running git prune):

Jan 21 2020, 5:13 PM · Phacility, Diffusion

Jan 15 2020

epriestley closed T13477: (2020 Week 3) Restart AWS host db025 (i-05bc80634586ef7a0) as Resolved.

This went through cleanly.

Jan 15 2020, 4:19 PM · Ops, Phacility
epriestley triaged T13477: (2020 Week 3) Restart AWS host db025 (i-05bc80634586ef7a0) as Low priority.
Jan 15 2020, 2:36 AM · Ops, Phacility

Nov 26 2019

epriestley triaged T13467: Expand the "ddata003" volume as Low priority.
Nov 26 2019, 6:00 PM · Phacility
epriestley added a comment to T13466: AWS instance termination may fail/hang indefinitely.

there is no way to bin/host query against the set of instances using a particular repository shard service

Nov 26 2019, 12:14 AM · Phacility, Ops
epriestley added a revision to T13466: AWS instance termination may fail/hang indefinitely: Restricted Differential Revision.
Nov 26 2019, 12:09 AM · Phacility, Ops