Page MenuHomePhabricator

PhacilityOrganization
ActivePublic

Details

Description

Phabricator SaaS

Recent Activity

Oct 26 2022

epriestley closed T13686: Disable Ubuntu unattended upgrades as Resolved.

I patched and partially deployed this in early August. Another unattended MySQL upgrade went out on Monday night, also didn't restart MySQL on affected hosts, and caused some downtime on hosts that didn't have the patch (to "disable unattended upgrades"). I've now deployed this everywhere, and am presuming this is fixed until evidence arises to the contrary.

Oct 26 2022, 7:53 PM · Phacility

Oct 25 2022

epriestley added a comment to T13686: Disable Ubuntu unattended upgrades.

See also PHI2219, PHI2220.

Oct 25 2022, 12:17 PM · Phacility

Jul 29 2022

epriestley triaged T13686: Disable Ubuntu unattended upgrades as Normal priority.
Jul 29 2022, 12:16 PM · Phacility

Apr 20 2022

epriestley lowered the priority of T11132: New Phabricator NUX from High to Wishlist.
Apr 20 2022, 10:43 PM · Design, Phacility, NUX
epriestley lowered the priority of T11456: Don't lose user in NUX flow because of Timezone issues from High to Wishlist.
Apr 20 2022, 10:43 PM · Design, Phacility, NUX
epriestley closed T10847: 30GB Phacility instance caused a series of cascading failures which left web services unreachable as Resolved.

There's nothing particularly useful or actionable here now, so closing it out. (I believe this was the most severe incident Phacility ever experienced while actively maintained.)

Apr 20 2022, 10:43 PM · Ops, Phacility
epriestley closed T12610: Audit behavior of LB healthchecks against *.phacility.com and secure.phabricator.com as Wontfix.

This hasn't caused any more problems in like 4 years, so I guess it's kind of whatever.

Apr 20 2022, 10:30 PM · Ops, Phacility
epriestley closed T13537: Support local port forwarding through Phacility cluster bastion hosts as Wontfix.

This isn't really resolved, but almost certainly does not make sense to pursue given the Phacility wind-down.

Apr 20 2022, 9:09 PM · Phacility
epriestley closed T13630: Move Phacility provisioning to Piledriver as Resolved.

Almost every host currently in production was provisioned with Piledriver and things have been stable for quite a while, so I'm calling this resolved. See elsewhere for issues with Ubuntu20, mail, etc.

Apr 20 2022, 7:10 PM · Almanac, Infrastructure, Phacility
epriestley closed T13641: Support "Disabled" devices in Almanac, a subtask of T13630: Move Phacility provisioning to Piledriver, as Resolved.
Apr 20 2022, 6:39 PM · Almanac, Infrastructure, Phacility
epriestley closed T13646: Add "E" to "variables_order" in Phacility environments as Resolved.

Moved the rest of this to T13640.

Apr 20 2022, 6:36 PM · Phacility, Infrastructure

Apr 19 2022

epriestley closed T13661: Give Phame configurable interact policies as Resolved.

I deployed this and it seems to be working properly.

Apr 19 2022, 9:07 PM · Phacility, Phame
epriestley closed T13674: Ubuntu20 systemd restart script does not reliably execute on Ubuntu20/m4 chassis hosts as Resolved.

Hey, it worked once. Good enough for me!

Apr 19 2022, 5:55 PM · Phacility
epriestley added a comment to T13674: Ubuntu20 systemd restart script does not reliably execute on Ubuntu20/m4 chassis hosts.

No dice. We need bin/upgrade to run before mysql because it has to mount the data volume. So now I'm trying this:

Apr 19 2022, 5:48 PM · Phacility
epriestley added a comment to T13674: Ubuntu20 systemd restart script does not reliably execute on Ubuntu20/m4 chassis hosts.

... service ... start rather than service ... restart ...

Apr 19 2022, 5:27 PM · Phacility
epriestley added a comment to T13674: Ubuntu20 systemd restart script does not reliably execute on Ubuntu20/m4 chassis hosts.

...probably tested...

Apr 19 2022, 5:22 PM · Phacility
epriestley triaged T13674: Ubuntu20 systemd restart script does not reliably execute on Ubuntu20/m4 chassis hosts as Low priority.
Apr 19 2022, 4:30 PM · Phacility

Apr 1 2022

epriestley added a comment to T13661: Give Phame configurable interact policies.

This has some rough edges that I'm not going to deal with for now:

Apr 1 2022, 7:52 PM · Phacility, Phame
epriestley added a revision to T13661: Give Phame configurable interact policies: D21755: Improve some UI/language for Phame posts when viewer doesn't have CAN_INTERACT.
Apr 1 2022, 7:49 PM · Phacility, Phame
epriestley added a revision to T13661: Give Phame configurable interact policies: D21754: Give Phame blog posts configurable interact policies, with a default policy of "Same as Blog".
Apr 1 2022, 7:41 PM · Phacility, Phame
epriestley added a revision to T13661: Give Phame configurable interact policies: D21753: Remove unused "MARKUP_FIELD_SUMMARY" for Phame posts.
Apr 1 2022, 7:16 PM · Phacility, Phame
epriestley added a revision to T13661: Give Phame configurable interact policies: D21751: Give Phame blogs mutable interact policies.
Apr 1 2022, 7:05 PM · Phacility, Phame
epriestley added a revision to T13661: Give Phame configurable interact policies: D21750: Fix double-bordered breadcrumbs in Phame blogs.
Apr 1 2022, 6:48 PM · Phacility, Phame
epriestley added a revision to T13661: Give Phame configurable interact policies: D21749: Remove ancient Remarkup constants from Phame and Maniphest.
Apr 1 2022, 6:46 PM · Phacility, Phame
epriestley added a revision to T13661: Give Phame configurable interact policies: D21748: Make Phame blog policies non-nullable.
Apr 1 2022, 6:43 PM · Phacility, Phame

Dec 19 2021

epriestley closed T11230: Phacility: Private Clusters as Wontfix.

See T12847. All the technical parts of this are now solved except for billing, but since Phacility is winding down I no longer plan to pursue it.

Dec 19 2021, 8:45 PM · Phacility
epriestley closed T8688: Attach and initialize backup volumes during `remote deploy` workflow as Resolved.

I resolved this in rCORE320b2854.

Dec 19 2021, 8:43 PM · Phacility
epriestley closed T12847: A Pathway Towards Private Clusters as Wontfix.

After T13630:

Dec 19 2021, 8:39 PM · Plans, Ops, Phacility
epriestley closed T12847: A Pathway Towards Private Clusters, a subtask of T11230: Phacility: Private Clusters, as Wontfix.
Dec 19 2021, 8:39 PM · Phacility
epriestley closed T13601: Support "SCA" / "3D Secure 2" in billing workflows as Wontfix.

Only one instance was impacted by this and I just credited them until 2099. I don't currently expect to pursue this.

Dec 19 2021, 8:26 PM · Phacility, Phortune
epriestley closed T13610: Support per-node billing for hosted Phacility instances as Wontfix.

I no longer expect to pursue this.

Dec 19 2021, 8:25 PM · Phortune, Phacility
epriestley closed T13618: When a Phacility "rbak" device does not exist, backups can run twice and converge to a "successful" but inconsistent state as Wontfix.
  • Hosts in the repo class are now build by Piledriver (see T13630), which automatically creates the rbak device entries, so this error isn't likely to occur again.
  • I also don't expect to launch any more hosts.
Dec 19 2021, 8:25 PM · Phacility
epriestley closed T13654: Wind Down Phacility Operations as Resolved.

I compacted secure onto new hardware (T13671) and shut down saux001 ("Land Revision") and sbuild001 (Harbormaster remote builds). I think all the remaining work is covered under T13630 (largely, just a handful of large database migrations remain).

Dec 19 2021, 8:23 PM · Phacility
epriestley closed T13671: Merge "secure003.phacility.net" into "secure001.phacility.net", then migrate to "m4.large" hardware as Resolved.

I just swapped configs over without merging the LBs, since it wasn't immediately obvious to me what the Application vs Classic state of the world is and swapping was good enough.

Dec 19 2021, 7:35 PM · Phacility
epriestley added a comment to T13671: Merge "secure003.phacility.net" into "secure001.phacility.net", then migrate to "m4.large" hardware.

The aphlict/notify stuff still needs to be tweaked. I think the snlb + slb setup can be merged into a single slb with "TCP (Secure)" forwarding now.

Dec 19 2021, 4:19 AM · Phacility
epriestley added a comment to T13671: Merge "secure003.phacility.net" into "secure001.phacility.net", then migrate to "m4.large" hardware.

Databases are moved and secure is out of read-only mode. I'm going to adjust repository configuration, then I should be able to tear down secure001.

Dec 19 2021, 4:11 AM · Phacility
epriestley added a comment to T13671: Merge "secure003.phacility.net" into "secure001.phacility.net", then migrate to "m4.large" hardware.

I'm going to put secure back into read-only mode now and move the databases to the new host.

Dec 19 2021, 12:08 AM · Phacility
epriestley added a comment to T13671: Merge "secure003.phacility.net" into "secure001.phacility.net", then migrate to "m4.large" hardware.

I brought up the new host and pointed the slb001 load balancer at it. The database is still on the old host, and the new host doesn't have repositories yet, but the basics seem to be working.

Dec 19 2021, 12:07 AM · Phacility

Dec 18 2021

epriestley added a comment to T13671: Merge "secure003.phacility.net" into "secure001.phacility.net", then migrate to "m4.large" hardware.

Merging 003 into 001 worked fine with a few expected tricks (e.g., when secure is in read-only mode, you can't push a change to take it out of read-only mode, since pushing is a write). Next up is launching a modern m4.large secure-pool host and then migrating the data.

Dec 18 2021, 7:57 PM · Phacility
epriestley added a comment to T13671: Merge "secure003.phacility.net" into "secure001.phacility.net", then migrate to "m4.large" hardware.

I'm putting secure into read-only mode now, with the intent of completing steps 1-5 above.

Dec 18 2021, 7:13 PM · Phacility
epriestley added a revision to T13671: Merge "secure003.phacility.net" into "secure001.phacility.net", then migrate to "m4.large" hardware: D21745: Add a "--database <name> ..." flag to "bin/storage dump".
Dec 18 2021, 7:06 PM · Phacility

Dec 17 2021

epriestley triaged T13671: Merge "secure003.phacility.net" into "secure001.phacility.net", then migrate to "m4.large" hardware as Low priority.
Dec 17 2021, 7:32 PM · Phacility

Dec 16 2021

epriestley lowered the priority of T13655: Provide a formal "destroyed" status for Phacility instances from Normal to Wishlist.

It would still be nice to have this from a completeness/correctness perspective, but other changes have made it less valuable:

Dec 16 2021, 2:58 PM · Phacility

Dec 11 2021

epriestley added a comment to T13630: Move Phacility provisioning to Piledriver.

I put all the database migration stuff everywhere and it appears stable. I'm hooking up Postmark as an outbound pathway now. If I get that working, I'll let it sit for a while and start migrating databases.

Dec 11 2021, 5:43 PM · Almanac, Infrastructure, Phacility

Dec 10 2021

epriestley added a comment to T13630: Move Phacility provisioning to Piledriver.

Finally, there are other some MySQL version issues which can be avoided with:

Dec 10 2021, 6:22 PM · Almanac, Infrastructure, Phacility

Dec 9 2021

epriestley added a comment to T13630: Move Phacility provisioning to Piledriver.

The new core/ support for the API is partially deployed; the new services/ support isn't anywhere yet.

Dec 9 2021, 11:13 PM · Almanac, Infrastructure, Phacility

Dec 8 2021

epriestley edited the content of Migrating Repository Shards.
Dec 8 2021, 9:39 PM · Phacility

Dec 4 2021

epriestley added a comment to T13630: Move Phacility provisioning to Piledriver.

The latest version of Phabricator itself is everywhere.

Dec 4 2021, 11:46 PM · Almanac, Infrastructure, Phacility
epriestley added a comment to T13630: Move Phacility provisioning to Piledriver.

I'm going to hold it until the weekend and try deploying then if things look calm on my end.

Dec 4 2021, 9:23 PM · Almanac, Infrastructure, Phacility

Dec 2 2021

epriestley added a comment to T13037: An attacker gained staff access to Mailgun and was able to read customer API keys.

I'm satisfied that we aren't violating our commitment to our customers by continuing to use Mailgun as a service provider...

Dec 2 2021, 10:39 PM · Phacility, Security, Mail