Drydock: Allocator

Fixed an allocator bug casued by treating new and existing resources differently.
Fixed an allocator bug caused by not counting right.

We've been hosting upstream builds through Harbormaster + Drydock for a couple of weeks now, but were seeing some issues with the allocator making a mistake and deadlocking a handful of jobs. I fixed two issues here: we were treating newly created resources a little differently from previously existing resources when they should be treated the same way, and we had a simple bug with a poorly written SQL query. These issues were conceptually capable of causing the problems we observed, were locally reproducible before the patches and not reproducible after the patches, and the problems have stopped occurring in production after applying the fixes.

While it's still a little early to declare Drydock stable, it has now been running in a production configuration for about a week with no unexpected behaviors.

Drydock: Authorizations

Built a new "Authorizations" mechanism.

Drydock allows users to make isolation vs performance tradeoffs: your tests can be running in a few seconds if your use case can accept that malicious or poorly written tests can theoretically interact with one another. In many reasonable scenarios, this is acceptable and the tradeoff is desirable.

However, there are plenty of cases where this isn't universal, or you may want to make some less aggressive tradeoff. Maybe, it's OK if tests on unreviewed code can interfere with tests on other unreviewed code, but not OK if they can interfere with builds or merges. Or where you just don't want your iPhone and Android builds sharing hardware, even if the hardware is technically indistinguishable except for the name. You can find some additional discussion of this in Drydock User Guide: Security.

The new "Authorizations" mechanism is an explicit whitelist for which applications and use cases are permitted to use which classes of resources in Drydock. For example, you can make sure that only your Android builds are running on your Android build tier. See T9519 for discussion.

We've had this in production for about a week with two separate build tiers (high-trust saux, low-trust sbuild). The behavior appears correct and setting up the configuration feels fairly reasonable, at least for now and in simple deployment scenarios like this. I anticipate expanding this mechanism in the future, but the fundamentals feel solid for now.

Drydock: Repository Operations

Built a new "RepositoryOperation" mechanism.

There are several use cases for performing "repository automation" operations (repository writes like commits, merges, and cherry-picks) within Phabricator. The first case we plan to support is a "Land Revision" button in Differential, which will work like the "Merge Pull Request" button on GitHub.

The core infrastructure component for this class of operation is Drydock, which manages working copies. Building on top of Drydock makes these operations scalable: we can perform a large number of independent operations on the same repository simultaneously, and you can add more hardware to get more parallelism. Now that Drydock is stabilizing, I've started building out DrydockRepositoryOperation. This is a simple indirection layer which allows applications which want to perform operations in this vein to coordinate and mostly write only specialized code particular to the operation they are performing.

The first concrete operation (DrydockLandRepositoryOperation) provides a "Land Revision" action from revisions, and is now technically functional, although the implementation is very rough. We've successfully used it to land a handful of revisions in the upstream. You can follow T182 for details.

Reporting Code Coverage

There is a walkthrough for a use case involving reporting code coverage in T9529. While the particulars of that issue are unlikely to apply to any specific problem, it may broadly be useful in understanding how to report data (and, particularly, coverage) into Harbormaster, and generally how to integrate Phabricator with third-party build and test systems where the model doesn't quite match with Harbormaster's model.

Harbormaster: CircleCI

CircleCI is an hosted external build system which we've seen some interest in from Phabricator users and Phacility customers.

We built most of a CircleCI integration for Harbormaster this week, but are missing a small (but critical) piece to complete it: there isn't currently any apparent way to use CircleCI to build a tag which is not the ancestor of any branch head. I hope to find a way forward and be able to ship this integration in the relatively near future. You can follow T9456 for updates.

Pygments Performance

One install ran into a performance problem with Pygments (see T9566). You can find a great volume of additional discussion in D14273. The upshot is:

• If you have an older version of Pygments, upgrading may improve performance substantially.
• At HEAD, JSON highlighting in Phabricator should be dramatically faster (this change didn't make it into stable this week).
• At tip, Pygments has much more reasonable performance on large single-line inputs, although this fix is not yet available in a release.
• We will recommend a Pygments upgrade for installs with an old version in the near future (see T9587).

Upcoming

Last week I hoped to complete Drydock Authorizations, stabilize Drydock, and prototype "Land Revision". Authorizations and stabilization went cleanly, but "Land Revision" isn't quite as far along as I wanted. It's technically a working prototype, but still in very rough shape, and may still be about ~2 weeks away for ambitious installs. This remains my focus heading into the next week.

ApplicationEditor (T9132), Nuance (T8783), and updates to Projects (T9378) and Phriction (T9379) are still the plans for the remainder of the year.