I'm going to try writing some more casual posts about engineering status to compliment the existing Changelog. The changelog is relatively dry (at least, for us) and the primary goal of the document is to make sure administrators are aware of migrations and security problems which may impact the complexity or timeline of upgrades. I can't fill it with too much nonsense without detracting from its ability to accomplish that goal. Instead, when changes come with context that's worth discussing or anecdotes that are worth retelling, I'll try providing them here.

As a secondary goal, this may provide some sense of what's happening day-to-day and make our short term plans more observable, as a compliment to the more static and longer term planning documents we already publish (Planning, Roadmap. Starmap).

Deeply Nested Blockquotes

Fixed an issue where deeply nested blockquotes could generate far too much markup when rendered in text mode.

Specifically, the problem was that a user made this comment while requesting changes to a revision:

This is an excellent comment.

Alas, when we rendered this comment in text mode to generate the plain text bodies for email, we did not faithfully preserve the intent of the comment. Instead, we incorrectly doubled the number of newlines in the input as each level of blockquoting was unwrapped.

Although the engine has a hard cutoff at 32 levels (because I did anticipate users trying to blow the stack with an input like >>> ... (millions of blockquotes) ... >>> k), this was still more than enough levels to produce an unreasonably large output which could exhaust available memory in the daemons while generating the mail.

After D14247, we no longer generate an exponentially greater amount of output text while recursively evaluating the blockquote rule.

Drydock and Harbormaster

The Phabricator upstream now self-hosts builds for commits and revisions in Harbormaster + Drydock.

Our recent focus has been on Drydock and Harbormaster, particularly T9252 (Drydock v1) and T9123 (Harbormaster v2).

• Harbormaster is mostly a build/CI system: it runs tests on commits and revisions.
• Drydock is mostly a tool that handles building working copies for other applications so they can do interesting things (like run tests).

The immediate goal in pursuing these changes is for Phabricator to self-host its own build/CI process. This is not especially valuable to anyone on its own (we don't have a strong focus on CI, and end users generally don't particularly care how we run tests), but serves as a reasonable, observable milestone in expanding the capabilities of Harbormaster and Drydock with other long-term goals in mind.

One such goal is a "Merge This" button in Differential, for allowing Phabricator to commit changes directly from the web interface (see T182).

Since earlier this week, we're hosting our own builds. There are still quite a few rough edges, but I believe all the major technical issues have solutions in place or nearly in place now.

We entered the week with two major open technical problems in Drydock. First, when we ran out of room in a resource pool, the allocator would still keep generating requests forever. Second, we needed a way to control which blueprints Drydock would be allowed to use of use to satisfy a request, to prevent untrusted jobs from being handed resources on trusted hosts (and to prevent Drydock from finding other technically correct but confusing/unintended solutions to allocation problems).

The allocator behavior has been reined in (see D14236), although this needs additional refinement and isn't a complete solution. It largely serves as a step to define allocator behavior more concretely in terms of a tradeoff between allocator performance and allocator waste. Having the allocator generate infinite waste (which is what it did at first) is never desirable, and it no longer does, but we can make refinements and expose configuration to further reduce the amount of generated waste and make waste-vs-performance tradeoffs more efficient.

The authorization behavior has been designed, at least roughly, and partially implemented (see T9519, D14251). The initial approach we're using is simple: things that want to use Drydock list the blueprints they want to use explicitly, and those blueprints must authorize the uses. This has the potential to be unreasonably labor-intensive, but it looks like an approach with approximately this level of granularity is minimally required and that we have a lot of design headroom to reduce the amount of labor it needs as problems arise later on.

I expect Drydock and Harbormaster to generally leave focus once authorizations are stable, although they'll continue to move forward in the background (particularly in connection with "Merge This"). We currently see little direct demand for Drydock so it probably won't formally unprototype for at least a few weeks.

Upcoming Plans

In the upcoming week I plan to complete Drydock authorizations and then make as much headway as possible on "Merge This" while continuing to refine Drydock and Harbormaster as we run into issues with them. Ideally, we'll have a functional upstream prototype of "Merge This" by the end of the week, albeit one laced with caveats and probably not in a state that installs can realistically deploy or configure themselves yet. General availability for ambitious installs is probably ~2 weeks away, and availability in the Phacility cluster is probably at least ~3-4 weeks away because there are additional concerns with instance isolation and Drydock almost certainly needs to unprototype first.

Beyond this, ApplicationEditor (T9132), Nuance (T8783), and updates to Projects (T9378) and Phriction (T9379) are looming on the horizon. For now, I'm cautiously optimistic that we may be able to at least touch on most of that by the end of the year. You can review Prioritized for some sense of where we anticipate spending time in the near future.