It's running out of disk space because Drydock isn't cleaning up the working copies and they're consuming all the disk space.

Because if I don't clean it up, then every build from then on will fail until I fix it?

Yes, it's the breadcrumbs.

fyi my Phabricator instance currently has 80 working copy resources open

This usually happens when rebasing an upstream repository after a significant amount of time (like six months). When this happens, there'll be hundreds of builds all running at the same time, which can result in hundreds of working copies for the same repository.

Because I have a lot of repositories, and there are multiple copies of each repository (because things are running in parallel), I will run out of disk space if I have multiple full copies of every repository in Phabricator kept on the build hosts for all time.

fyi I ended up implementing Protobuild.exe --build and a bunch of other functionality (like support for build scripts), which negated the need for having separate build plans for each platform

Oh that was the task I was looking for before making this one.

The way I'm getting around this now is I added a Herald rule "Commit message does not contain [experimental]". Then when pushing to non-master branches we add "[experimental]" to the commit message. When we want to put it on master, we squash and amend to remove the text and then push.

Because the publish step isn't run command. The publishing step is actually 8 other "Run Build plan" steps; these in turn have run command, but there's about 10 run command steps (these build plans clone a repository, update the binary artifact in them, then commit and push).

Also FYI the way we are working around this issue now is whenever someone wants to push to a non-master branch, they go into herald and disable the herald rule, push to their branch, then turn the rule back on again. When they want to push to master, they amend the commit to update the timestamps (which changes the hash), and then they push to master (which imports as a different commit).

Neither of those tools let me use my own hardware, which is necessary for my use case. In actuality, GitHub has nothing to do with it, it just happens to be the authoritative repository instead of Phabricator. I still have the same problem regardless of whether or not I make Phabricator the authoritative repository.

Based on my memory, I think it's possible that exception is the cause of the issue; when it occurs, the exception ends the current worker task with a permanent failure, and the task won't be revisited. Thus the build remains pending because the worker that would have performed the next update failed to do so.

At that scale, is there any reason to not use Phabricator's repository hosting instead of GitLab? You can mirror repositories from Phabricator to GitLab if you still need some GitLab functionality (see the Mirrors option under Edit Repository).

@epriestley I have a reasonable implementation of this working on the new Drydock architecture. I want to pursue this before EC2 so we can ensure that all of the locking mechanisms are working correctly before we use the new Drydock on something that potentially costs money (and in the event that Drydock goes haywire, may cost thousands of dollars). KVM offers a safe alternative, because if Drydock goes haywire, the worst that will happen is that it won't be able to allocate new VMs on the KVM host due to out of memory errors.

That is an issue we've also encountered; hopefully it will be fixed when the staging area of repositories are included inline on a repository instead of being separate (i.e. they use the same URL, but staging is just a set of tags / branches that commits are pushed to).

I know of at least one open source project that is planning to use this heavily, so in order to keep my fork on my phabricator instance (instead of moving it to GitHub where the original is based), phabricator will need to support LFS.

Keep in mind those test releases will be based off diffs, not commits in that model, so we almost certainly can't re-use any of the built artifacts since they aren't integrated.

After giving this some more thought, the "I have to update lots of build plans" issue might be negated when Harbormaster supports embedding build plans (not just calling them). When that happens I could define a Lease Windows Host build plan with a Lease Host step with the right configuration, and then just ended that Lease Windows Host build plan where needed.

Modeling "Deploy" as "run a build plan" seems very under-powered for real deploy processes. But maybe adding a finite number of deploy actions which are each build plans ("Deploy to staging", "deploy to limited production") would be good enough.

Also there was some mention elsewhere about "meta repositories" (I think I brought it up on the discussion of working copies), which would be a "repository" that tracks other repository branches. Each of the tracked repositories would be a folder when the working copy is checked out, but primarily this model gives us a unique ID for the combination of repositories, and would allow buildables to be formed as sets of repositories. If we built that mechanism, it could be re-used in the release app when those releases consists of multiple components or repositories.

In T9530#139478, @avivey wrote:

"release is more than one repository" is a purely theoretical use case, because I was brainstorming; I don't actually want to handle it right now.

(give me a heads up before this task is made public so I can remove
screenshots)

Where I can, I intend to push in the direction of immutable releases: In case of bug, abort the release and start a new one with a new number. I think the costs of throwing away a number are [almost always] negligible vs the ops of multiple binaries with the same number.

Also to add to that:

So I wrote a release management tool for PageUp People called Pipeline; it's written as an extension of Phabricator, although it runs against an older version (before the Drydock / Harbormaster changes). Pipeline can be found at https://github.com/PageUpPeopleOrg/phabricator-pipeline. Unfortunately it doesn't look like we're publishing the patched version of Phabricator anywhere.

Protobuild is written in C#; it could abstract the build command to call, but I've avoided doing that because I don't want to hide this invocation. Too many build systems wrap commands which then wrap other commands and I always find it painful to pull them apart again to find out what's going on. So instead, Protobuild just generates the input solution files which can then be opened in any IDE or any build tool on the given platform.

I think largely the issue with that is that in this case the build system isn't controlled by me or open source. On Windows, the Microsoft-controlled MSBuild tool is used to build code, and on Linux / Mac the open source tools are used to build code. We can't replace those tools with something universal without also sacrificing the ability to open projects in the IDE. Protobuild (the project generator) goes a long way towards generating different project formats for different platforms, but it doesn't abstract the different build tool paths or command invocations.

Like if I wanted to build MonoGame on Windows as a user, from the command-line (which you would almost never do, you'd use the IDE), I would run the same steps as Harbormaster.

Harbormaster also does Protobuild.exe --generate, but it has to run a different command line build tool after that depending on the host platform. Normally the graphical IDE just hides away that command invocation from a regular user.

Most humans would load up MonoGame in their installed IDE, and wouldn't need to worry about the command-line tools to do that build. For humans it's "Protobuild.exe --generate" (it assumes current host platform, but users can pass "iOS" as an argument to target mobile platforms for example), and then opening the .sln file in whatever the default IDE is.

So here's like a snapshot of a few build plans:

@hach-que, are there cases I'm missing where double approval would pose an unreasonably high maintenance burden in your use cases?

@epriestley I just tried to start reimplementing some of my patches on top of HEAD and I've run into a problem:

Yes, it does.

You can't add Conduit tokens without MFA though right?

There are two use-cases here:

We're already impersonating them; we do the standard arc install-certificate process to configure Arcanist, and then we want to automatically configure SSH access afterwards. This is to make configuration of a developer's machine only require one manual step (authenticating with Phabricator for Arcanist).

Because they're developer machines VMs, we want pushes to be recognized as that developer, so they have the correct policies applied.

I believe so? Maybe only some clients do?

403 is access denied, which Git clients treat as "bad credentials". When the repository is disabled, there's no credentials which will make the check pass.

(We also need solution A in some form for our use of Drydock, so if upstream doesn't have that, we'll end up patching an implementation of solution A on top of the new API anyway)

fyi solution (A) is implemented in the revisions I wrote that add custom attributes for blueprints. I'm guessing these will be reasonably straightforward to translate to the new API, since it's mostly just updating the filtering checks (lease attributes matching resource attributes and lease attributes matching blueprint attributes).

The recommended way of running Arcanist under Windows is from either the command prompt or msysgit, which can be installed from https://git-for-windows.github.io/, with the official PHP for Windows binaries installed in some directory and added to your PATH. Guidance on how to set up Arcanist on Windows can be found at https://secure.phabricator.com/book/phabricator/article/arcanist_windows/.

fyi libvirt takes just a little over a minute to spin up a Linux build machine using copy-on-write disks (this includes waiting for SSH to come up and connect)

omg @epriestley if this is an intended pun it has been going over my head for like 2 years now

This requires some changes to infrastructure because we'd need to track per commit whether auto close has fired for it. After that, implementing a build step that looks at the current commit buildable and all of its parents that auto close hasn't fired for,and then process them. Then the user would turn off autoclose at the repository level.

yeah I know this is super minor, but it's also not an obvious thing to discover so I figured I'd document it so you're aware of it

I remember trying to make something implement PhabricatorMarkupInterface a while back for a custom app, and it was so painful and I couldn't figure out what was going on so I ended up just using PhabricatorMarkupOneOff instead, even though it didn't feel like the right way to do things.

Yeah, a lot of the issues mentioned in that Java thread apply to C# as well, although we also have the difficulty that there's no easy way of mapping changed files to individual tests in assemblies.

Just to let you know, I used this panel to find out all of the revisions that were affecting individual Drydock files so that I could merge down changes correctly.

.arcbuild would certainly assist with abstracting a lot of complexity that XUnitTestEngine (and the newer variants) have to deal with, especially if we allowed build steps in .arcbuild to designate artifact files for testing (so that the unit test engine doesn't have to calculate the paths to assemblies ugh)

It might be nice to have some sort of .arcbuild file to describe how the code needs to be built before tests are run (this could then also be used by Harbormaster?)

The other thing is that /p:SkipTestsOnBuild=True is already passed in, so the build scripts can already detect if they're being run as part of arc unit anyway. You could just configure your MSBuild scripts to have something like this to solve the problem:

Yeah, it involves passing /p:Key=Value to MSBuild. It'll be extremely difficult to get right because of Windows escaping rules though.

I know sometimes I've asked in IRC channels "is this a bug or intended?" before filing a task. It feels like a real time channel for asking whether or not something is intended is the best option, rather than filing a task?

I think improving Conpherence is one of the major blockers to dropping IRC. The big things that are missing right now are T7580 and T6868. Right now the general chat in Conpherence feels dead because it doesn't indicate anyone is online to help with something, and this compounds the problem because then no-one adds messages to it and it feels even more dead.

I should mention, sometimes people are CC'd because we want them to know that a task has been created, but this doesn't necessarily mean they want to know when every action happens on that task forever into the future. Having some way of distinguishing between "I need to give someone a heads up about this tasks existence" and "I want to know whenever anything happens on this task" would help here as well.