Harbormaster (and, to a small degree, Drydock) are an ongoing drain on support because they do things users want and they're good enough that experienced users like @hach-que (and several other major installs) can use them to good effect, but still rough and less sophisticated/experienced users hit a lot of issues trying to set them up.

My ideas in the general case are basically:

• Rename "Beta" to "Prototype", or another term which more clearly communicates the very limited usability some of these applications may have (i.e., these are not Gmail "Beta"). I think this would help, but I'm not sure if it would help enough to justify doing the config migration, which is kind of a pain.
• Put a big "ABANDON HOPE" warning on the config screen where you enable these apps. This is trivial so I think it's worth doing regardless.

In this specific case:

• I think this issue is mostly unique to Harbormaster/Drydock and slightly to Phrequent. Other beta applications are almost purely positive as beta (e.g., we get feedback about use cases and a sense of how exciting apps are, and few/no support requests). Because the current system works well for most applications, I don't think we need to make major changes to it (like developing applications in secret for a long time).
• Part of the problem is users writing "Phabricator + Jenkins" blog posts which can fall out of date and may not caveat the availability of upstream support for the integration. We could try to head this off by writing a "Phabricator + Jenkins" documentation article which just says "wait patiently".

We could also move Harbormaster forward so it supports Jenkins easily, but I really don't want to do that until after SAAS. I'm not sure how much work that is, but I think it's not trivial.

I definitely agree we need to do something here. Potentially we could do the same as releeph.installed and use a DB migration to automatically enable harbormaster.installed and drydock.installed for any install that has Beta applications installed?

I'd also be interested to know how many people attempt to integrate Jenkins with Harbormaster because they want to use Jenkins, and how many want to integrate Jenkins because Harbormaster doesn't yet have all the functionality they need (and so they need to pass it off to a more developed build engine). There's no real practical way of finding this out though.

We could also make the migration a little bit smarter and only enable harbormaster.installed / drydock.installed if they have build plans / blueprints configured respectively, which should appropriately disable these applications for a bunch of installs that have Beta on but aren't using the applications yet.

I'd also be interested to know how many people attempt to integrate Jenkins with Harbormaster because they want to use Jenkins

Yeah -- it's possible the number that would switch to Harbormaster fully is significant, which makes a small move forward to Jenkins support less attractive than a larger move forward to a state which could replace Jenkins for many users.

I also think that explicitly supporting external build servers like Jenkins opens a can of worms where it's basically "please add a build step for every other build server in the known universe", which doesn't seem like a very scalable solution (in particular, Jenkins is okay-ish, but there are definitely build servers out there that have an absolutely horrific design / architecture, and with which integration would be extremely difficult).

Obviously I'm of the opinion that getting Harbormaster to a point where it can replace existing build servers is more valuable for people than integrating with an existing build engine, but I could also be misunderstanding how easy it is for people to change this stuff over. I'd expect for large companies it would be far more difficult because of inertia to change, but individuals, open-source projects and small companies would find it potentially easier (i.e. I expect the largest difficult is making the decision to change, rather than changing over the build plans).

Having used Jenkins, Buildbot, CruiseControl.NET, Bamboo and a few others over the course of developing software both personally and at work, I don't believe anyone actually uses these build engines because they're nice to use (almost every build server transition I've seen comes down to "well, it's not as bad as the previous one"). While Harbormaster isn't exactly a shining beacon of usability at the moment, many existing build engines either suffer performance issues, are quite heavy in terms of resource usage, are poorly architected, are expensive, don't scale and / or are just downright convoluted to configure. I think with the direction that Harbormaster is heading, it will be able to address a significant number of those problems and we'll see people opting to change their build server to Harbormaster rather than integrate with their existing one.

Is this warning strong enough?

Generally agreed on all the Harbormaster stuff. I see Harbormaster as the primary build engine in most cases in the long term and want that to be the use case we prioritize, with a few quality-of-life bindings for other common build platforms (basically, Jenkins, at this point) and a "here's how you can build your own that we won't support" for everything else.

IMO, most of the beta applications are somewhat useable and probably not a significant drain on support resources.

Harbormaster and Drydock are "more beta" than other apps... Could we label these as "alpha" perhaps? And have separate configuration for phabricator.show-alpha-applications and phabricator.show-beta-applications`?

Ideally, I don't want to add more configuration if we can help it, or do anything which requires us to guess beforehand whether an application will be a support drain or not.

In T6084#8, @hach-que wrote:

...
Having used Jenkins, Buildbot, CruiseControl.NET, Bamboo and a few others over the course of developing software both personally and at work, I don't believe anyone actually uses these build engines because they're nice to use (almost every build server transition I've seen comes down to "well, it's not as bad as the previous one"). While Harbormaster isn't exactly a shining beacon of usability at the moment, many existing build engines either suffer performance issues, are quite heavy in terms of resource usage, are poorly architected, are expensive, don't scale and / or are just downright convoluted to configure. I think with the direction that Harbormaster is heading, it will be able to address a significant number of those problems and we'll see people opting to change their build server to Harbormaster rather than integrate with their existing one.

Personally I would like to have an integrated build server, however with the direction that Harbormaster is going, it will not be it. We have about 200 active git repositories in a Phabricator installation, there are a lot of similarities, but there are also enough differences that we would need an prohibitive amount of build plans for them all. For me, the perfect solution is what travis is doing, having a per repository file that tells the CI what and how to do it.

You can have a single build plan that clones the repo and then runs a script (located in the repo) to do the build, then just apply the same build plan to all repositories via Herald.

@hach-que Fyi, one reason one might want to integrate with Jenkins is that you have to share the existing resource with other people/projects as you can't get your own hardware resource for whatever reason. That said, I'm happy if a Jenkins-agnostic way to integrate with 3rd party build-bots remains available.

We plan to improve Jenkins support over time, but we plan to improve Harbormaster as a first-class build platform more.

This project's ability to move forward is just heavily constrained by support costs, and right now people showing up in IRC with a blog post and a dream about using Jenkins are consuming a lot of support effort. This works today, but doesn't work well enough that we can help people set it up yet.

@hach-que, were you about to develop Jenkins support in your fork or somewhere else?

One potential feature that can be useful is:

1. configure Jenkins url and access credentials (username & api key) on the Harbormaster application configuration page
2. add new Harbormaster step to notify Jenkins about new commit in the repository (call is different depending on repository type, but in general you post to http://yourserver/jenkins/git/notifyCommit url)

Things, like triggering Jenkins build doesn't require extra coding, because they can be achieved via regular API call.

I don't have any plans to extend Harbormaster with any build steps for external build servers. My focus is primarily on improving Drydock to allocate hosts so that Harbormaster can be used directly, because I think that's more valuable long term (with external build server support is intended mainly for migration).

Harbormaster wins because it's more tightly integrated into Phabricator, but feature-wise it's really not a Jenkins.

I think there is no point to invent the bicycle if you can offload all work to Jenkins which has tons of plugins and for doing all kinds of stuff.

If you want to offload your builds to Jenkins then you can, using the HTTP request mechanism that is already present. This mechanism works for any build server that accepts HTTP requests to start builds.

Fundamentally external build servers won't ever have as good integration as Harbormaster, because they don't represent the concepts the same way that Harbormaster does. And implementing conversions for N external build servers isn't very maintainable.

If you want to offload your builds to Jenkins then you can, using the HTTP request mechanism that is already present. This mechanism works for any build server that accepts HTTP requests to start builds.

I'm aware of that, thanks. Stuff I was talking about is an alternative workflow for working with Jenkins: T6518

And implementing conversions for N external build servers isn't very maintainable.

I agree. Supporting only major ones, like Jenkins is enough for now. It's of no cost to implement custom build step in local fork if anybody needs to.