Page MenuHomePhabricator

Phacility Launch Status
Closed, ResolvedPublic

Assigned To
Authored By
Jun 5 2012, 1:24 PM
Referenced Files
"Mountain of Wealth" token, awarded by btrahan."Haypence" token, awarded by staticshock."Like" token, awarded by allixsenos."Yellow Medal" token, awarded by avivey."Piece of Eight" token, awarded by chad.


If you want updates on Phacility (the SaaS version of Phabricator that @btrahan and I are working on), CC yourself here and we'll let you know when we make progress and when we have something ready for early adopters to try out.

(I linked this on Quora as a sort of "Cc yourself if you want status updates".)

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

I wish I knew about pokemon :(

This comment was removed by epriestley.
epriestley changed the visibility from "All Users" to "Public (No Login Required)".Mar 17 2014, 4:10 AM

would pay *at least* $1 per user for Saas

I second @phil's comment.

I spend U$45 per month to keep my own EC2 instance. I need to take care of backups and I find it slow.

I'd happily pay the same (U$45 + U$1+ per user) per month.

Unknown Object (User) added a subscriber: Unknown Object (User).Oct 1 2014, 11:13 PM
Ore4444 removed a subscriber: Unknown Object (User).Nov 5 2014, 10:46 AM
Ore4444 added a subscriber: Ore4444.
Ore4444 removed a subscriber: Ore4444.

Do you guys have any idea what the process will look like to migrate stuff from our current self-hosted situation to your servers? Will there be a simple import process?

The very first version won't have a simple self-serve import process. Roughly:

  • We don't know exactly how much demand we'll have from existing installs vs new installs, so we aren't putting a lot of resources into import tools up front. For example, we've heard from a number of people that they'd use Phabricator if a hosted version was available, so presumably they don't need import tools.
  • Depending on demand, we'll probably use a mixture of rough tooling and manual coordination to import the first few existing installs. That should give us a better sense of how to build and improve the tooling technically. In theory, the process is simple; in practice, we haven't actually run through it with enough real third-party data to be sure.
  • In the long run, we hope to have self-serve import/export tools, where you run some command in Phabricator to produce an archive file and can upload it to the service (or download your install as an archive if you want to leave the service), but these may take some time to build and will compete against other features for priority.
  • There are some hard cases around user accounts and authentication given how that stuff is planned to work in the initial version. For example, if your local user accounts authenticate via corporate LDAP which is on a VPN, we can't possibly use that same credential source in the hosted version. The path forward here isn't clear yet.

We'll make this process more clear once we're ready to help installs import data, but the initial feature set will focus on new installs.

We don't know exactly how much demand we'll have from existing installs vs new installs, so we aren't putting a lot of resources into import tools up front. For example, we've heard from a number of people that they'd use Phabricator if a hosted version was available, so presumably they don't need import tools.

I'd expect that this would roughly be affected by:

  • Installs that have customizations won't be able to use the hosted version for obvious reasons.
  • Anyone running small or self-managed infrastructure, where they already have hosting resources available (i.e. small, independent teams where time is a greater resource than money).

There are some hard cases around user accounts and authentication given how that stuff is planned to work in the initial version. For example, if your local user accounts authenticate via corporate LDAP which is on a VPN, we can't possibly use that same credential source in the hosted version. The path forward here isn't clear yet.

Just generally, if a business is already using Google Apps, they probably have the authentication connector which synchronises LDAP / Active Directory accounts with Google authentication. For businesses in this situation, the "Login with Google" adapter will probably work fine.

General update about progress: we've built this, and it almost all works right now.

Our product is a single large Phabricator cluster with logical instancing on each request: we do not deploy a virtual machine per instance. This means that instances have greater and more consistent access to hardware resources, and we can scale on most dimensions by adding hardware to the pool without service interruptions. This architecture took some time to develop, but our expectation is that it will give us a more scalable, robust, and stable platform in the long run. Overall, this architecture is similar to the architecture used by other large, scalable web applications like Facebook.

We're planning to roll it out like this:

VersionDescriptionRough TimelineAvailabilityUsable ForNotes
v-2InternalNowInternalDevelopmentPreview for development.
v-1Closed AlphaDays?SelectiveTestingNo more data wipes, but not production ready.
v0Closed BetaWeek?SelectiveReal WorkReady for real users, missing a few features.
v1Open BetaWhen ReadyPublicReal WorkEssentially complete, but we'll likely run into some kinks. Basically, a production release with a weak SLA expectation.

If you're interested in being part of the closed alpha or closed beta, we expect to be looking for a few volunteers / guinea pigs soon. In particular, this is what to expect during these phases:

  • Closed Alpha: We won't be wiping data any more, but won't recommend this for real work yet. This is more of a "kick the tires" phase: we'd like to gather feedback about stuff that's missing, broken, unexpected, confusing, etc., during this phase.
  • Closed Beta: We'll support this for real work. We'd like to get a few smaller teams (ideally with use cases covering all the VCSes and applications) using it, but limit the number of users and the size of installs to make sure we can keep up with issues as we work in the last set of mostly-quality-of-life improvements. We won't wipe or lose data, but there will be some service interruptions as we make changes and deploy them.

Once the closed beta is stable, we'll let anyone sign up in the Open Beta phase. This will essentially be the release product, we just want to set expectations that it will be pretty new so there will probably be some issues we didn't forsee that we need to work through. We won't lose data, but we expect there will be minor service interruptions. We'll continue to work to improve the service, although a lot of the work planned for this phase is really just Phabricator feature development (like Conpherence v2, SMS, performance tools, etc).

If you're interested in following progress in more detail:

  • The Phacility workboard is up to date and lists exactly what work is blocking each phase. We're working through these items roughly-in-order now.
  • The Phacility Cluster Wiki has technical documentation about what we've built and how it works. This is primarily an internal document, but it may be useful in understanding the cluster (or interesting on its own).

To address a couple of questions which were asked earlier in this thread (and elsewhere) specifically, now that we have more detail and can provide better answers:

How much of the SAAS product is open source?

Almost all of it. We've open sourced everything except code which we think is only useful for running a service which competes with us directly. The major foundational components which have general-purpose uses (billing, subscriptions, clustering, instancing) are all in the upstream (although some are still pretty prototype-y). By line count, today, about 98% of the code we deploy is open. The majority of the closed code is the administrative GUI which allows you to manage instances. You can find a detailed list of what we deploy at Phacility Cluster Repositories.

Will there be a simple import process?

Not initially. You can find some discussion in T7149. Generally, this is a hard technical problem because third party data may reference services we can not access (like LDAP and private repository URIs) and there's no clearcut way to resolve many of these references after import. We'd like to stabilize everything else and get a better sense of demand before we try to tackle this.

Will there be a simple export process?

We'll build this before we leave open beta, but it won't be available initially. See T7148 for discussion. This is something we think is very important, but there are a mixture of technical, security and product challenges. On the security side, it's bad if an administrator account can "export" a live install and read things like session keys out of it, then impersonate other users on the live install. To some degree this will be unavoidable, and we may resolve this by simply putting a high process barrier ("fax us on company letterhead...") around the "export" process, but we'd rather have a less-perilous process which we can make easier to use ("click a button"). If you want to leave the service before we have a self-serve export process ready, we can help you get your data out manually.

How much will the service cost?

The service will cost $20/user/month.

For users running software businesses and evaluating this cost against the cost of paying staff to run this infrastructure in-house, we expect this will be compelling. This price point is comparable to other offerings targeting this segment of the market.

For some users (including open source, non-profits, students, etc.) who are making a different evaluation (for example, when this service would not replace a portion of a technical employee's salary), this offering will probably be less compelling. For now, some alternatives are the "free" plan (self-host, since it's open source), or there is at least one hosting provider that offers much less expensive plans.

We may eventually offer additional hosting plans, but this isn't on our short-term roadmap and won't happen until much later this year at the very earliest.

What's included? What's not included?

In general, everything that Phabricator supports will work (including all the separately-configurable features like notifications, inbound mail, HTTP / SSH repository hosting, CDN, etc), plus backups, scaling, support, and data export (before leaving beta; see above). A few services aren't fully hooked up yet and may not be available until the Open Beta. Stuff that won't be supported:

  • The cluster uses a centralized auth system, so we won't support connecting to an in-house authentication source like LDAP. Generally, auth will work more like other hosted services where you invite users via email.
  • We won't support running custom code, for a bevy of technical and security reasons.
  • You don't get SSH access to the machine (VCS SSH operations work properly).
  • We don't support custom domains: your instance will be <instance>
  • A handful of configuration options are locked, mostly to prevent users from accidentally breaking instances.
  • Although the majority of devices are deployed redundantly, we don't yet deploy all services in a fully redundant way. We expect to be able to recover from hardware failures quickly, but recovery may not be as quick as it will be in the future when we can, e.g., promote a read-only replica to master while swapping the failed device. Until we leave open beta, we expect development will interrupt service more often than hardware failure, so we don't plan to focus on this until things are otherwise stable.
  • The whole cluster runs the same version of the software, and all instances will upgrade simultaneously and be kept near HEAD. We won't be auto-deploying HEAD, and will have have process before code is deployed to make sure it's stable, but we expect installs to generally be no more than about a week behind HEAD. In particular, you can not freeze your instance at a specific version, and the product will continue to change over time (like other SAAS products). We do not plan to fork or freeze a "stable" release or anything like that. Although this probably isn't surprising since pretty much every SAAS product works like this, Phabricator does have a relatively high rate of change and has gone through multiple major revisions in the last few years. You should expect that we'll continue to move the product forward.

We may eventually offer a higher-end "Enterprise Hosting" product which gives you a dedicated, isolated cluster that we administrate. This would likely support custom domains, custom code, full host access, freezing versions, etc., but this isn't on our short-term roadmap.

I will not be offended as this is tangential if it is removed from this thread :)

I have said it before but wanted to reiterate. If there is anything I can do, or can encourage Wikimedia to do on your behalf please let me know.

Things that may be of service:

  • A written testimonial about how valuable we find Phabricator and how we use it.
  • I would totally get on a call with some prospective and sing your praises. I'm generally well spoken and I have been using Phabricator now for 3-ish years. I have nothing but good things to say.
  • I don't know if you guys want to do the logo's of people who use Phabricator on some landing page, but if so I (or I'm positive the same goes for @qgil) would be willing to try to sort it out from our perspective.
  • We have some one-off but well used tooling that facilitated our migration from Bugzilla, RT, and to a lesser extent Trello. If anyone is stuck with that (I know you guys have your own migration patterns in mind though) I could share the experience.
  • I know there was a talk recently given at FOSDEM about some of our experiences that may be beneficial:
    1. Video to come:
  • Moral support (not including back rubs but maybe water bottle refilling)

So yeah, great stuff. Super excited for you guys.

Quick update:

  • The exact phasing ended up being pretty muddy since the early phases got held up by minor UX issues while later phase work moved forward at pace, but we're more-or-less in the Closed Beta phase now with much of the Open Beta work completed.
  • If you want to check out how things look, you can sign up on This currently requires administrative approval. We'll consult our crystal ball and approve you if if it prophesizes good fortune.
  • You should expect that pretty much everything works now (if you find something that doesn't work, is confusing, seems missing, etc., let us know) and you can launch instances and invite users, etc., but please don't do actual work yet since we might end up needing to wipe data again or something (we don't expect to, but just don't consider the infrastructure stable enough for production use yet).
  • If you get a bill before Open Beta is announced, just ignore it -- we aren't charging anyone yet. You can't pay it right now anyway, and it's either us testing something or a bug. We'll make it clear when the service is leaving the testing phase and becoming a paid service, and make sure no one is getting charged for anything they didn't mean to purchase.

(And @chasemp, thanks for the offer! We'll let you know when we actually do some marketing stuff.)

@epriestley any ETA on the all-clear for Real Work (to borrow terminology from the previously published rollout schedule)?

I'd roughly guess sometime next week, but it depends on how much stuff turns up and how hard it is to fix (currently, things look good). We moved our internal stuff over less than 24 hours ago, and I'd like to have at least a few days of stable use before we move forward.

The Phacility workboard has the most up-to-date view of what remains; we'll approve it for real work approximately after finishing all the stuff in the "v1" column.

(I also have some personal commitments at the end of next week which might hold it up over the weekend, since I want to be fully available when we do move forward.)

A few more particulars have solidified and nothing major has come up, so here's a tentative launch plan:

  • We'll OK this for real work tomorrow morning (Saturday, February 21st). I'll note it here and update the instructions on
  • We'll move into open beta in about a week, on Monday, March 2.

We have some limited staff availability next week (before the launch), so it may take us a couple hours longer than usual to respond to support requests if you catch us at a bad time. We'll be fully available after the open beta launch.

We'll begin charging for instances when we move into open beta. If you have a test instance and don't want to be charged for it, disable it before then (or just don't pay us, and we'll suspend it for you eventually). Everything is free until the open launch (we'll issue a service credit from the beginning of time until the launch time when we do launch).

Let us know if you encounter issues. Some of the tooling and automation will probably be a little rough for a while, but we should be able to resolve most things quickly by hand until we figure out what we need to build better tools for.

A few minor things came up, but instances are now good to go for "real work".

This is launching, uh, "real soon", and is definitely not launched at all yet, so don't, say, try to use it. Because it hasn't launched. Despite this, I'm closing this task, for reasons.

Okay it is really launched now.