Page MenuHomePhabricator

Document one or two reasonable workflows for Github repositories
Open, NormalPublic

Description

Unless you already have a good idea how Phabricator works, it's not necessarily really clear from the documentation how you'd integrate it into a project that's hosted on Github. Adding the repository is straightforward and the UI guides you through that part, but after that, it's not clear what to do next.

I'm setting up Phabricator for a company that uses Github-hosted private repos and I'd love to be able to point them to some reassuring documentation to show them an initial workflow they can try while they're getting the hang of it, preferably one that represents the accumulated wisdom of other people who're using similar setups.

Here are some example questions I've either already been asked or am anticipating.

Do you still need to push to Github to get a change reviewed?

How, if at all, does Phabricator interact with Github pull requests? Does a pull request get created when a reviewer accepts in Differential?

Who merges the change into the master branch? Do they do it locally or via the Github UI?

What is the equivalent of accepting and merging a diff from someone who doesn't have push access to the main repository and is working on their own fork of the code?

Event Timeline

sgrimm raised the priority of this task from to Needs Triage.
sgrimm updated the task description. (Show Details)
sgrimm added projects: GitHub, Documentation.
sgrimm added a subscriber: sgrimm.

My approach to answering these questions is to basically say, "Once you're using Phabricator, Github becomes a dumb git repository server and you should just completely ignore its UI." I hope that's actually true.

The quick answer is that this:

Once you're using Phabricator, Github becomes a dumb git repository server and you should just completely ignore its UI.

...is substantially correct. There are some shades of grey and nuances if this isn't actually desirable (e.g., if you still want to interact with pull requests somehow).


Generally, we've tread lightly here for a few reasons:

  • The workflows we like are good at scale, but rare in the wild. The workflows that are common in the wild have flaws at scale. But we don't want the intro doc to be like "you have to change everything you're doing now, it's all bad and won't scale, do this instead".
  • We support a wide variety of workflows for almost every step, and usually they all have advantages and disadvantages.
  • Long ago, I assumed that most users would be strongly committed to a workflow they were used to, and would mostly want to bend Phabricator to use that workflow, so the main challenge would be getting users over bumps when they tried to apply a preconceived workflow. This is true to some degree, but seems less true in the wild than I'd guessed.
  • Diviner is ugly and slow to regenerate documentation until T988, which is close to ready if I can ever get a chance to work on it again.
  • There's a selection bias effect in the feedback we receive, in that most feedback we get is from users who use the software, so they've necessarily survived the install process and often figured out some kind of workflow. Consequently, we hear a lot more "add features" than "make it easier to install".

Here's the workflow we use for Phabricator itself. This isn't the only way to do things, but we think it works well, and believe it will scale based on experience at Facebook. Generally, this is very similar to the workflow used at Facebook.

Hosting

  • From 2011-2013, Phabricator's authoritative remote was a public GitHub repository. We imported from GitHub into Diffusion. In modern Phabricator, this is the "Import an Existing External Repository" option, although it was the only option at the time.
  • In late 2013, shortly after we built repository hosting, we moved the primary host to Phabricator itself, and configured an automatic mirror to GitHub in Diffusion. This made it easier for us to control access, allowed us to write Herald rules, and let us test the implementation. Most Phabricator users still use the GitHub mirror (it auto-pushes a couple seconds after the authoritative one gets updated).
  • The major advantages of hosted repositories are simplified access control, access to pre-commit Herald rules, other pre-commit features (we'll block --force pushes by default, for example), and the push log. Using an imported repository instead has no impact on any of the review/audit workflows.

Branching

  • We develop in local git branches, which we never push into the remote. We merge them into master locally before pushing.
  • We push directly to master in the remote, following the practices discussed in this document.
  • We essentially never create long-lived feature branches, or any other branches in the remote other than master.
  • We control access to features in development with runtime configuration.
  • This approach has a variety of advantages and disadvantages, see the linked document. This is basically the same as what we did at Facebook. One major advantage in all cases is that merge risk is spread across many smaller events, rather than systematically concentrated into a few large events.

Merging

  • We use arc land to --squash-merge changes into master, so master is linear and each commit has a full commit message. Every commit on master is a working state. This approach (roughly, "one idea is one commit") is described in this document.
  • We create merge commits very rarely.
  • This approach has advantages at scale, see the linked document. Same stuff as Facebook, again.

Pull Requests / External Contributors

  • All employees have direct push access. Employees do not use pull requests.
  • We lightly encourage contributors to send revisions in Differential rather than pull requests, but still take changes from pull requests occasionally.
  • I merge accepted Differential revisions from non-commiters with arc patch --nobranch Dnnn and then git push. There is a button to do this automatically in the UI sometimes (see T182) but it's contributed and doesn't work especially well yet. We don't currently use it, although some users do.
  • I merge pull requests manually from the command line (using commands like git fetch X/x, git cherry-pick Y, git rebase -i master, and git commit --amend). Pull requests often require some level of editing which means I have to do this anyway, and their commit messages often aren't very good or complete, I want to reference the request in the commit message, and I want to avoid generating merge commits. Clicking the "Merge" button on GitHub also works fine, it just usually leads to a less desirable end state.

Code Review and Audit / Internal Contributors

  • Normal development happens on local branches. For details see my exact workflow, see this document. The rough shape of things is git checkout -b feature to create a local feature branch, git commit to make changes, arc diff to send those changes for review, and then arc land to merge and push the reviewed changes.
  • Almost all changes go through pre-commit review in Differential by another employee. Generally, code that leaves a developer's machine en route to the remote has been tested, reviewed, and is complete and deployable.
  • Changes which don't receive review are automatically flagged for audit by a Herald rule. We use this rarely, primarily for trivial or obvious changes.

Releases

  • We don't currently distinguish between master and releases. We haven't had stability issues with this approach yet, and probably won't start doing real releases until we do. This won't scale indefinitely, but Phabricator doesn't currently have mature release management tools (Facebook's tool, Releeph, is technically in HEAD, but is probably not in a usable state for anyone except Facebook).

GitHub

  • We use GitHub purely as a dumb repository server (and now as a dumb mirror).
  • We deal with Issues and Pull Requests when they arrive on GitHub, but there's no special workflow in either case. They're just like someone mailing us a patch or asking a question on StackOverflow or whatever other secondary channel this sort of stuff might come in on.
  • None of us touch GitHub during normal workflows.

Not sure how helpful that is, but let me know if there's anything I can clarify or expand on, etc.

There's also a bunch of future work here, none of which is likely to land too soon, but some of which might be relevant by later this year:

  • Releeph is complicated, but I believe we will eventually rename its "Pick Requests" to "Pull Requests" (see T3644), and it will become usable for requesting merges across branches in the general case, either Facebook-style ("merge this to the release") or GitHub style ("merge this to master"). This workflow should generally end up being familiar to users who are familiar with pull requests.
  • We will probably let you create something like a Differential revision out of a branch or commit range eventually, similar to Gerrit or GitHub pull requests. We think this is a worse workflow (for example, it reduces our ability to automatically correct errors using lint), but it's much more familiar for many users.
  • We're building a tool called Nuance which is intended to streamline human task queues (like responding to support emails or content reports). This will probably include some tools to make it easier to handle items like pull requests, stack overflow questions, GitHub issues, tweets, Quora questions, etc. -- anything with an API, pretty much. This is probably only relevant if you get a bunch of pull requests and issues, but want some other system to be authoritative (e.g., we'd like to turn pull requests into revisions and issues into tasks).
  • Generally, we've slowly improved support for interfacing with external systems (see T3179, T589). This is a low priority, but we have a fair amount of infrastructure here now that we didn't 9 months ago, at least.
  • I don't know when I'll finish T988, but the documentation should get better quicker once I do.

For what it's worth, here's a little intro document I wrote for the company I'm helping out. Company-specific stuff replaced with XXX here to make this easier to reuse if anyone wants to reuse it. If you don't mind, I'll also copy-and-paste a lot of what you wrote. They are a pretty small shop with just a few generalist developers of more or less equal responsibility, hence the recommendation for everyone to add Herald rules to make themselves review everything.


XXX uses Phabricator for code review and issue tracking. This is a brief introduction to Phabricator; see its user documentation for gory details.

Initial Setup

  1. Get an administrator to create a user account for you, and click the link in the welcome email to validate your account. (You have probably already done this or you wouldn't be reading this wiki page.)
  2. Install Arcanist on your local host. This command-line tool (`arc```) is the main thing you'll use to interact with Phabricator as a developer. The install is generally pretty simple, just cloning two git repositories and adding a directory to your search path.
  3. Tell arc who you are by running arc install-certificate http://phabricator.XXX.com/. This will tell you to log into the web site to get a certificate string, which you copy and paste back into your terminal window.
  4. _If you want to automatically be added to the reviewer list for all code edits_, add a Herald rule with a condition of "Repository is any of XXX" and an action of "Add me as a reviewer". The rule name is yours to decide -- go nuts. As will be apparent from the UI, it's possible to set up much more sophisticated matching criteria, e.g., only automatically include you on reviews of particular parts of the code base. But "everyone can review everything" is an okay starting point while the team is small.

A quick overview of Phabricator's pieces

When you log into Phabricator you'll see a nav bar on the left that gives you quick access to a bunch of Phabricator components. Here are the ones you're likely to deal with regularly.

  • Differential is the code review tool, analogous to the pull request UI on Github.
  • Maniphest is the issue-tracking system.
  • Phriction is a wiki.

Workflow

This section describes how a change goes from a developer's local git repository to the integration repository on Github. It describes one possible workflow, but not the _only_ possible workflow.

An important thing to note is that this workflow doesn't use any of Github's code review or pull request mechanisms; it treats Github as a repository host and nothing more.

Developer

Doris Developer is happy with her change and wants to get it integrated. She commits it locally and makes sure it merges cleanly into the latest master branch revision:

git commit -a
git fetch
git rebase origin/master

Then she submits it to Phabricator for review.

arc diff

Oops! A PEP8 error! Doris fixes her indentation and runs git commit -a --amend to update her revision, then tries arc diff again.

The code lint looks good this time, so arc pops up whatever editor Doris' EDITOR environment variable specifies, and, if she didn't include it in her commit message already, she fills in the "Test Plan" section to tell a reviewer how to test her change. If there are particular people she wants her code to be run past, she adds their usernames to the "Reviewers" line.

Arcanist submits the change to the Phabricator server for review. Then it updates Doris' local commit to include the URL of the revision on the server. Her work done for now, Doris goes and gets a sandwich.

Reviewer

Roger Reviewer sees an email notification about Doris' change, and clicks the link in the email to jump to Differential, Phabricator's code review interface.

He reads the summary and test plan, then, secretly loathing his mouse's scroll wheel, hits j to jump to the first change. (? shows the keyboard shortcuts.)

Reading the change, he discovers a problem. He clicks and drags to select a range of line numbers in the right-hand column of the diff view, and enters a comment. He does this a few more times in other areas of the diff, then enters a general comment in the textarea at the bottom of the diff, selects "Request Changes" from the Action drop-down, and hits the Clowncopterize button to send the clowny code back to its author.

Developer

Doris sees the email with Roger's review notes. She replies to the email to answer a few of the questions then edits the code in her local repository to clean up some of the problems.

When she's ready to try again, she updates her local revision and resubmits it.

git commit -a --amend
arc diff

(Because it amended her commit message the first time it submitted the change, arc knows this is an update to the existing diff rather than a brand-new one.) She types a brief description of her changes and exits her editor, and arc sends it back to the server for another round of review.

Reviewer

Roger sees the email notification and checks out Doris' changes. Everything looks good this time, so he selects "Accept Revision" in the "Action" dropdown at the bottom of the diff and hits the Clowncopterize button.

Developer

Doris pushes the changes to the integration repository.

git push origin master

Alas! Someone else has pushed another change in the meantime, and the push fails. She grabs the latest changes and rebases on top of them.

git fetch
git rebase origin/master

She tests her changes again; they look good, so she pushes again and this time the push succeeds.

Differences from Github workflow

This section highlights a few things to be aware of if you're coming from Github.

Developer merges to master

On Github, the project owner is responsible for merging changes into the integration branch, usually by way of accepting a pull request. This is pretty easy but breaks down if multiple people are working on the same part of the code base at the same time, since it puts the owner in the position of having to resolve merge conflicts. And even if there aren't merge conflicts, the owner will end up integrating a version of the code that has at no point actually existed on any developer's computer.

Phabricator, on the other hand, expects the developer to push to the integration branch. The downside is that this adds an extra step after a change is approved. But the upside is that if other changes have been integrated while the review process was going on, the developer gets to do the merge locally and _test the resulting code_ (or at least do a quick sanity check) before actually updating the integration branch. During intensive development of new code, this can prevent a significant number of bugs.

Github is just a file server

At no point during the example workflow above did anyone log into Github. In fact, in a full Phabricator workflow, Github is really just hosting the repositories and not doing much else. It's still adding some value (it acts as an offsite backup, for example) but once you're comfortable with Phabricator, you won't find yourself interacting with Github's UI too much, if at all.

hwinkel added a subscriber: hwinkel.

@thz, please have a look at this description.

chad triaged this task as Normal priority.Nov 2 2014, 10:22 PM

@sgrimm In your large post, you have a link that says 'Add a Herald Rule'. This link appears to be intended as a template so you can replace the domain with your own domain.However, clicking it actually brings you to the working domain http://phabricator.xxx.com. The result is funny and unexpected, but likely not intended. Maybe it should be changed to phabricator.example.com to prevent people from being linked to porn :)

@epriestley I have noticed that, in your first comment above, two links, titled "in this document", are broken (links are located in sections "Branching" and "Merging"). It would be nice to have those links to presumably useful and important documents fixed.

Hi All,

 I am not able to setup git connected to phabricator. I did lots of research and was able to setup phabricator, local git, local SVN servers. But git and SVN are not getting connected to phabricator. 
 OR
In other words, I can say that we are not able to add the projects and code to phabricator through git. Can anybody help me in providing step by step documentation?.

Can anybody help me how to setup git connectivity in a local phabricator server?.

Do we have a detailed step by step documentation on this?.

The bellow error is the one which I am getting and I am not able to create anything. Please help me.

Diff Parse Exception: Expected a hunk header, like 'Index: /path/to/file.ext' (svn), 'Property changes on: /path/to/file.ext' (svn properties), 'commit 59bcc3ad6775562f845953cf01624225' (git show), 'diff --git' (git diff), '--- filename' (unified diff), or 'diff -r' (hg diff or patch). >>> 1 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDlGDIYXQboQwa7S85WaaAtNOVRuJ1zmu/wtRlHfNuYumz+lFFQrznvR3WuQWSqzadT/qJfT9KV4/3NreBofXktXApPEqbvV9f8w6m0SwSt8WT5e4d2SSlevWqdoWesflWBy1vb1YwvRaU+tYXpAAOOJ5H7sC4oGkxm2xF5e38W3uvHG/7Ac+asyH1Lr1mMrbPw1mrTLUqd8yVBtq8DVDwTW3rJREAZezGdBBBroNHYjWBgJOUyR2zF4uTDNjih9ov/mugQhyKeHUudOQjxJfOCgvuLEMzZn04rUf/TNIMel3KnxJ0bqOxQSymspEKJzZ+755QDXrfZBZHlYXVqzqMecI3n4Gb5Mcst/A9lF/iD+PJ5ZtRmei7D4oRm4KcyrHn2hWLnDSPaT7bfraAMw0jf9lx6sCCPuas3YaIUvH5+/ylvRpbE1vv0654ykZHFMtXv8JZHDCjzaRZfz6+3zQ2rWQHKvokbqmvHrWYy97ni/hORGQeT+VcKrnh4p7/vHxKjnQEbY4VUYiZI46o4V8ylfPN+ykkQsn99a9cXv8fhZfLz6QtZ1QLqqhC2rUai6S1+07Sjp/IbZaJxCMy2AQzCSRU3HS6+UFG1/YX2rg37UKTOc6mR1f/KJdu+jDESDjxJRSgqQa2X2G7R54SDuBD8x6JJYOikQOdKNSoWiCFqxQ== sateesh.kundam@soctronics.com

@skundam this is not the expected way to get support. Please see Support Resources.