Page MenuHomePhabricator

General support for multiple URIs for a repository
Closed, ResolvedPublic

Description

This is partly covered by T4705 and T4245 but has a slightly different use case.

We serve repositories from multiple URLs: https://git..../reponame and https://git..../ro/reponame. Each of these has a different purpose. The first is the main URI, the second is a read only replica which doesn't require kerberos authentication. We're also considering some other special prefixes (/backdoor/, etc.). For a variety of reasons we can not serve the read/write version to phabricator (as an externally hosted repo). This means that repository auto-detection doesn't work.

The remote URI for this working copy is
"https://git.../reponame".

No repositories matched the query. Check that your configuration is
correct, or use "repository.callsign" to select a repository explicitly.

Since we may many hundreds of repositories manually adding a .arcconfig to each one is time consuming an expensive. We have managed to eliminate the need for phabricator.uri and base by setting them in /etc/arcconfig. Obviously for callsigns this approach won't work. We'd like to be able to list server-set a set of alternate URIs that map the same repository.

This is also useful if a repo gets renamed or moved to a different host.

Related Objects

Event Timeline

eadler created this task.Feb 16 2016, 5:08 PM
eadler moved this task from Restricted Project Column to Restricted Project Column on the Restricted Project board.
eadler updated the task description. (Show Details)Feb 16 2016, 5:47 PM
avivey added a subscriber: avivey.Feb 16 2016, 6:25 PM

Also probably related to T8831.

epriestley moved this task from Backlog to vNext on the Diffusion board.Feb 16 2016, 9:43 PM
epriestley moved this task from vNext to v3 on the Diffusion board.Feb 16 2016, 9:50 PM
epriestley edited projects, added Diffusion (v3); removed Diffusion.
epriestley moved this task from Backlog to URI Stuff on the Diffusion (v3) board.
scode added a subscriber: scode.Feb 17 2016, 1:34 AM
eadler moved this task from Restricted Project Column to Restricted Project Column on the Restricted Project board.Feb 23 2016, 6:13 PM

Broadly, here are the general issues we've currently seen with Diffusion URIs:

  • Here, users clone from one place and Diffusion fetches from a different place.
  • Here, repository URIs may move, but you may still want the old URI to be recognized. For example, /facebook/phabricator/ moved to /phacility/phabricator/ on GitHub, but both URIs continued to work. We should be able to recognize them as the same repository.
  • In T7516, a user would like API access to clone URIs.
  • In T4705, I think the original use case is similar to "clone and fetch from different places", but with the additional caveat that we have some weird magic around whether usernames are included in clone URIs for users which it would be nice to get rid of or simplify.
  • Also in T4705 and elsewhere, users have a hard time finding diffusion.ssh-port.
  • In T5406, a user wants a read-only clone of imported repositories so they can use Phabricator's more granular access control to re-serve a GitHub private repository.
  • In T8831, which is probably the inverse of this, a user wants arc to recognize mirror URIs.
  • Future work in T4292 may create multiple read/write URIs for hosted repositories (e.g., phabricator.arctic.mycompany.com/... and phabricator.oceania.mycompany.com/...).
  • Current and near-future work in T4245 means that Diffusion repositories already have multiple live URIs (/diffusion/<id>/ and /diffusion/<callsign>/ both work for repositories with a callsign, and /source/<name>/ is planned to work soon).

My plan to deal with all of this is to unify mirror URIs, fetch URIs, clone URIs, and push URIs into a single type of "Repository URI" object. Each repository will have one or more associated URIs, any of the URIs would be recognized as being "the repository", and each URI will have options to enable mirroring, reading, writing, fetching, and display (of course, some of these are mutually exclusive).

To add aliases (like the case here), you'd use some New Repository URI option, add the URI, and then not set any other flags on it.

To move a repository (like /facebook/phabricator/ to /phacility/phabricator/), you'd add the new URI, then use some Make This the Fetch URI to move the remote source of the repository from the old location to the new one.

For T7516, the API would just provide a list of URIs with all their attributes (is this readable, writable, remote, local, fetched, mirrored, etc). The endpoint doing this would be the new diffusion.search, after T10337.

For T4705, same as above but maybe there are some options on the URI detail to edit user/port stuff.

For T5406, and elsewhere, URIs could have a flag like "Show this as a clone URI for users", which would be off for the Phabricator URIs for hosted repositories by default.

For T8831, all URIs would be recognized by arc. You'd add mirrors by adding a URI first, then using Activate Mirroring.

For T4292, all supported live URIs would be present in the list.

I would also like to simplify the creation workflow and the "hosted / imported" distinction and process of swapping between them. I think we can mostly do this by making state an implicit function of which URIs are configured (e.g., if you have a fetch URI, you're an imported repository) but maybe it will need to remain somewhat explicit.

At the end of the day, we'll end up with something like this on the repository detail screen:

+==========================================================+
| URIs                                   [ + Add New URI ] |
+----------------------------------------------------------+
|  ( ssh://.../ro/x.git           [V Fetch]             )  |
|  ( ssh://.../x.git              [^ Clone (Read/Write)])  |
|  ( ssh://phabricator/x/         [^ Clone (Read)]      )  |
|  ( ssh://phabricator/123/                             )  |
|  ( https://backup/x/            [M Mirror]            )  |
|  ( https://.../ro/x.git                               )  |
+----------------------------------------------------------+

..and you'd use this to add new mirrors, change (or remove) where the repository was fetched from, add new display URIs for users to figure out what to clone, add new aliases which purely make non-canonical repositories recognizable to arc, and configure Phabricator's repository serving behavior.

eadler moved this task from Restricted Project Column to Restricted Project Column on the Restricted Project board.Apr 8 2016, 6:40 PM

Some of the changes associated with T4292/T10748 -- principally, D15742 -- have pushed this forward, although it doesn't really do anything yet:

This substantially exists at HEAD, but Diffusion doesn't actually use it yet.

I plan to cut stable for this week, then move Diffusion over to use the new code after the release cut. Ideally I'll get through it tomorrow, but that may be a little optimistic.

I'll provide a more detailed update on T10748 shortly.

epriestley closed this task as Resolved.May 4 2016, 11:32 PM
epriestley claimed this task.

I'm going to call this resolved now -- there are still substantial rough edges with this and the rest of T10748, but we're running on the new code and everything else should mostly be UI stuff. T10922 has general guidance and I'll open an errata task to track cleanup shortly.

urzds added a subscriber: urzds.Jul 12 2017, 11:09 AM