Page MenuHomePhabricator

Implement `diffusion.repository.edit`, for creating and editing repositories via the API
Closed, ResolvedPublic

Description

repository.create is an old pile of junk, and should be replaced with a modern EditEngine-driven diffusion.repository.edit endpoint.

Currently, Diffusion uses a multi-page create flow for repositories. EditEngine doesn't support this. We either have to get rid of the flow in Diffusion or support the flow in EditEngine.

I lean strongly toward throwing this flow away: it doesn't feel very good, and I think we can do better by reducing the amount of information required to create a repository, blending the ideas of "hosted" vs "imported", and moving toward a more flexible model of repository URIs (see T10366).

We don't want to end up with a single gigantic edit form so EditEngine will still need some concept of pages, but this can be a very lightweight concept which doesn't require the browser to retain state during the flow.

Directly adjacent work:

  • T10366 - The current URI and "hosted/imported" model probably does not translate cleanly to EditEngine, and T10366 describes the future model that we want to move toward.
  • T6848 - The current edit UI is users-with-edit-permission-only, but should turn into more of a "manage/details" interface where only the actual editing interactions require edit permission.
  • T9554 - Should be resolved here since this is now an explicitly supported method for performing imports.

Vaguely related:

  • We should probably resolve "local path" uniqueness (T4039) here, either by fixing it or giving up.
  • This should obsolete and remove bin/repository edit.
  • Mirroring never really got formalized, although this runs into the URI stuff (T6131).
  • "Create similar repository" (T6722) is maybe obsoleted by just providing a reasonable API endpoint? We could also conceivably let ?template=... and HTTP pre-filling also fill values for control pages which aren't visible. Sort of messy but I don't immediately hate this and maybe it's actually fine? We should resolve this or give up, in any case.
  • This resolves T9604, which can probably just merge here.
  • This or T10366 resolves T9679.
  • T10241 is probably a strict bug that should resolve here or in T10366.
  • T8611 is probably a strict bug that should resolve here.

Very vaguely related:

  • T7221 (git:// shouldn't require credentials)
  • T6655 (Generate SSH keys when creating credentials)

The pathway forward is likely:

  • Construct a parallel editpro flow which is based on EditEngine, and expose the Conduit endpoint based on the new engine. They'll start powerless and grow in strength over time.
  • Clear whatever blocks we need to in order to get creation working as a single page flow. Principally, the hosted/imported distinction needs to go away, and methods like $repository->isHosted() need to be defined indirectly in terms of associated URIs. Unclear how much of T10366 needs to happen to make this plausible. This part is the most nebulous.
  • Port the rest of the fields, creating a single gigantic mega-form in the editpro workflow. This part is time consuming but straightforward.
  • Implement "pages", which is probably relatively easy since EditEngine has seemed like the right amount of abstraction so far and should accommodate this gracefully.
  • Replace the old UIs.

Revisions and Commits

rP Phabricator
D15844
D15842
D15841
D15839
D15838
D15836
D15835
D15832
D15829
D15825
D15822
D15821
D15817
D15816
D15815
D15813
D15811
D15809
D15808
D15805
D15804
D15801
D15800
D15740
D15742
D15741
D15737
D15736

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

T10751 and T4292 are reaching a steadier state now and have some indirect/soft dependencies on this (for example, the new panel for monitoring the cluster status of a particular repository is part of this new UI) so I expect to drive this forward at least somewhat in the next few days. I'm not sure how ambitious I'll feel about meddling with the URI code before the release cut, but there's a good chunk of basic work remaining before that becomes an issue.

Recent progress is just modernizing a few more panels (Storage, Staging, Automation, "Allow Dangerous Changes"). All these transaction types now have API support, too.

Next up is the rest of the panels (remaining: Symbols, Branches, Actions) and then I'll figure out how ambitious I feel about tackling the URI stuff, which I think is the major blocker for cutting the UI over.


GoalRefTimeNotes
Storage, StagingD15800, D158010.5 HoursPort "Storage" and "Staging Area" UI / transactions.
Automation, DangerD15804, D158050.5 HoursPort "Automation" and "Dangerous Changes" UI / transactions.
Subtotal1 Hour
Cumulative Total4 Hours

This is now mostly built as a new "shadow UI" and we're nearly ready to cut over and start having it affect repository behavior.

  • diffusion.repository.edit can now create and edit (almost?) everything.
  • diffusion.repository.search can now retrieve (new) URI information. This information isn't authoritative yet.
  • diffusion.uri.edit can now create and edit URIs. These aren't authoritative yet.

There are a handful of things that still don't work, probably some stuff I missed, and lots of opportunities to polish the UI and provide more guidance about workflows, but the fundamentals seem to be in pretty good shape.

I'm going to cut the stable branch before adding code to let any of this actually do any harm, then make the new URI information authoritative over the next day or two.

The complexity of the migration is still a bit of an unknown since I haven't planned it out in any real detail yet, but I currently expect this to be fully shipped by next week's release.


GoalRefTimeNotes
Symbols, Branches, ActionsD15808, D15809, D15811, D158131 HourPort "Symbols", "Branches" and "Actions" UI /transactions.
URI EditingD15815, D15816, D15817, D15821, D15822, D158253 HoursInfrastructure for querying and editing repository URIs.
Subtotal4 Hours
Cumulative Total8 Hours

This is now live in HEAD of master, and deployed to secure. Here's phabricator/ on this install:

Screen Shot 2016-05-04 at 5.08.49 PM.png (992×1 px, 133 KB)

Screen Shot 2016-05-04 at 5.08.51 PM.png (992×1 px, 150 KB)

This is still rough, particularly from a UI perspective, and installs that are not brimming with ambition and optimism should avoid upgrading until it has some time to stabilize.

  • T10922 has general guidance on changes and upgrade impact.
  • T10923 has known issues and errata.

Next steps are to let this stabilize a bit and work through the errata.

The deployment went fairly cleanly so I currently expect to promote normally this week, although that could change if we run into issues in the next couple of days or the whole thing spontaneously bursts into flames or whatever.


GoalRefTimeNotes
PagingD158320.5 HoursSupport pagination in EditEngine.
Misc CleanupD15829, D15835, D15836, D15837, D15838, D158391 HourTie up some loose ends.
MirroringD158410.5 HoursMigrate mirroring.
CutoverD158424 HoursMigrate everything else.
Subtotal6 Hours
Cumulative Total14 Hours

We hit a couple of minor issues (e.g. D15856, D15865) in the last few days, but this seems to be working in production.

However, I declined to promote it to stable this week because the UX is still pretty rough in places (outdated documentation, missing hints in the UI, etc). I plan to clean up errata (T10923) to resolve these issues, and it should actually fully ship in this upcoming release.

I think this is substantially complete now. There will probably still be a trickle of issues as this gets more use and T10923 has some stuff that didn't get fully addressed here, but everything I've been tracking is in reasonable shape as far as I'm aware.

As far as core API access goes, this document should walk you through how to create new repositories with the API:

This document has a general overview of *.edit endpoints:

The API method pages themselves have details about available fields and transactions:

For general UI changes:

Let us know if you run into any issues as you start using this stuff, I'm sure there at least a few things that slipped through the cracks. Generally, the expectation is that every web capability is now fully exposed through the API.

These changes will promote to stable in the normal way later this week, a little more than 48 hours from now. They are also live on this install, although we don't provide public access to repository creation or management so you can only get a rough sense of them (you should be able to browse the new management interfaces, though: like this, for example).


GoalRefTimeNotes
ErrataT109234 HoursCleanup, UI/UX, documentation, errata.
Subtotal4 Hours
Cumulative Total18 Hours

Moving this to "Paused" since we're waiting on feedback / deployment / bug reports / things to catch on fire.

This appears to be stable and working properly, at least for the most part.