Allow external systems to edit the content of dashboard panels so they can publish information
Open, Needs TriagePublic

Description

Plausible use cases:

  • Copy announcements in some other system into Phabricator for greater visibility.
  • Publish deployment / test / release state held in some other system to a panel to make it easier to see everything in one place.

Original Description

We have a MOTD panel that we'd like to update programmatically. We also have other panels we'd like to implement that would be useful to have programmatic access to ("List what's currently broken" or "Can we do deploys?")

eadler created this task.Apr 22 2016, 4:49 PM

This one seems OKish to me -- I think those use cases are pretty reasonable and root-problem-ish? I'm assuming "waht's currently broken" and "can we do deploys" are information known by some other system or tool.

An alternative approach might be to write a custom panel which pulls data off some other API, but I think it's reasonable to want to push the data into Phabricator instead of pull it from CurrentDeployStateBotPage or whatever. Although maybe an "iframe dashboard panel" would be easier as a stopgap.

The general case of this -- API access to dashboards as a whole, and configuring/editing dashboards purely over the API -- seems a little flimsy and harder (although maybe not really so bad), but just editing panel content seems reasonable to me.

chad added a comment.Apr 22 2016, 5:32 PM

My concern is users will install their own dashboards and this doesn't particularly solve anything. I don't know what the core problem is to suggest an alternative solution.

A Phriction Panel which we talked about briefly would provide this automagically without wiring Dashboards to Conduit. Phame also is a reasonable panel for providing status updates.

epriestley renamed this task from Provide a dashboards conduit API to Allow external systems to edit the content of dashboard panels so they can publish information.Apr 22 2016, 5:34 PM
epriestley updated the task description. (Show Details)

Here's an example I could imagine us doing: we have some button on admin.phacility.com that says "Start a deploy". When that's pressed it causes these actions to occur:

  • Tweet about it on @phacilitystatus.
  • Update a panel on this install so anyone coming here to report things has a heads up.
  • Swap whoever pressed the button over to become the ops oncall.
  • Publish some notification to instances/admin.

When the deploy ends, we'd revert that state.

We could do that with Phame, but it would be silly to publish frequent/lightweight actions like "Starting a deploy" / "Deploy complete" to a blog, and not very visible.

I don't think it's a big deal if some users create their own dashboards -- they can include panels like this if they want the information, or drop them if they don't. As long as it's merely informative and not critical, I think it's reasonable to say "here's a way to get this useful-but-noncritical information broadcast a little more widely".

chad added a comment.Apr 22 2016, 5:39 PM

I think we also have a "notify my users about event x or y or downtime" task floating around somewhere else.

Doing a Phriction panel, an iframe panel, or a custom API panel would also be reasonable approaches.

chad added a comment.Apr 22 2016, 5:40 PM

We also talked about a "Rooster" system as well. Something outside of dashboards that is tied to a time or is user-closable.

Maybe T8145 / T5231, but I think the driving use case there was "notify users of Phabricator downtime" specifically. I'm imagining the driving use cases here not affecting all users and not directly impacting Phabricator, just having wide enough impact that there's interest in surfacing them more broadly. e.g., "We're having a production incident affecting video upload, ops is investigating" is important, but not "every Phabricator user must be notified" important.

(I agree that this is a bad solution for notifying users of downtime/maintenance on the current instance, and that rooster/broadcast is better.)

A few things that might motivate writing an API panel (which pulls from external sources) instead:

  • If you're trying to consolidate several sources, that'll be a bit messy when you're pushing via Conduit. The different sources could overwrite one another's updates. So you might need to write something to pull from all of them anyway, and that might as well just be the panel itself?
  • Alternatively, you could have each system publish to a separate panel, but there's no way to merge/hide panels right now. We could provide that or you could do it via the API, but that seems messier than just having the panel logic do it.
  • If you're reading an external system that updates somewhat frequently (say, a couple times an hour) you'll be generating 20K transaction logs a year on the panel. Not the end of the world, but not ideal, and worse for more frequent updates. These logs are useless, and pulling would do away with them.
chad added a comment.Apr 22 2016, 5:55 PM

Yeah in those cases, writing your own "Internal Tools Status" application would probably be better.

cinek awarded a token.Apr 23 2016, 9:51 AM
eadler added a project: Restricted Project.Apr 25 2016, 6:02 PM
eadler moved this task from Restricted Project Column to Restricted Project Column on the Restricted Project board.
eadler moved this task from Restricted Project Column to Restricted Project Column on the Restricted Project board.May 13 2016, 9:52 PM
eadler moved this task from Restricted Project Column to Restricted Project Column on the Restricted Project board.May 23 2016, 6:12 PM
eadler moved this task from Restricted Project Column to Restricted Project Column on the Restricted Project board.
epriestley added a project: Prioritized.EditedMay 23 2016, 6:33 PM

We aren't really sold on this as a specific use case, but can give you a modern dashboard.panel.edit which will let you implement it. If there are weird followups arising from this (like difficulty managing the 300,000 transactions on the panel detail a year from now) we may be somewhat less sympathetic to them than we would be to a more on-label use case.

I'm guessing this is about 4 hours of work, but I need to consolidate outstanding Dashboard stuff before committing to that. Some of the panel edit transactions (like whatever's going on with T10612) may be more involved than I recall.

epriestley moved this task from Backlog to Preflight on the Prioritized board.May 23 2016, 6:34 PM
eadler added a comment.EditedMay 23 2016, 6:36 PM
In T10855#172123, @chad wrote:

A Phriction Panel which we talked about briefly would provide this automagically without wiring Dashboards to Conduit.

We would be okay with this too.

eadler moved this task from Restricted Project Column to Restricted Project Column on the Restricted Project board.Jun 22 2016, 3:14 PM
chad added a comment.Jul 1 2016, 9:53 PM

I might be able make a Phriction Panel

chad added a comment.Jul 1 2016, 9:59 PM

No, no I can't. I can't make a Phriction Panel.

epriestley moved this task from Preflight to The Queue on the Prioritized board.Jul 2 2016, 7:24 PM

The actual panel transactions that we need to update for this aren't too bad. The board transactions are a little more involved, but we can skip them.

Panels currently use CustomFields to define fields, but EditEngine is better in that role. I'm not sure how rocky the ModularTransaction / CustomField interaction is yet, but pathway forward is something like:

  • Add an EditEngine + "pro" controller for the easy fields.
  • Convert transactions + custom fields in one huge messy change? I'm not sure how reasonable it is to split this into smaller pieces. This isn't too huge, but also not bite-sized. Since CustomFields already work with EditEngine, it's also possible we don't really need to do this and can make both work, then migrate the data in one change, then migrate all the actual fields.
  • The "Query" panel has the highly-magical query selection fields that are going to be a bit of a pain to bring over, but I don't really see a clean way to get out of it. Even if we build "Create a panel from this query...", per T5307, we still need some control that lets you edit from the panel itself.
  • Swap to ModularTransactions somewhere along the line here, maybe just at the end.
  • Then just drop in the Conduit endpoints.
eadler added a comment.Jul 3 2016, 5:51 AM

FWIW we'd be perfectly fine with the following:

  • allow editing phriction documents via Conduit
  • create a panel which shows the contents of a specific phriction page

We probably need to do almost all of this work before adding new panel types anyway, since building a new panel type on top of CustomFields would make moving everything from CustomFields to EditEngine harder.

Offhand, the only real way I see to possibly cheat and solve this faster is a dedicated MOTD-ish mechanism. I suspect we'll build something eventually -- we have a number of adjacent use cases (NUX, security notifications, maintenance notifications) but I don't really have a concrete idea of what would best solve all of these use cases yet. Such a mechanism might be a poor fit for more state-based information like "can we do deploys" or "what is currently broken", too.

epriestley moved this task from Backlog to v2 on the Dashboards board.Jul 3 2016, 9:17 PM
epriestley edited projects, added Dashboards (v2); removed Dashboards.

Turns out that CustomFields sort of just work properly with EditEngine out of the box, so D16226 provides a technically-functional dashboard.panel.edit, which allows you to edit text panels like this, for example:

$ echo '{
  "transactions": [
    {
      "type": "custom.text",
      "value": "new text"
    }
  ],
  "objectIdentifier": 32
}' | arc call-conduit --conduit-uri http://local.phacility.com/ --conduit-token <conduit-token> dashboard.panel.edit

Note that "custom.text" is that string literally, "new text" is the new value, and 32 is the panel ID.

This has a large collection of limitations and infrastructure problems:

  • You can't create panels since there's no way to specify a panel type (this requirement is somewhat common across object types, but I believe unprecedented in *.edit API methods, and might be nontrivial to resolve).
  • The API endpoint doesn't tell you which transactions are available per panel type (also unprecedented).
  • This API will change in the future: type will become something like "text" instead of "custom.text" when I convert from CustomFields to EditEngine.
  • This is backed by what is essentially a copy-pasted EditEngine version of the edit flow, so two different edit flows that share no code currently exist in master.

We'll correct these problems in future changes, but you can start using the endpoint immediately if you have uses for it. None of them should affect behavior, except the eventual change to the correct type constant.

GoalRefTimeNotes
Conduit AccessD162260.5 HoursExtremely rough API access.
Subtotal0.5 Hours
Total0.5 Hours
urzds added a subscriber: urzds.Aug 29 2016, 9:06 AM

I was looking for similar functionality (publish the state of some external system on a panel) and instead of updating the entire panel I think it would be nice to use Phlux or Paste:

Have the external system update a key/value in Phlux using conduits and then render the key inline in the panel.
Or the external system updates a Paste though conduit and then show the paste in the panel. This is already possible except for how the paste is rendered. For this to work the paste should have an option to be rendered inline, without line numbers, borders and background.

I think this is a much cleaner solution than having an external system updating all text of a panel just to display its status, especially it there is other text in the panel. It also solves the problem if you want to show the status of multiple external systems in the same panel.

stevex added a subscriber: stevex.
mves added a subscriber: mves.May 9 2017, 6:00 AM
gou1 added a subscriber: gou1.May 29 2017, 2:29 PM