Page MenuHomePhabricator

Add Conduit API methods for workboards
Closed, DuplicatePublic

Description

The conduit API for workboards is incomplete. For example, maniphest.gettasktransactions returns workboard column changes:

{
  "taskID"          : "15427",
  "transactionType" : "projectcolumn",
  "oldValue"        : {
    "projectPHID" : "PHID-PROJ-qmupmk5jjolmiykh6mjt",
    "columnPHIDs" : []
  },
  "newValue"        : {
    "projectPHID" : "PHID-PROJ-qmupmk5jjolmiykh6mjt",
    "columnPHIDs" : [
      "PHID-PCOL-cuk3i2aefn5wupau7fox"
    ]
  },
  "comments"        : null,
  "authorPHID"      : "PHID-USER-y4dtzcupomndedax6usr",
  "dateCreated"     : "1400806451"
}

but AFAICT there's no way to:

  • get the name for a column given its PHID
  • get the columns for a project
  • get the tasks assigned to a column

Context -- I'm using workboards to implement an kanban board, and want to gather sufficient data to produce a cumulative flow chart. For now I'm just hardcoding the column PHIDs for my projects of interest.

Event Timeline

raincoastchris raised the priority of this task from to Needs Triage.
raincoastchris updated the task description. (Show Details)
raincoastchris added a project: Workboards.
epriestley added a subscriber: epriestley.

It will probably be a while before we get to this, we're still changing how workboards work and any API we shipped today would probably need major revisions between now and un-beta'ing workboards.

It would be nice to add such informations, specially on manifest.info

I would find it helpful to filter the manifest query based on project column name

The column names for a task within a project are accessible through the conduit API with a little work. Here is a gist containing a python script to get the tasks associated with a project and workboard column then print them grouped by status.

https://gist.github.com/robwiss/9c7e2bcf2af063635288

I know this isn't the same as support in the conduit API but perhaps it could help some people.

The thing I can't seem to find in conduit is the ordering of tickets within columns. Anyone know how to access that?

Are workboards 'un-beta' now, and stable enough for it to be worth putting some work into this? I can't promise to have time, but I'm certainly interested in having a look at this.

Changes like T10569 should probably happen first. This is likely to be very complex, and is probably not a good task for a new contributor.

Eventual APIs here also probably don't map cleanly to generated EditEngine APIs, so I'd like to see a broader collection of use cases. I think querying columns is straightforward and moving tasks by applying transactions directly to them is worth sorting out in the context of T6027, but that leaves many features of workboards not accessible via API.

In particular, this original use case doesn't make much sense to me:

I'm using workboards to implement an kanban board

Specifically, I'm not sure how a kanban board differs from workboards or what kind of problem this is the best solution to. Since this request is nearly two years old, it's possible that the addition of point counts to columns resolved this.

For this use case:

want to gather sufficient data to produce a cumulative flow chart

...a better solution is likely consuming data from Facts after T1562. See also T4171. It is not currently possible to query the data required to produce an accurate chart efficiently, and workboard APIs won't provide this.

Thanks a lot for putting your time into replying to my comments. I am thinking squarely in the context of T6027, and was looking primarily at the bullet points here as a stepping stone towards that. i.e.

  • get the name for a column given its PHID
  • get the columns for a project
  • get the tasks assigned to a column

and merged from T10599,

  • move task between columns

Rather than a full API, these seem like low-hanging fruit to me, and at least the start of an API, and would lay good groundwork for T6027. Or am I wrong?

You're right, I don't want to approach T10569 as a new contributor. At the end of the day, I'm just searching for what's best for me to work on, that you're willing to accept, towards T6027. I know you shoulder the maintenance burden.

I also agree entirely with your concerns about the stated use case of this task.

Should I be continuing discussion/work about stepping stones towards T6027 in that task, rather than commenting in tasks which seem more strongly related to what I perceive those steps to be?

To provide some context, until recently, the API was in bad shape. Basically, the first time we wrote an application we'd maybe write some application.whatever methods, which shared some code with the rest of the application but not all of it, and then the API and application wouldn't always get changed in the same way. This resulted in methods like repository.create which can do maybe 20% of what the web UI can, does it in different, inconsistent ways, and edits made by the API might look or work differently than web UI edits. In bad cases, they'd even do different policy/permissions checks. This caused many many problems for a very long time.

With the "v3" API methods (application.edit, application.search) we've mostly fixed this. A new infrastructure component called EditEngine contains all the logic for generating edit forms, the "Actions" dropdown, and the Conduit API. Likewise, the existing SearchEngine component has expanded and now generates both the web UI and the Conduit API. While not every capability available in the UI is available in the API, this has brought them much, much closer than they used to be.

A good example of this progress is in Maniphest:

  • v1 API: maniphest.info: Fetches limited, inflexible data by ID only for a single task.
  • v2 API: maniphest.query: Fetches more, inflexible data by a wider range of parameters.
  • v3 API: maniphest.search: Fetches much more flexible data by the full range of parameters usable in the web UI.

This is the entire implementation of a v3 API method:

<?php

final class ManiphestSearchConduitAPIMethod
  extends PhabricatorSearchEngineAPIMethod {

  public function getAPIMethodName() {
    return 'maniphest.search';
  }

  public function newSearchEngine() {
    return new ManiphestTaskSearchEngine();
  }

  public function getMethodSummary() {
    return pht('Read information about tasks.');
  }

}

That it -- all the actual behavior is driven by EditEngine.

I don't want to move backward into the v1/v2 API style, so the way forward on this task isn't just "write some endpoint code that runs a query". That is straightforward, and you can copy/paste any existing older API method and drop it in src/extensions/ if you just need it for something. It would take very little time to implement projects.getcolumns(phid), projects.getstuffincolumn(phid), etc., but doing implementations like this will create many maintenance problems for a long time.

Column-driven and workboard-driven queries (where you're asking a question about a column or workboard) are complicated to implement properly because neither columns nor workboards have SearchEngine classes, and workboards don't even have their own PHIDs. We may allow workboards to be attached to other objects in the future (for example, see T5793). If we do, they'd almost certainly need to be assigned PHIDs and dedicated storage, and it would no longer make sense to have API methods in the projects.* namespace, and any APIs would probably be broken.

Moving objects between columns has similar concerns. We may allow other objects to appear on workboards in the future (for example, pholio mocks and calendar events). If we do, the transaction probably needs to become a "workboard" transaction rather than a "maniphest" transaction, and this would break existing APIs. The current structure of the transaction is also probably not where it needs to be in the long term, so the first step forward here may be to rewrite how the transaction is handled and extract it from Maniphest. This is complicated and perhaps blocked by T9789, an enormous infrastructure task.

Regardless of all of this, I don't want to implement anything without having a good understanding of why we're implementing it: what problems does it solve? In the future, we're likely to need to make changes to anything we implement, and if we don't have a record of why we built something or what problems we intended to solve with it, it's hard to change it without breaking users who are using it. The use cases for these features are not self-evident to me as the best solutions for any class of problems I can come up with.

For example, if every use case is "build charts" and that's the only thing these APIs are actually really useful for, we shouldn't waste time building or maintaining this and should pursue T1562 instead -- that's a better solution to that class of problems, and our time would be better spent solving the problem directly rather than building a workaround which lets users solve it poorly and indirectly.

eadler added a project: Restricted Project.Aug 5 2016, 5:24 PM

Just want to +1 that this would be a very useful feature for us. I think our usecase is similar: We have a lot of teams (especially ops teams) using phab workboards and columns extensively. Rather than try to shoe horn different types of reporting, etc. into phab they are relying on external systems/tools, but it is pretty painful to extract data on what column tasks currently live in.

Edit: To give a bit more detail/clarity, I just received a request to output a CSV of the task ID and title of all tasks in the In Review column of a particular project. See get a lot of this type of request and have devised ways to do it but they tend to be fairly onerous.

@raincoastchris the context of this issue caught my eye because I've really wanted Phab to generate a cumulative flow chart and Kanban statistics for some time now. I took a stab at it today and modified an existing python script to generate a cumulative flow chart and Kanban statistics (https://github.com/lyahdav/analytics-limn-analytics-data/tree/kanban_stats) but configuring it is a bit painful because there is no easy way to get the names of all the columns in a workboard right now.

I'm curious, do you have your own implementation of something similar somewhere online?

I believe that using phid.query should be helpful.

A curl like this

curl https://projects.example.com/api/phid.query \
    -d api.token=api-token \
    -d phids[0]=PHID-PROJ-abcdefghijkl

will return

{
  "PHID-PROJ-abcdefghijkl": {
    "phid": "PHID-PROJ-abcdefghijkl",
    "uri": "https://projects.example.com/tag/myproject/",
    "typeName": "Project",
    "type": "PROJ",
    "name": "# MyProject",
    "fullName": "# MyProject",
    "status": "open"
  }
}

Any PHID will just work.

I'm going to merge this into T12074. Summarizing things so far and my short-term plans:

  • Many problems described here appear to be related to charting. See T4171 (and T1562) for discussion of those use cases. We expect reporting use cases to primarily be served through Facts. As with other feature requests, it is important that we motivate feature development here by understanding which problems we are solving. I do not believe these APIs are the best solution we can provide to any charting problems.
  • Although Maniphest has not yet moved to ModularTransactions (T9789) they are now well-defined and relatively stable and I am no longer concerned that they may interact here.
  • Work elsewhere on T6027 provided a "column" action which can be applied to tasks via maniphest.edit, moving them between workboard columns. I am satisfied with this approach and with the technical underpinnings.
  • I expect to move forward with the technical plan in T11036 to handle subproject columns, so that does not directly interact here or create peril.
  • While T5474 isn't particularly well-defined, I think it's on sufficiently firm ground that I don't expect it to interact here.
  • T5793 still interacts negatively here but I'm going to move forward in a way that does not directly accommodate that use case, and effectively lock columns and boards to projects for eternity. This doesn't preclude some sort of feature like "personal boards" or letting a task have a board or letting a board have a board, but they would act through a project intermediary. I suspect this implementation is dramatically more reasonable anyway, and putting boards on anything anyone wants would be a tangled mess that was very "pure" in some sense but mostly just weighed us down with needless complexity in service of marginal use cases.
  • Although these choices largely define what editing will look like, I don't plan to implement the EditEngine or column edit endpoint yet since no one has actually requested them, as far as I can see.