Page MenuHomePhabricator

Report on team velocity
Open, Needs TriagePublic

Description

This is a followup to the Conpherence discussion about documenting WMF reporting use cases; it is intended to support Phacility's research into implementing reporting.

Use Cases

  • As a Product Manager, I want to compare my gut feeling of amount and type of team activity over the last quarter versus reality (as measured by Phabricator task completion) so that I can get a reality check and adjust my gut feeling.
  • As a Product Manager, I want to find out if a team's recent velocity is down or unplanned work is up, so that I can decide how to triage upcoming work.
  • As a Team Manager, I want to see levels and trends in team velocity so that I can make better informed staffing level decisions.

Sample solution

Design and Implementation issues

  • The fundamental issue in providing this information to users is, what is the scope of tasks to be reported on? The general answer is, all tasks that my team is responsible for and none that they aren't, but defining this in Phabricator has varied from "all tasks in project X" to combination of projects to combinations of intersections of projects and even more complicated arrangements. If a report supported only "all tasks in project X", this inflexibility would probably simplify the effort of getting different teams to use consistent processes.
  • The division of velocity into subcategories is, in the sample, based on fairly complicated criteria.
  • Some teams like reporting by sum of points, but most don't use points and so count is the only metric available.
  • The sample report reports by rolling time periods (e.g., the last ~30.5 days, not "March to date"), for two reasons: teams like seeing feedback is quickly as possible, as a sort of "reward" for closing stories; and because it keeps all data points comparable.
  • Teams have found value in weekly, bi-weekly, monthly, and quarterly summaries. Monthly is probably the single most universally useful resolution, because it is small enough to provide feedback in time for teams to use it, but large enough to hide a lot of noise.

Event Timeline

ablekh added a subscriber: ablekh.Mar 15 2017, 11:21 PM

@jaufrecht Nice description. A quick comment on the attached image. Please take no offense, but IMHO, from the UX / usability perspective, this approach (using the same color's gradient or, in your case, shades) is pretty terrible and, as far as I know, is recommended against as the data visualization best practice. Hope this helps.

@ablekh Thank you.

Regarding using color gradients on the chart, do you mean that the problem is that users cannot accurately match shade 17, for example, from the legend to shade 17 on the chart? If so, then I should probably clarify the use cases that have led us to this design. (Note that this task is not a request for the sample chart pictured; we've found that the sample chart satisfies the use cases shown, but are open to other solutions. And the use cases can be partly satisfied without any subcategorization.)

In the sample chart, the shading from light to dark reflects user-specified priority. When our users look at the colors, they are looking for trends: Has the velocity chart gone from light to dark? If not, then the team hasn't been progressing consistently or the work planning has been out of order or inconsistent. Is there a blob of light color in isolation on the right side? If so, why are we doing low-priority work - does an old project need to be halted, or is the work mis-prioritized (in which case we might need to re-triage it)? Users aren't trying to find out e.g. what the third color bar from April means—if they were, I think we'd need labels on the bars, or an interactive report, or both.

Given those priority-based use cases, do you have a more effective approach in mind? Or are you pointing out a different issue?

Thanks

ablekh added a comment.EditedMar 15 2017, 11:54 PM

@jaufrecht My pleasure. In general, using color gradients or shades is fine (and, actually, pretty much standard) for emphasizing trends. However, I would expect a use of gradients or shade of a single color to visualize a trend of a single category (topic, subject). In your case, though, the image presents trends in that fashion for multiple categories, which makes them difficult to distinguish and track. I would prefer and suggest using different colors for different categories and using a particular color's gradients or shades for each category across the time. Does it make sense to you?

That does make sense; we're using that multi-color approach for the report in the example in https://secure.phabricator.com/T12403. But we also put labels per-category, which is possible in that report because the categories are continuous over time to form ribbons, unlike this chart. For this chart, I think the mental model is that all categories are really sub-categories of the parent category "stuff we have worked on", and users only need to distinguish between high-priority and low-priority, hence the shading.

In fact, for all three use cases above, there is a simple version of the use case that is fulfilled with no sub-categories at all, just a total count per time period. Then there's a more complex one where users want to differentiate between high and low priority, and the shading solves that. Then there's the maximal cases, where users actually want to know what exactly happened data point by data point. We tried using different colors, but once there are more than ~10 categories, it was impossible to make out, so we don't have a sample solution for that.

pasik added a subscriber: pasik.May 12 2018, 1:48 PM