Page MenuHomePhabricator

Plans: Portals and Facts
Open, NormalPublic

Description

Setup Work

Prior to building Facts, I'd like to put Dashboards/Panels on firmer footing. This is mostly fixing bugs and making minor improvements, although I'd also like to build "Portals" while we're here.

  • See T13272 for general improvements to existing dashboards and panels.
  • See T13275 for "Portals". A portal is a page with a list of dashboards and other resources in a menu.

Facts and Charts

I'd like to schedule this alongside a MVP of Facts and get a "Chart" panel built at the same time. See T4171 / T1562. Outstanding use case for Facts:

PHI203 describes some specific use cases, but I think the corresponding toolset to build for it is just the straightforward "agile" tools.

In one use case here, a team wants to be better at predicting whether they'll hit a relatively near-term deadline or not ("we often end up in situations where we are 75% of the way through the sprint, but have only done 25% of the planned commitments"). Conceptually, I'd like to imagine an experienced project manager (and, really, every member of an experienced technical team) should probably "know" this with more accuracy than a tool can provide, although this may be unfair. But even if the human understanding of this is better than the chart understanding, I think having a chart can make communication about velocity much easier, and may be a useful tool in working with engineers to convince them that their planning skills could use improvement while framing it as a challenge to overcome rather than a nagging confrontational mess. If nothing else, a chart can be a reality check against the temptation to overpromise.

T12459 asks for forecasting. I don't plan to pursue this initially because I believe forecasting is very difficult and I'm hesitant to ship a product which makes claims about the future without having any kind of feedback loop so we can verify that the forecasts are accurate or work to correct them. If charting sees significant use, we may be able to pursue this in the future. Simple forecasts, like "will this project ever complete", tend to be obvious from the shape of the chart anyway since you can look at it and see that the trend line is headed up or down. Since part of Facts will also just be API access to datapoints, installs could build this externally in the short term.

T12410 asks for some relatively specific reports, including a not-exactly-time-series report. I don't expect to pursue these use cases in the near term. Other strategies exist for finding long-surviving tasks within a project (query open tasks ordered by date created).

T12403 asks for a specific chart based on relatively straightforward data. We probably won't be able to produce this chart for now, but I expect to be able to produce the data.

T12177 is a different request and asks for charts around review lifecycles. The general goal here is to encourage good engineering behavior (like small, well-separated changes) by communicating that you'll get changes reviewed faster if you do the legwork to set them up properly (maybe this isn't true universally, of course). Another goal is to put social pressure (and, perhaps, performance review pressure) on teams and individuals with poor review participation and highlight teams and individuals who are responsive.

The latter goal is somewhat dangerous -- "Hall of Heroes" essentially led to open revolt -- but I suspect we can ease into this with little contention.

An upstream goal is to replace the awful ManiphestReports view with something maintainable.


Generally, I expect to focus on building a fairly traditional "burndown" chart first. This will let us replace the Maniphest reports view, generally speaks to most use cases we've seen, and should generally cover most of the basic types of fact extraction and reporting that other charts will eventually need.

The facts we need to extract to render a burndown chart are:

  • Scope changes: tasks created into a project, later tagged with a project, or removed from a project. (For users, assignment changes.)
  • Estimation changes: point value changes on tasks. (For users, point value changes for assigned tasks).
  • Completions/resurrections: tasks moved to a "closed" state from an "open" state, or to an "open" state from a "closed" state.

Errata:

  • The datapoint generation workflow should check for (and reject) duplicate datapoints since these are likely errors, at least until we encounter a case where they aren't errors.

Related Objects

Event Timeline

epriestley triaged this task as Normal priority.Feb 15 2018, 8:52 PM
epriestley created this task.
  • Fact cursors are currently per-object. I think they should likely be per <object, engine> pair so that subsets can be rebuilt more quickly for testing/development and the failure of one engine will not stop all other engines from processing data. On the other hand, fact processing is currently pretty fast and I don't expect to write a ton of engines so I'm not sure how critical this really is. Engines likely don't care how cursors work unless they depend on facts extracted by other engines and need to guarantee they execute later.
  • FactEngine->getFactSpecs() isn't really authoritative -- it takes a fact definition externally. I think this should become the authoritative generator. That is, this piece of code needs to go sooner rather than later:
$types = array(
  '+N:*',
  '+N:DREV',
  'updated',
);
  • The factRaw table has a text32 column for fact type and two varchar(64) columns for PHIDs. These might be better handled by creating a separate dimension table to reduce the size of each row in the raw fact table. This probably isn't a big deal to do later, although it would be nice to do it in the shorter term. If this was moved to a dimension table, it's possible that a given PHID could have different dimensional representations when used by different facts. I think this is probably confusing and not useful, but something to keep in mind.
  • The aggregation stuff is currently tough for me to wrap my head around and probably needs clearer use cases. I think "aggregation" also currently refers to a mixture of "aggregation across dimensions" (e.g., how much did scope increase today across all projects?) and "accumulation" (e.g., how many objects have ever been created since the beginning of time?). Since these are both just performance-oriented changes I expect to de-emphasize pursuing them until we can generate useful charts with performance issues, then move runtime aggregation/accumulation to the extractor.

One overarching issue here is that facts don't have a clear permissions model. It's bad if anyone can go look up the burndown chart for #critical-security-issues and then correlate that chart against tasks showing up in #weird-things-that-happen-when-you-hold-down-the-letter-a or whatever and get a hint about a security concern. In the general case, charts like "Salary by Employee", "Acuisitions Affecting our Stock Price", etc., are also potentially problematic.

It's also impractical to run policy checks on each datapoint, since the thing will be unusable if we do. This is inevitably going to mean that Facts is policy-violating to some degree.

Some approaches we could take include:

  • To query datapoints by dimension (like a project or user), you must be able to see that dimension. But maybe this needs to be a separate/stronger check, since many users may be able to see #critical-security-issues, just not see the tasks it contains. And everyone can see other users.
  • Actual facts (like "salary") could have policies. This is probably mostly relevant for third-party objects with particularly valuable data or third-party use of Facts as a charting application, both of which are probably far away.
  • We can restrict the ability to generate charts with novel datasets, either in general ("by default, only administrators can create a new chart") or on a dimension or fact basis ("only users who can edit #critical-security-issues can create new charts using datasets which select dimensional data for it"). Users who could see the dimensions could still view charts created by someone else, they just couldn't go into Facts and whip up their own version of the chart.
  • We could make charts that intentionally lie to users. However, applying deterministic noise which resists filtering seems difficult.
  • We could restrict the ability to see granular data (e.g., only weekly aggregations or bigger) or to see recent data.

(A "create rough version" action which optionally: strips axis labels; mutates the data a bit randomly; anonymizes all dimensions; and re-renders the chart in a hand-drawn XKCD-style might be interesting -- like "I want to illustrate my point by showing an image ofthis in a blog post, but I don't want to criticize individuals or actually disclose the exact numbers".)

I'm just going to keep an eye on this for now and do the obvious things (e.g., you have to be able to see something to see dimensional facts about it) but this may need some kind of real solution before Facts can leave prototype.

See PHI385. I think I stabilized the dashboard arrangement issues in D19123, but the arrange workflows are still in somewhat rough shape.

Just to briefly touch this:

The actual fact extraction seems to mostly work correctly: https://secure.phabricator.com/fact/chart/?y1=tasks.count.create

There's also a page which shows which facts are extracted from each object to make debugging the extractors easier: https://secure.phabricator.com/fact/object/T13083/

I'm planning to do some work on the UI next so that it's possible to produce at least basic, plausibly-useful charts. The actual bit where you specify a chart (series, range/domain, colors, labels, title, etc.) might be a bit "big blob of JSON" for now. Ideally, I get this just good enough to replace Maniphest Reports, delete them, kill scope on Facts there, then build "Portals".

Not sure if it's expected in the current state, but here's one that doesn't seem well-formed: https://secure.phabricator.com/fact/chart/?y1=tasks.open-count.status

The task.open-count.status counts open tasks that are the result of a status change (an existing task being closed, or a closed task being reopened). Since it's much more common for existing open tasks to be closed than for existing closed tasks to be reopened, the series should go negative pretty quickly and stay there over time, more or less ending up with a big negative number of "number of tasks ever closed".

I think the UI just has problems drawing this right now. You can see that the Y axis has kind of tried to do something, but it's going in the wrong direction (with -6 at the top of the screen and a line going off into the sky). This might be a real issue with the data, too, but the UI is definitely getting something wrong, so I think it might not be any more complicated than fixing the UI to handle negative values properly.