Analysis workflows for Unit Tests information
Open, Needs TriagePublic

Description

We gather a lot of information about unit tests, but some of it is hard to extract.

  • When looking at a branch, where each commit has a build, and the last 5 commits are broken, there's no easy way to understand if the same test is failing for all of them (Because someone broke it and nobody fixed it), or each one has it's own fail (Because the tests are flaky).
  • Detecting Flaky tests is hard. Developers are supposed to say "I didn't break this test, it's flaky" and file a task, but more often they say "I didn't break this test, not my problem," followed closely by "The build is failing, but it's probably not my new code that I just invented that's wrong, so I'll just land this and move on". </rant>
  • Some way of tracking test performance over time / commit history, or just see the slowest ones in a build, to find the ones that need improvements.
avivey created this task.Dec 16 2016, 7:54 PM

I think these are all reasonable things to want to do.

T9524 is sort of adjacent product-wise, although I think there's not much technical overlap.

eadler added a project: Restricted Project.Jan 1 2017, 3:15 AM