Page MenuHomePhabricator

Multiple coverage columns for different types of code coverage
Open, Needs TriagePublic

Description

At my job we mainly use Python, an untyped language. However, we've recently started introducing PEP 484 / MyPy annotations, a compile-time layer that lets us add optional static typechecking to our codebase. This leads to files that have some statically-typed lines, and some lines that are not statically-typed.

It would be nice to have a second coverage column in our code reviews (in addition to the test coverage column) to track whether or not something is statically typed.

Other potential usecases I can think of for >1 coverage column:

  • A similar use case can be applied to TypeScript, which is another partially-statically-checked environment
  • Splitting up "is this line covered by unit tests or integration tests?"
  • For cross-platform libraries, it would be great to have, for instance, "is this covered in Win32 tests?" or "is this covered in OSX tests?" columns

Some background:

You can sort of get an idea for what the type coverage column would be used for based off this sample:


(This is a real HTML report generated by MyPy and would be trivial to convert into the N-U-C style that ArcanistUnitTestResult setCoverage() uses).
typed_function is typed because it has a PEP 484-comatible # type: ... comment. It takes in an int, and outputs a boolean.
untyped_function is untyped because it doesn't have the # type: ... comment.
main_fn is straightforward until we reach line 12 - here, since we call into an untyped function, the return type is uncertain; hence 12, 13 and 14 are marked as 'untyped' (red).


The crux of this request is that I would like to be able to display this data in Arcanist in addition to the existing code coverage column, as (poorly) mocked up here:


(The first column being "is this line covered in a test?," the second column being "is this line completely typechecked?" Colors don't have to be customizable, it's just for readability.)