Page MenuHomePhabricator

Treat "skipped" unit tests as less interesting than "passed"
ClosedPublic

Authored by epriestley on Mar 1 2016, 2:05 PM.
Tags
None
Referenced Files
Unknown Object (File)
Mon, Apr 1, 6:53 PM
Unknown Object (File)
Mar 5 2024, 9:26 AM
Unknown Object (File)
Mar 5 2024, 9:26 AM
Unknown Object (File)
Mar 5 2024, 9:26 AM
Unknown Object (File)
Mar 5 2024, 9:26 AM
Unknown Object (File)
Mar 4 2024, 7:06 PM
Unknown Object (File)
Feb 13 2024, 9:46 AM
Unknown Object (File)
Feb 11 2024, 4:03 PM
Subscribers
None

Details

Summary

Ref T10457. Skipped tests are almost always well-behaved (e.g., testWindows(), but the test is running on Linux) and not interesting, and we do not expect well-written, solid systems to necessarily have 0 skips.

Although skips could indicate that you have missing dependencies on a build server, and thus be a bit interesting, I think they almost always indicate that a particular test is not expected to run in the current environment.

If we wanted to tackle this problem in granular detail, we could eventually add a "Missing" status or similar which would serve as "a skip you could reasonably fix in this environment", but I don't think that's too interesting.

Test Plan

Here's an example of a build result with skips: B10875

Screen Shot 2016-03-01 at 6.04.22 AM.png (325×525 px, 44 KB)

I think this is clearer as "Passed", as this is the expected production state of the build.

Locally, looked at some builds.

Diff Detail

Repository
rP Phabricator
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

epriestley retitled this revision from to Treat "skipped" unit tests as less interesting than "passed".
epriestley updated this object.
epriestley edited the test plan for this revision. (Show Details)
epriestley added a reviewer: chad.
chad edited edge metadata.
This revision is now accepted and ready to land.Mar 1 2016, 2:46 PM
This revision was automatically updated to reflect the committed changes.