Page MenuHomePhabricator

Treat "skipped" unit tests as less interesting than "passed"
ClosedPublic

Authored by epriestley on Mar 1 2016, 2:05 PM.
Tags
None
Referenced Files
Unknown Object (File)
Fri, Dec 13, 8:52 PM
Unknown Object (File)
Wed, Dec 11, 7:14 PM
Unknown Object (File)
Wed, Dec 11, 7:07 PM
Unknown Object (File)
Sat, Dec 7, 9:52 AM
Unknown Object (File)
Thu, Dec 5, 11:53 PM
Unknown Object (File)
Thu, Dec 5, 11:27 PM
Unknown Object (File)
Nov 9 2024, 5:23 PM
Unknown Object (File)
Oct 27 2024, 12:55 PM
Subscribers
None

Details

Summary

Ref T10457. Skipped tests are almost always well-behaved (e.g., testWindows(), but the test is running on Linux) and not interesting, and we do not expect well-written, solid systems to necessarily have 0 skips.

Although skips could indicate that you have missing dependencies on a build server, and thus be a bit interesting, I think they almost always indicate that a particular test is not expected to run in the current environment.

If we wanted to tackle this problem in granular detail, we could eventually add a "Missing" status or similar which would serve as "a skip you could reasonably fix in this environment", but I don't think that's too interesting.

Test Plan

Here's an example of a build result with skips: B10875

Screen Shot 2016-03-01 at 6.04.22 AM.png (325×525 px, 44 KB)

I think this is clearer as "Passed", as this is the expected production state of the build.

Locally, looked at some builds.

Diff Detail

Repository
rP Phabricator
Branch
dry13
Lint
Lint Passed
Unit
Tests Passed
Build Status
Buildable 10898
Build 13448: Run Core Tests
Build 13447: arc lint + arc unit

Event Timeline

epriestley retitled this revision from to Treat "skipped" unit tests as less interesting than "passed".
epriestley updated this object.
epriestley edited the test plan for this revision. (Show Details)
epriestley added a reviewer: chad.
chad edited edge metadata.
This revision is now accepted and ready to land.Mar 1 2016, 2:46 PM
This revision was automatically updated to reflect the committed changes.