Page MenuHomePhabricator

Provide a straightforward way to create and update CI test statuses for a revision
Closed, ResolvedPublic

Description

For a given revision, we want to run a bunch of tests on our CI infrastructure and report the results of those tests back on the revision page.

For example, we want to know the following things about a patch when it is applied to our project:

  • Does the project compile?
  • Do unit tests pass?
  • Do integration tests pass?

Each one of these can be tested and reported on as the results become available.

It's currently possible to report on this stuff for a revision by creating a custom unit test engine and adding "postponed" test results for each of the things you want to report on.

This is quite heavy-handed, though.

It would be better to have a direct way to create and update test results on a revision after the revision has been created. Evan mentioned something to this effect on #phabricator.

I'll add one thing to Evan's comments there, which is that it would be useful to be able to create a test before the results are available. That way reviewers can see that such and such tests are still running and their results are pending.

Related Objects

StatusAssignedTask
DuplicateNone
Resolvedepriestley
Resolvedepriestley
Resolvedepriestley
Resolvedepriestley
ResolvedNone
Resolvedepriestley
Resolvedepriestley
Wontfixhach-que
Resolvedhach-que
DuplicateNone
Resolvedepriestley
DuplicateNone
Resolvedhach-que
ResolvedNone
ResolvedNone
Duplicatehach-que
OpenNone
InvalidNone
Resolvedhach-que
Resolvedepriestley
OpenNone
Resolvedhach-que
OpenNone
ResolvedNone
DuplicateNone
Resolvedhach-que
Wontfixepriestley
Wontfixepriestley
Wontfixepriestley
ResolvedNone
Resolvedepriestley
OpenNone
OpenNone
OpenNone
Resolvedepriestley
Resolvedbtrahan

Event Timeline

nchammas raised the priority of this task from to Needs Triage.
nchammas updated the task description. (Show Details)
nchammas added subscribers: epriestley, avive.
nchammas added a subscriber: nchammas.

I'll add one thing to Evan's comments there, which is that it would be useful to be able to create a test before the results are available. That way reviewers can see that such and such tests are still running and their results are pending.

@epriestley, just to double check regarding this point:

Is there any way today to mark a unit test as "Started", or are we limited to "Postponed", "Pass", and "Fail"?

Harbormaster can show that a build is running. If a build is running, arc land will warn you that there's still a running build, too. There's no test-specific mechanism on top of this and we probably won't build one (Harbormaster builds have plenty of other status information, and users who want more granular information about status can click on the build).

epriestley triaged this task as Normal priority.May 22 2015, 5:58 PM

I'll add one thing to Evan's comments there, which is that it would be useful to be able to create a test before the results are available. That way reviewers can see that such and such tests are still running and their results are pending.

This isn't documented yet and probably needs some adjustments, but you can now post partial results (e.g., update a build immediately as tests complete). Users can also see that the build is running more clearly now (they could already do that before, but we pull it in more directly now).

Is this sufficient for your use case, or is the ability to create placeholders for individual tests still important?

Is this sufficient for your use case, or is the ability to create placeholders for individual tests still important?

@epriestley - I'll have to defer to others to comment as I am not currently in a position to evaluate Phabricator, having moved on from the team where I was using it.

Ah, thanks! I think we're probably in fairly good shape once everything is properly documented so I'll plan to close this out and we can look at placeholder results in the future if there's a compelling need for it from somewhere else.