Page MenuHomePhabricator

Advice on tracking benchmarks/performance per diff or commit
Closed, ResolvedPublic

Asked by jack on Oct 15 2015, 4:00 AM.

Details

We have benchmarks we can perform on code and it's somewhat ambiguous where to store and surface this. Ideally there would be some database that tracks some performance metric against commits or diffs. My current thinking is to run these as a commit hook on the staging area and just add a message to diffs about the results. Phabricator has lots of applications. Would this make a reasonable addition/feature request: a database to store performance metrics tagged to commits or diffs and then a visualization of these over time or per diff/commit?

Does there exist open source software that can do this currently?

Answers

epriestley
Updated 1,465 Days Ago

If you just upload your benchmarks as "unit tests" with the cost as the "duration", I think that should be a mostly-reasonable way to store the data. You'd have to write custom charting on top of it today, though.

I think charting is a reasonable thing for the upstream to eventually do (even for non-benchmark tests, it's reasonable to show how much each test costs over time, and how the total cost of the test suite is changing) but charting stuff is generally a major project entangled with Facts / T1562.

This would also let you "fail" a benchmark if it took too long, and have that failure directly show up on revisions before they landed to warn authors/reviewers about performance regressions.

New Answer

Answer

This question has been marked as closed, but you can still leave a new answer.