Page MenuHomePhabricator

Show progress when running `arc lint`
Needs ReviewPublic

Authored by joshuaspence on Jul 7 2015, 8:13 AM.
Tags
None
Referenced Files
Unknown Object (File)
Mon, Mar 4, 6:09 PM
Unknown Object (File)
Feb 15 2024, 7:35 PM
Unknown Object (File)
Feb 14 2024, 7:11 AM
Unknown Object (File)
Feb 3 2024, 5:44 AM
Unknown Object (File)
Jan 6 2024, 1:05 PM
Unknown Object (File)
Dec 26 2023, 10:39 PM
Unknown Object (File)
Dec 26 2023, 8:09 PM
Unknown Object (File)
Dec 18 2023, 10:50 PM
Tokens
"Love" token, awarded by hach-que.

Details

Summary

Fixes T8760. Running arc lint --everything can take a long time. Add a progress bar to provide a visual indication of the progress.

Test Plan

Ran arc unit --everything and saw a progress bar.

Diff Detail

Repository
rARC Arcanist
Branch
master
Lint
Lint Passed
Unit
Tests Passed
Build Status
Buildable 7167
Build 7379: [Placeholder Plan] Wait for 30 Seconds
Build 7378: arc lint + arc unit

Event Timeline

joshuaspence retitled this revision from to Show progres when running `arc lint`.
joshuaspence updated this object.
joshuaspence edited the test plan for this revision. (Show Details)
joshuaspence added a reviewer: epriestley.

This is a bit gross and I'm open to a suggestions for a better implementation.

Alternatively, we could output lint results progressively rather than all at the end... this is probably better overall.

joshuaspence retitled this revision from Show progres when running `arc lint` to Show progress when running `arc lint`.Jul 7 2015, 12:44 PM
joshuaspence edited edge metadata.

Alternatively, we could output lint results progressively rather than all at the end... this is probably better overall.

This is probably significantly more work but I am happy to work on it if it is deemed to be a better solution.

This is probably significantly more work but I am happy to work on it if it is deemed to be a better solution.

I think this is possibly not desirable in normal use (e.g., without --everything). Particularly, it means we can not sort results (e..g, put test failures at the top/bottom or put all lint messages in the same file together). I'm not sure we actually do that stuff right now, but I'd like to sort test failures to the bottom at some point since hunting for them in moderately-large output is a bit of a pain right now.

Conversely, the progress bar doesn't seem very desirable for CI, since it isn't diagnostically meaningful and will log a bit weird. Is the intended consumer here CI or humans?

This progress bar is also probably very nonlinear in many cases (e.g., in this codebase the XHPAST linter is like 90% of runtime but will be 10% of the bar, I think?) so it's probably not very useful for predicting remaining time.

The intended consumer is for humans. At the moment, I am doing some large sweeping changes over our codebase and applying various linter fixes (essentially, one linter rule at a time). Currently I run git ls-tree -r --name-only HEAD | grep -E '\.php$' | xargs -n128 arc lint --lintall --apply-patches instead of arc lint --everything, purely so that I get some level of indication as to what is going on.

Could you run arc lint --everything --trace instead? If "almost", could we add more logging there to make the answer "yes"?

Running arc lint --everything in our repository takes a very long time, maybe 3-4 hours on my lapotp. One of the problems with arc lint --everything is that all of the output (and autofixes) are done at the end of the process. This means if I want to terminate the process after an hour or two then I have wasted time. This is another reason I use the git ls-tree hack currently.

I suppose that the progress bar doesn't really solve any of the issues that I am having, it just seemed like a nice to have. That is, I think that it is odd that a long-running command provides no output to indicate that it is actually running. I feel that without some sort of visual indication the user is likely to suspect that the command has hung.

On my machine, it looks lik e`phabricator/` takes about ~14m -- do you have a sense of how much slower your runtime is because your codebase is way larger, vs you're doing more linting, vs some combination of both / something else?

I think most of it is external libraries actually. We have one particular external library which has a 27,000 line PHP file. The PHP file is so large that I can't render it in the PHPAST application. It looks like it takes over an hour just to lint this library. Admittedly, we shouldn't be linting external code, but that's a different story.

Any update on when this change can be merged?

src/lint/engine/ArcanistLintEngine.php
558

@joshuaspence, it would provide more up to date progress information if you'll update progress for each processed path in executeLinterOnPaths method. This way you'll see progress bar update not after chunk of paths were processed by a single linter, but after single path was processed.