Split large path lists into blocks when linting

Authored by epriestley.


Split large path lists into blocks when linting

Fixes T5097. When linting a large list of paths (e.g., with --everything), do this internally:

$chunks = array_chunk($paths, 32);
foreach ($chunks as $chunk) {

This keeps the advantages of parallelism and artifact sharing for future-based linters, without having memory usage grow in an unbounded way.

These callbacks changed:

  • willLintPath(): Useless, no meaningful implementations. Internalized the required side effect and broke the hook.
  • didRunLinters(): Now useless, with no meaningful implementations. Broke the hook.
  • didLintPaths(): New hook which executes opposite willLintPaths().
  • lintPath(): Linters no longer need to implement this method.

XHPAST now has an explicit way to release shared futures.

These minor changes also happened:

  • Formalized the "linter ID", which is a semi-durable identifier for the cache.
  • Removed linter -> exception explicit mapping, which was unused. We now just collect exceptions.
  • We do the canRun() checks first (and separately) now.
  • Share more service call profiling code.
  • Fix an issue where the test harness would use the path on disk, even if configuration set a different path.

Test Plan:

  • Ran arc lint --everything in arcanist/.
    • With no chunking, saw unstable memory usage with a peak at 941 MB.
    • With chunk size 32, saw stable memory usage with a peak at 269 MB.
    • With chunk size 8, saw stable memory usage with a peak at 180 MB.
  • Ran with --trace and saw profiling information.
  • Created this diff.

Reviewers: joshuaspence

Reviewed By: joshuaspence

Subscribers: epriestley

Maniphest Tasks: T5097

Differential Revision: https://secure.phabricator.com/D12501