Page MenuHomePhabricator

Pull Git filesize logic into a separate LowLevel query and use more Iterators
ClosedPublic

Authored by epriestley on Nov 22 2018, 12:22 AM.
Tags
None
Referenced Files
F18748772: D19830.diff
Sat, Oct 4, 12:36 AM
F18733855: D19830.id.diff
Tue, Sep 30, 10:43 PM
F18712562: D19830.id.diff
Mon, Sep 29, 5:27 AM
F18702418: D19830.diff
Sun, Sep 28, 12:02 AM
F18568969: D19830.id47390.diff
Wed, Sep 10, 12:32 AM
F18568968: D19830.id47355.diff
Wed, Sep 10, 12:32 AM
F18527130: D19830.id.diff
Sat, Sep 6, 6:24 PM
F18509949: D19830.id.diff
Sep 5 2025, 3:42 AM
Subscribers
None

Details

Summary

Depends on D19829. Ref T13216. See PHI908. The current implementation is kind of a lot to live in CommitHookEngine and will likely fail if git diff-tree produces more than 2GB of output.

Pull it out and make it slightly more robust against enormous commits. It's probably limited by this, now:

implode("\n", $every_path)

We could replace that with some PhutilReverseRopeSource primitive or something but since we don't have one of those and it seems unlikely that we'll hit this case in practice, I left it here for now with just the easy stuff converted to be stream-oriented.

Test Plan

Used this script to test the query against various commits, got good results:

<?php

require_once 'scripts/init/init-script.php';

$viewer = PhabricatorUser::getOmnipotentUser();

$repository = id(new PhabricatorRepositoryQuery())
  ->setViewer($viewer)
  ->withCallsigns(array('P'))
  ->executeOne();

var_dump(
  id(new DiffusionLowLevelFilesizeQuery())
    ->setRepository($repository)
    ->withIdentifier($argv[1])
    ->execute());

Used this to find large commits in history and pull filesizes (worked great, although our largest commit only touches a couple thousand paths):

for hash in `git log --format=%H`; do echo -n $hash; echo -n ' '; git diff-tree -r --no-commit-id $hash | wc -l | awk '{print $1}'; done | awk '{print $2 " " $1}' | sort -n

Diff Detail

Repository
rP Phabricator
Lint
Lint Not Applicable
Unit
Tests Not Applicable