Page MenuHomePhabricator

repo update fails for repos with 1000s of closing branches
Open, Needs TriagePublic

Description

I added two repositories (one medium sized - 2493 commits, and a large one - 30750 commits, 4100 branches), to phabricator, which runs on a Ubuntu virtual machine.
Everything works fine with the medium sized one, but the large repository gives an Update error in the diffusion interface after everything was imported successfully. Manually triggering the update gives a more detailed error message:

$ ./repository update R8
[2016-04-28 14:30:42] EXCEPTION: (CommandException) Command failed with error #127!
COMMAND
git log --format='%H' '452c5e0a060b7a3f8a7d7a716e075b9340e7b7ac' --not 'b73ecc96d189a4b1bce533f69949ceecfc313e63' '9cb3ad03fa6f67aadf813e7559e553ecb76eeffa' 'dac0122184f2292c69001f48f079d80f6b07bb66' '2e0dbd5538723fe324fa3eab7b65da78959c4bc0' '060a01c3503f4dfe457a81e3607cf100ba9b820a' 'ae8b7f83f17fd64949bfba54dc60ffc0cf43e522' '7a603b1e4010ddbc57ccec3a5f188b5122efa520' 'd32ab0149f57e4ab8d91f88ed63fe960c7cb6cc9' '5715350376715796c7e6725b47186e4dd285f144' 'e67108d89f7fe7c9292212581f47d5debd7d0e6a' '9a7fd3f4fb9b7eb0af879360e5516a5a09045edf' '56b42ce22ff63bdc6a2a2c1548e10ffecec6abe5' 'bf4b424e76f501bf4817ab418741d94045af5ae8' '9c8db4b7d35727e5f82faa3fe62a30d0e0d537fc' '0a4ce7da26bfcfb136656861b0d4b27126a512cc' 'e8935abec2bec98736229cbfe68e9c9934f8c932' '8ab6c3924ac2ffa5acb27aaf93ff6968006c33cb' '8081f142c5d7bc2b992aac2dfef373d9fd0d8f7c' '9d91a643ef65166f4b93a4fe896145f61ed1710b' 'dd7dda1d4f8a7b68a1ce6092b7e110c5c4764eb6' '530345fe839ba9de102c447f0c30d9eef1755dd5' 'ad0e3972468a967556b513b72... (191,581 more bytes) ...

STDOUT
(empty)

STDERR
(empty) at [<phutil>/src/future/exec/ExecFuture.php:416]
arcanist(head=master, ref.master=737f5c0df976), phabricator(head=master, ref.master=abf37aa97947), phutil(head=master, ref.master=a46f7b7e5d80)
  #0 ExecFuture::resolvex() called at [<phabricator>/src/applications/repository/storage/PhabricatorRepository.php:479]
  #1 PhabricatorRepository::execxLocalCommand(string, string, string, array) called at [<phabricator>/src/applications/repository/engine/PhabricatorRepositoryRefEngine.php:361]
  #2 PhabricatorRepositoryRefEngine::loadNewCommitIdentifiers(string, array) called at [<phabricator>/src/applications/repository/engine/PhabricatorRepositoryRefEngine.php:260]
  #3 PhabricatorRepositoryRefEngine::updateCursors(array, array, string, array) called at [<phabricator>/src/applications/repository/engine/PhabricatorRepositoryRefEngine.php:73]
  #4 PhabricatorRepositoryRefEngine::updateRefs() called at [<phabricator>/src/applications/repository/management/PhabricatorRepositoryManagementUpdateWorkflow.php:160]
  #5 PhabricatorRepositoryManagementUpdateWorkflow::updateRepositoryRefs(PhabricatorRepository) called at [<phabricator>/src/applications/repository/management/PhabricatorRepositoryManagementUpdateWorkflow.php:99]
  #6 PhabricatorRepositoryManagementUpdateWorkflow::execute(PhutilArgumentParser) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:410]
  #7 PhutilArgumentParser::parseWorkflowsFull(array) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:303]
  #8 PhutilArgumentParser::parseWorkflows(array) called at [<phabricator>/scripts/repository/manage_repositories.php:22]

I wouldn't be surprised if the error is caused by the fact that the commandline is too long, since the limit on my system indeed seems to be less than 191,000 bytes.
Anyhow, since the theoretical POSIX smallest allowable upper limit is 4096, shouldn't phabricator use xargs or something for commands like this?

Version info:

phabricator abf37aa9794790fbc0e215102519e9ffc7e99b7e (Wed, Apr 13)
arcanist 737f5c0df976fe2b3178aac6ab7feb3d3621d99e (Sat, Apr 9)
phutil a46f7b7e5d8073a91e89f7facc9fea62baaca7e8 (Sun, Apr 10)

Event Timeline

How many branches/tags in the large repository?

$ git branch --list -a | wc -l
4547
$ git tag  --list | wc -l
13189

does ./repository update work for the smaller repo?
I don't think that's an issue with the length of the command (I'd expect a different exit code - 126, and I think our command execution bypasses this limitation). Exit code 127 might mean that when you invoke it in this way, it can't find git.

OK, I was able to reproduce this; The key is to have 1000s of autoclose branches.

avivey renamed this task from Git log command fails on large repository to repo update fails for repos with 1000s of closing branches.Apr 28 2016, 7:32 PM
avivey updated the task description. (Show Details)

Interestingly, if I cut and paste the whole command (Over 4200 hashes) on the command line, it works fine (I think my command line limit is at 2MB on Ubuntu 14.4).

@Chiel92: The "workaround" is probably to limit the autoclose to just master (Repository pageEdit RepositoryEdit Branches).

@avivey Turning off the autoclose feature, fixed the error. I'm fine with this. Thanks for the pointer!

I actually encouter the same problem. Setting autoclose on master only fixed it.