Page MenuHomePhabricator

Indexing search documents uses a lot of memory
Closed, ResolvedPublic

Description

josh@ip-10-182-88-66:/usr/src/phabricator$ ./bin/search index --all --type CMIT --background
Indexing 148147 object of type CMIT.
[===================-                                              ]  29.5%[2015-06-17 13:47:55] EXCEPTION: (Exception) Failed to passthru proc_open(): proc_open(): fork failed - Cannot allocate memory {>} (Exception) Failed to passthru proc_open(): proc_open(): fork failed - Cannot allocate memory at [<phutil>/src/future/exec/PhutilExecPassthru.php:99]
arcanist(head=master, ref.master=9a7c4d87a850), phabricator(head=master, ref.master=ca8adac1eb40), phlab(head=master, ref.master=36469f5efae6), phutil(head=master, ref.master=3b4da3f33153)
  #0 <#2> PhutilConsoleProgressBar::eraseLine() called at [<phutil>/src/console/PhutilConsoleProgressBar.php:145]
  #1 <#2> PhutilConsoleProgressBar::draw() called at [<phutil>/src/console/PhutilConsoleProgressBar.php:74]
  #2 <#2> PhutilConsoleProgressBar::redraw() called at [<phutil>/src/console/PhutilConsoleProgressBar.php:64]
  #3 <#2> PhutilConsoleProgressBar::update(integer) called at [<phabricator>/src/applications/search/management/PhabricatorSearchManagementIndexWorkflow.php:99]
  #4 PhutilExecPassthru::execute() called at [<phutil>/src/future/exec/execx.php:50]
  #5 phutil_passthru(string, TempFile) called at [<phutil>/src/console/format.php:195]
  #6 phutil_console_get_terminal_width() called at [<phutil>/src/console/PhutilConsoleServer.php:79]
  #7 PhutilConsoleServer::handleMessage(PhutilConsoleMessage) called at [<phutil>/src/console/PhutilConsole.php:204]
  #8 PhutilConsole::writeMessage(PhutilConsoleMessage) called at [<phutil>/src/console/PhutilConsole.php:288]
  #9 PhutilConsole::getErrCols() called at [<phutil>/src/console/PhutilConsoleProgressBar.php:171]
  #10 PhutilConsoleProgressBar::getWidth() called at [<phutil>/src/console/PhutilConsoleProgressBar.php:163]
  #11 PhutilConsoleProgressBar::eraseLine() called at [<phutil>/src/console/PhutilConsoleProgressBar.php:154]
  #12 PhutilConsoleProgressBar::done(boolean) called at [<phutil>/src/console/PhutilConsoleProgressBar.php:176]
  #13 PhabricatorSearchManagementIndexWorkflow::execute(PhutilArgumentParser) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:406]
  #14 PhutilArgumentParser::parseWorkflowsFull(array) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:301]
  #15 PhutilArgumentParser::parseWorkflows(array) called at [<phabricator>/scripts/search/manage_search.php:21]

Revisions and Commits

Event Timeline

joshuaspence raised the priority of this task from to Needs Triage.
joshuaspence updated the task description. (Show Details)
joshuaspence added a project: Search.
joshuaspence added a subscriber: joshuaspence.

I ran this same command on a separate host and stopped the phd daemons before running the command. Doing so allowed the command to complete successfully.

That stack trace is bonkers, I think we're calling tput cols every time we update a progress bar.