Page MenuHomePhabricator
Create Task
Maniphest T12310

Phd out of memory error
Closed, InvalidPublic
Actions

Description

found in output of phd log command after phd was running for ca 6 hours

Daemon 552 STDO [Thu, 23 Feb 2017 16:22:25 +0100]
Daemon 552 STDO [Thu, 23 Feb 2017 16:22:25 +0100] Fatal error: Out of memory (allocated 5281415168) (tried to allocate 18446744071986036716 bytes) in /phab/phabricator/src/applications/files/storage/PhabricatorFile.php on line 1021
Daemon 552 FAIL [Thu, 23 Feb 2017 16:22:27 +0100] Process exited with error 255.

phd is running under systemctl (CentOS)

similar thing happened (but at a different time) yesterday.
There is no meaningul information in phd log around that time, apart from the fact that processes were started and stopped multiple times since 16:

Daemon 591 INIT [Thu, 23 Feb 2017 16:11:58 +0100] Starting process.
Daemon 591 DONE [Thu, 23 Feb 2017 16:12:15 +0100] Process exited normally.
Daemon 592 INIT [Thu, 23 Feb 2017 16:12:16 +0100] Starting process.
Daemon 592 DONE [Thu, 23 Feb 2017 16:12:32 +0100] Process exited normally.
Daemon 593 INIT [Thu, 23 Feb 2017 16:12:33 +0100] Starting process.
Daemon 593 DONE [Thu, 23 Feb 2017 16:12:49 +0100] Process exited normally.
Daemon 594 INIT [Thu, 23 Feb 2017 16:12:50 +0100] Starting process.
Daemon 594 DONE [Thu, 23 Feb 2017 16:13:07 +0100] Process exited normally.
Daemon 595 INIT [Thu, 23 Feb 2017 16:13:08 +0100] Starting process.
Daemon 595 DONE [Thu, 23 Feb 2017 16:13:24 +0100] Process exited normally.
Daemon 596 INIT [Thu, 23 Feb 2017 16:13:25 +0100] Starting process.
Daemon 596 DONE [Thu, 23 Feb 2017 16:13:42 +0100] Process exited normally.
Daemon 597 INIT [Thu, 23 Feb 2017 16:13:43 +0100] Starting process.
Daemon 597 DONE [Thu, 23 Feb 2017 16:13:59 +0100] Process exited normally.
Daemon 598 INIT [Thu, 23 Feb 2017 16:14:00 +0100] Starting process.
Daemon 598 DONE [Thu, 23 Feb 2017 16:14:16 +0100] Process exited normally.
Daemon 599 INIT [Thu, 23 Feb 2017 16:14:17 +0100] Starting process.
Daemon 599 DONE [Thu, 23 Feb 2017 16:14:34 +0100] Process exited normally.
Daemon 600 INIT [Thu, 23 Feb 2017 16:14:35 +0100] Starting process.
Daemon 600 DONE [Thu, 23 Feb 2017 16:14:51 +0100] Process exited normally.
Daemon 601 INIT [Thu, 23 Feb 2017 16:14:52 +0100] Starting process.
Daemon 601 DONE [Thu, 23 Feb 2017 16:15:08 +0100] Process exited normally.
Daemon 602 INIT [Thu, 23 Feb 2017 16:15:09 +0100] Starting process.
Daemon 602 DONE [Thu, 23 Feb 2017 16:15:26 +0100] Process exited normally.
Daemon 603 INIT [Thu, 23 Feb 2017 16:15:27 +0100] Starting process.
Daemon 603 DONE [Thu, 23 Feb 2017 16:15:43 +0100] Process exited normally.
Daemon 604 INIT [Thu, 23 Feb 2017 16:15:44 +0100] Starting process.
Daemon 604 DONE [Thu, 23 Feb 2017 16:16:01 +0100] Process exited normally.
Daemon 605 INIT [Thu, 23 Feb 2017 16:16:02 +0100] Starting process.
Daemon 605 DONE [Thu, 23 Feb 2017 16:16:18 +0100] Process exited normally.
Daemon 606 INIT [Thu, 23 Feb 2017 16:16:19 +0100] Starting process.
Daemon 606 DONE [Thu, 23 Feb 2017 16:16:35 +0100] Process exited normally.
Daemon 607 INIT [Thu, 23 Feb 2017 16:16:36 +0100] Starting process.
Daemon 607 DONE [Thu, 23 Feb 2017 16:16:53 +0100] Process exited normally.
Daemon 608 INIT [Thu, 23 Feb 2017 16:16:54 +0100] Starting process.
Daemon 608 DONE [Thu, 23 Feb 2017 16:17:10 +0100] Process exited normally.
Daemon 609 INIT [Thu, 23 Feb 2017 16:17:11 +0100] Starting process.
Daemon 609 DONE [Thu, 23 Feb 2017 16:17:27 +0100] Process exited normally.
Daemon 610 INIT [Thu, 23 Feb 2017 16:17:28 +0100] Starting process.
Daemon 610 DONE [Thu, 23 Feb 2017 16:17:45 +0100] Process exited normally.
Daemon 611 INIT [Thu, 23 Feb 2017 16:17:46 +0100] Starting process.
Daemon 611 DONE [Thu, 23 Feb 2017 16:18:02 +0100] Process exited normally.
Daemon 612 INIT [Thu, 23 Feb 2017 16:18:03 +0100] Starting process.
Daemon 612 DONE [Thu, 23 Feb 2017 16:18:19 +0100] Process exited normally.
Daemon 613 INIT [Thu, 23 Feb 2017 16:18:21 +0100] Starting process.
Daemon 613 DONE [Thu, 23 Feb 2017 16:18:37 +0100] Process exited normally.
Daemon 614 INIT [Thu, 23 Feb 2017 16:18:38 +0100] Starting process.
Daemon 614 DONE [Thu, 23 Feb 2017 16:18:54 +0100] Process exited normally.
Daemon 615 INIT [Thu, 23 Feb 2017 16:18:55 +0100] Starting process.
Daemon 615 DONE [Thu, 23 Feb 2017 16:19:12 +0100] Process exited normally.
Daemon 616 INIT [Thu, 23 Feb 2017 16:19:13 +0100] Starting process.
Daemon 616 DONE [Thu, 23 Feb 2017 16:19:29 +0100] Process exited normally.
Daemon 617 INIT [Thu, 23 Feb 2017 16:19:30 +0100] Starting process.
Daemon 617 DONE [Thu, 23 Feb 2017 16:20:04 +0100] Process exited normally.
Daemon 618 INIT [Thu, 23 Feb 2017 16:20:05 +0100] Starting process.
Daemon 618 DONE [Thu, 23 Feb 2017 16:20:21 +0100] Process exited normally.
Daemon 619 INIT [Thu, 23 Feb 2017 16:20:22 +0100] Starting process.
Daemon 619 DONE [Thu, 23 Feb 2017 16:20:39 +0100] Process exited normally.
Daemon 620 INIT [Thu, 23 Feb 2017 16:20:40 +0100] Starting process.
Daemon 620 DONE [Thu, 23 Feb 2017 16:20:56 +0100] Process exited normally.
Daemon 621 INIT [Thu, 23 Feb 2017 16:20:57 +0100] Starting process.
Daemon 621 DONE [Thu, 23 Feb 2017 16:21:14 +0100] Process exited normally.
Daemon 622 INIT [Thu, 23 Feb 2017 16:21:15 +0100] Starting process.
Daemon 622 DONE [Thu, 23 Feb 2017 16:21:31 +0100] Process exited normally.
Daemon 623 INIT [Thu, 23 Feb 2017 16:21:32 +0100] Starting process.
Daemon 623 DONE [Thu, 23 Feb 2017 16:21:48 +0100] Process exited normally.
Daemon 624 INIT [Thu, 23 Feb 2017 16:21:49 +0100] Starting process.
Daemon 624 DONE [Thu, 23 Feb 2017 16:22:06 +0100] Process exited normally.
Daemon 625 INIT [Thu, 23 Feb 2017 16:22:07 +0100] Starting process.
Daemon 625 DONE [Thu, 23 Feb 2017 16:22:23 +0100] Process exited normally.
Daemon 626 INIT [Thu, 23 Feb 2017 16:22:24 +0100] Starting process.

version information:

phabricator 37ac0ada178c70179036b6047d6ff7ffd1849da8 (Tue, Feb 14)
arcanist bc9b70508edf224a15a3bdb45998f7fd75847765 (Wed, Feb 15) 
phutil d37145bd144e255591083cb851102f17f6c30663 (Tue, Feb 14)

phd is running under apache account

Event Timeline

voltcode created this task.Feb 23 2017, 3:44 PM
epriestley edited projects, added Bug Report (Needs Information); removed Bug Report.Feb 23 2017, 3:56 PM
epriestley added a subscriber: epriestley.

We need reproduction instructions which allow us to reproduce this issue in a clean environment. See Contributing Bug Reports and Providing Reproduction Steps.

If you are unable to provide instructions we can follow to reproduce this issue within a few days, we will close this report.

chad renamed this task from Phd out of memory error to Phd out of memory error.Feb 23 2017, 4:50 PM
voltcode added a comment.Feb 24 2017, 8:17 AM

I found that svn repo import was stuck on a very large branch import. I marked the repo as imported, suddenly the herald rules and notifications started to work properly (which was my other problem that I wanted to solve with my instance).

I will observe whether this OOM happens again.

My suggestion is to add more logging/messages for such cases, so admin can quickly connect OOM error to repository import failure. The repo was stuck at 99% and I was not aware whether there were any problems with the import. I stumbled upon the repository importing command by accident, when I tried to test a Herald rule and it was the only place were the system warned me that notification business logic may not work without importing completed.

chad closed this task as Invalid.Mar 4 2017, 5:18 AM
Phabricator · Built by Phacility