This one is obviously ghosts. How could this possibly be anything but ghosts?

I suspect this might be related to T4752, or similar. I'm open to other ideas on data to try and collect about this scenario. I'll try to keep an eye on the project to see when the icon reverts and get logs at the relevant time.

OK so I've been keeping an on my icons this week to see when it would happen. The ghosts indeed came back and I grabbed logs during the shortest timeframe I could narrow it down to - roughly between 1pm and 8pm today ("narrow", sorry).

(made some minor modifications to Nginx log to remove actual IP and hostnames)
{P1978}

{P1979}

The last exception in daemons.log:

06-May-2016 20:36:27 America/New_York] [2016-05-06 20:36:27] EXCEPTION: (FilesystemException) Failed to remove file '/var/local/phacility/phabricator/filestore/1a/3e/94c5ea8fadb9f60811517d0eaca9'! at [<phutil>/src/filesystem/Filesystem.php:325]

After some level-5 sleuthing I added .jpg to that file unable to be deleted. Lo-and-behold my yellow truck

more of a mustard, really.

Do phabricator ghosts dream of mustard trucks

Seems to be happening on quite a lot of Wikimedia's phabricator projects: https://phabricator.wikimedia.org/T128160 ... At least I think it's the same bug. I have no idea what might be causing this.

The recent changes to the local filestore permission did not seem to resolve this (my truck went missing again). However, now that the logs aren't totally polluted I may see the exception related to this. I'm not positive whether this provides more insight or not, though.

[25-May-2016 13:22:23 America/New_York] [2016-05-25 13:22:23] EXCEPTION: (PhutilProxyException) Permanent failure while executing Task ID 4733388. {>} (PhabricatorWorkerPermanentFailureException) Unable to load transactions: PHID-XACT-DREV-vl56ah6csf7wlan. at [<phabricator>/src/applications/transactions/worker/PhabricatorApplicationTransactionPublishWorker.php:112]
[25-May-2016 13:22:23 America/New_York] arcanist(head=stable, ref.master=d54cb072facd, ref.stable=1439aaa87183), phabricator(head=stable, ref.master=7cc8a73d1efd, ref.stable=2c67d9c8ace2, custom=2), phutil(head=stable, ref.master=83f09f6c5a03, ref.stable=76425eaa8125)
[25-May-2016 13:22:23 America/New_York]   #0 <#2> PhabricatorApplicationTransactionPublishWorker::loadTransactions(DifferentialRevision) called at [<phabricator>/src/applications/transactions/worker/PhabricatorApplicationTransactionPublishWorker.php:19]
[25-May-2016 13:22:23 America/New_York]   #1 <#2> PhabricatorApplicationTransactionPublishWorker::doWork() called at [<phabricator>/src/infrastructure/daemon/workers/PhabricatorWorker.php:122]
[25-May-2016 13:22:23 America/New_York]   #2 <#2> PhabricatorWorker::executeTask() called at [<phabricator>/src/infrastructure/daemon/workers/storage/PhabricatorWorkerActiveTask.php:171]
[25-May-2016 13:22:23 America/New_York]   #3 <#2> PhabricatorWorkerActiveTask::executeTask() called at [<phabricator>/src/infrastructure/daemon/workers/PhabricatorTaskmasterDaemon.php:22]
[25-May-2016 13:22:23 America/New_York]   #4 PhabricatorTaskmasterDaemon::run() called at [<phutil>/src/daemon/PhutilDaemon.php:183]
[25-May-2016 13:22:23 America/New_York]   #5 PhutilDaemon::execute() called at [<phutil>/scripts/daemon/exec/exec_daemon.php:125]

Observe:

• Choose the default colour+icon picture for the project
• In the project history, click the link to the file that has been assigned as the picture
• Note that the file is temporary, so will be cleaned up by the garbage collector

In source code:

• Files for built-in content (including generated icons) are made temporary by PhabricatorFile::loadBuiltins.
• Existing files with the same content are retrieved by PhabricatorFile::buildFromFileDataOrHash.

Analysis:

• This problem would have been introduced when the generated colour+icon picture began to be displayed on the Edit Picture page.
• When the Edit Picture page is displayed, an icon, and a temporary file used to serve it, is generated (expires after 7 days so will be collected as garbage next time the daemons tidy up after that).
• When the same picture is chosen, the content is identical so the same temporary file is retrieved and used.
• Of course, this will only affect projects that don't have a permanent picture file created for them matching their colour+icon prior to the Edit Picture page being shown (e.g. when they are created) or the Edit Picture page just uses the existing permanent file, too, rather than creating a temporary one. This would include old projects and projects whose colour or icon has been changed so no longer matches an already-generated picture.

Solution:

• Make temporary files permanent or longer lived if necessary when retrieving an existing file by content hash, so the file persists as long as all consumers are expecting it to.
• D16243: Ensure files live as long as intended.

All files from before you upgrade to D16270 are haunted and subject to ghastly behavior.

Files selected/created after you upgrade to D16270 should stick around.

After upgrading and several weeks of the icons not disappearing,