Page MenuHomePhabricator

Older project's icon changes back after a few days
Closed, ResolvedPublic

Description

One of my projects (it's one of my earlier ones created last year), whenever I update it to set the icon, it lasts for a few days and then resets. The icon I'm trying to change it to is one of the defaults based on the icon/color selected for the project, it would look like the others in this screenshot (the third one is the problematic project):

The newer projects I've created recently (few weeks) don't have this problem. The way I change the icon is by ProjectManageEdit PictureUse Picture(First Item)

I've tried changing this setting several times. After a few days it ends up reverting back. This shows in the timeline for the project:

Install

Config
storage.mysql-engine.max-size: 0
storage.local-disk.path: /var/local/phacility/phabricator/filestore
storage.s3.bucket: null

security.alternate-file-domain: null

Event Timeline

This one is obviously ghosts. How could this possibly be anything but ghosts?

I suspect this might be related to T4752, or similar. I'm open to other ideas on data to try and collect about this scenario. I'll try to keep an eye on the project to see when the icon reverts and get logs at the relevant time.

isfs added a subscriber: isfs.May 5 2016, 12:56 PM
cspeckmim added a comment.EditedMay 7 2016, 2:03 AM

OK so I've been keeping an on my icons this week to see when it would happen. The ghosts indeed came back and I grabbed logs during the shortest timeframe I could narrow it down to - roughly between 1pm and 8pm today ("narrow", sorry).

(made some minor modifications to Nginx log to remove actual IP and hostnames)
{P1978}

{P1979}

The last exception in daemons.log:

06-May-2016 20:36:27 America/New_York] [2016-05-06 20:36:27] EXCEPTION: (FilesystemException) Failed to remove file '/var/local/phacility/phabricator/filestore/1a/3e/94c5ea8fadb9f60811517d0eaca9'! at [<phutil>/src/filesystem/Filesystem.php:325]

After some level-5 sleuthing I added .jpg to that file unable to be deleted. Lo-and-behold my yellow truck

chad added a comment.May 7 2016, 2:16 AM

more of a mustard, really.

Do phabricator ghosts dream of mustard trucks

Seems to be happening on quite a lot of Wikimedia's phabricator projects: https://phabricator.wikimedia.org/T128160 ... At least I think it's the same bug. I have no idea what might be causing this.

The recent changes to the local filestore permission did not seem to resolve this (my truck went missing again). However, now that the logs aren't totally polluted I may see the exception related to this. I'm not positive whether this provides more insight or not, though.

/var/tmp/phd/log/daemons.log
[25-May-2016 13:22:23 America/New_York] [2016-05-25 13:22:23] EXCEPTION: (PhutilProxyException) Permanent failure while executing Task ID 4733388. {>} (PhabricatorWorkerPermanentFailureException) Unable to load transactions: PHID-XACT-DREV-vl56ah6csf7wlan. at [<phabricator>/src/applications/transactions/worker/PhabricatorApplicationTransactionPublishWorker.php:112]
[25-May-2016 13:22:23 America/New_York] arcanist(head=stable, ref.master=d54cb072facd, ref.stable=1439aaa87183), phabricator(head=stable, ref.master=7cc8a73d1efd, ref.stable=2c67d9c8ace2, custom=2), phutil(head=stable, ref.master=83f09f6c5a03, ref.stable=76425eaa8125)
[25-May-2016 13:22:23 America/New_York]   #0 <#2> PhabricatorApplicationTransactionPublishWorker::loadTransactions(DifferentialRevision) called at [<phabricator>/src/applications/transactions/worker/PhabricatorApplicationTransactionPublishWorker.php:19]
[25-May-2016 13:22:23 America/New_York]   #1 <#2> PhabricatorApplicationTransactionPublishWorker::doWork() called at [<phabricator>/src/infrastructure/daemon/workers/PhabricatorWorker.php:122]
[25-May-2016 13:22:23 America/New_York]   #2 <#2> PhabricatorWorker::executeTask() called at [<phabricator>/src/infrastructure/daemon/workers/storage/PhabricatorWorkerActiveTask.php:171]
[25-May-2016 13:22:23 America/New_York]   #3 <#2> PhabricatorWorkerActiveTask::executeTask() called at [<phabricator>/src/infrastructure/daemon/workers/PhabricatorTaskmasterDaemon.php:22]
[25-May-2016 13:22:23 America/New_York]   #4 PhabricatorTaskmasterDaemon::run() called at [<phutil>/src/daemon/PhutilDaemon.php:183]
[25-May-2016 13:22:23 America/New_York]   #5 PhutilDaemon::execute() called at [<phutil>/scripts/daemon/exec/exec_daemon.php:125]
isfs added a comment.Jul 6 2016, 11:07 PM

Observe:

  • Choose the default colour+icon picture for the project
  • In the project history, click the link to the file that has been assigned as the picture
  • Note that the file is temporary, so will be cleaned up by the garbage collector

In source code:

  • Files for built-in content (including generated icons) are made temporary by PhabricatorFile::loadBuiltins.
  • Existing files with the same content are retrieved by PhabricatorFile::buildFromFileDataOrHash.

Analysis:

  • This problem would have been introduced when the generated colour+icon picture began to be displayed on the Edit Picture page.
  • When the Edit Picture page is displayed, an icon, and a temporary file used to serve it, is generated (expires after 7 days so will be collected as garbage next time the daemons tidy up after that).
  • When the same picture is chosen, the content is identical so the same temporary file is retrieved and used.
  • Of course, this will only affect projects that don't have a permanent picture file created for them matching their colour+icon prior to the Edit Picture page being shown (e.g. when they are created) or the Edit Picture page just uses the existing permanent file, too, rather than creating a temporary one. This would include old projects and projects whose colour or icon has been changed so no longer matches an already-generated picture.

Solution:

All files from before you upgrade to D16270 are haunted and subject to ghastly behavior.

Files selected/created after you upgrade to D16270 should stick around.

chad awarded a token.Jul 11 2016, 4:29 PM

After upgrading and several weeks of the icons not disappearing,