What specifically is your update process? See https://github.com/phacility/phabricator/blob/master/scripts/install/update_phabricator.sh for our recommended sequence.

I have this issue now. Just did a git pull an hour ago on arcanist, libphutil and phabricator. Restarted apache and the phabricator daemons, but the warning is still there. I also did bin/phd stop and check if there were no daemons left and they were indeed all gone. There are no errors in the daemons.log file. One thing I checked is the APC cache (I have APC enabled) but its cache is cleared when Apache is restarted or reloaded. I have no idea what to do about it. Ignoring the warning doesn't feel right; it must be caused by something and that should be fixed.

Personally I think the different title / issue message cleared up a lot of confusion, because it's now very clear as to what the problem is when the issue pops up.

I haven't seen more issues from users on this recently, so I think we either survived the initial backlash and/or writing a novel helped.

Review as soon as possible

Any ideas how to fix it?

upsforgrabbsing as we'll deal with this more as part of high availability stuff

Maybe this is good now, but if not, I think your outstanding ideas are:

◀ Merged tasks: T6042.

Assuming that explanation covers things, but yell if stuff is still unclear.

In this case I think it was just me configuring different UI headers to different tiers.

What configuration differs between tiers? I sort of imagine moving all the tier-dependent stuff elsewhere, although I don't have a clear vision for it yet.

Whoops... Toggling "ignore" does change the config though, so isn't this warning showing up correctly?

Closed by commit rP7ecbc707846f.

Setup issues are cached, toggling ignore is probably just clearing the cache.

Whoops... Toggling "ignore" does change the config though, so isn't this warning showing up correctly?

Toggling

Not sure if this is new, but with HEAD toggling 'ignore' on a setup issue triggers a setup issue for 'Phabricator Daemons Need Restarting'.

If you run multiple copies of the daemon, it's expected that they'll occasionally fail to acquire locks, because another copy of the daemon is busy updating the repository.

I might have been, yes. Is this not supported?

I'm going to repurpose this for fixing the hang, we definitely shouldn't stall the queue over mail issues.

This was caused by Docker killing off the SETGID bit on postqueue / postdrop executables. See https://gist.github.com/porjo/35ea98cb64553c0c718a.

From IRC:

This is the backtrace on an actual child PID:

Backtrace from daemons:

Is it possible to get {D9307} committed? as this would certainly go a long way to help new users (like me) understand that even though the phd.user is described as 'Specify a system user to run the daemons as.' it doesn't currently mean that the users will now automatically run as that user.

◀ Merged tasks: T5997.

Closed by commit rP06882a99cf92.

There are 14 queued tasks; none of them have been processed since I originally made the task. The MetaMTA tasks are now sitting on an expires of -39245.

Closed by commit rP7cdee1884ec8.

You can use bin/mail show-outbound --id <id> to inspect a "Void" message for an explanation as to why it was voided. Usually it means the recipient declined to receive the notification as an email, so we dropped the mail and did not deliver it.

This is what the leased tasks show as well:

I ran bin/mail send-test and then bin/mail list-outbound almost immediately after; this is what I saw:

Have you confirmed that sending mail really is a quick operation -- for example, by using bin/mail send-test? We've seen environments where this operation takes longer than one might expect.

• In FeedPublisherHTTPWorker, before making the request, pull feed.http-hooks from config.
• If the URI is no longer present, throw PhabricatorWorkerPermanentFailureException to permanently fail the request.

• D5663 added a reasonable-looking way to detect these processes, but bin/phd never got the glue code to call it.
• This task is otherwise still relevant:
  • Add code in bin/phd stop and bin/phd restart to warn the user that there are processes which look like daemons that don't have corresponding PID files, and that they can use --force to terminate them.
  • With --force, terminate these processes.
  • The steps in https://secure.phabricator.com/T2374#8 should still let you reproduce this in modern phd.

Closed by commit rP6f246bd35190.

I think that plan ^^^ is a pretty good one, and this crops up somewhat-often.

We should do this as a security thing -- although we aren't currently aware of anything sensitive in the logs, there has been sensitive stuff in the past, and it's reasonable to think there may be sensitive stuff in the future. Basically:

In T5896#7, @epriestley wrote:

Do you have ~10,000 branch heads or bookmarks in this repository?

Do the branches shown in the web UI make sense to you? Are they obviously wrong?

Trace: {F192790}
I tried Phab with CentOS7 and python 2.7.5 and hg 2.6.2 on the same repo, and have the same result
In branch history there is offset 1100, so it's near this number of branch heads

http://phabricator/diffusion/F/branches/default/?offset=1100

But

[root@Phabricator F]# hg branches | wc -l
221

And there are no bookmarks

Specifically, I think this issue is related to the enormous length of the command. The command is supposed to look like:

Do you have ~10,000 branch heads or bookmarks in this repository?

Can you run the command again with the --trace parameter specified?

Are you running multiple copies of the PullLocal daemon?

I hit this same issue when trying to use PhabricatorGlobalLock inside Drydock, which appeared to only occur with lots of frequent requests to Drydock. I ended up using beginReadLocking instead, but obviously this is a problem that impacts existing code as well.

I merged T5326 here since I plan to move "Combined Log" to bin/phd log. Then grep lets you filter, and we could add some other flags if other things make sense.

✘ Merged into T5405.

◀ Merged tasks: T5326.