✘ Merged into T1279.

T1279#84

Thank you very much for the information. Regarding identify defects per program is because it has a core and need to identify how many such defects have been detected in a program, but may be different programming tasks that have been implemented by different programmers.

Could there be some sort of watchdog daemon which does nothing except for signalling the daemons to restart when the config changes?

There's also a performance/polling issue, but the big thing is just the peril of swapping config at runtime and having no way to be sure exactly what that will do.

Oh, I didn't know that happened :/... I figured the daemons would load config from MySQL as requested, but I'm guessing that's too much of a performance hit?

In T5968#33, @epriestley wrote:

where the latter seems to be much more important?

No users have hit issues with the code being different that I've seen, but quite a few have hit issues with config being different.

where the latter seems to be much more important?

I think a primary point of confusion is that this is measuring configuration state and not "the code is different", where the latter seems to be much more important? (I read the error message as something to do with code, because it says "out of date configuration" and not "mismatched configuration")

Yeah, project issues are a pretty cool place to hang out.

Nope, just hanging out and following interesting looking tickets before our next upgrade.

In theory, I guess we could also show the affected daemons directly in the error message.

Oh, another thought is that we could checksum only the database configuration, or checksum the different configurations separately. This would have fixed @joshuaspence's issue by letting us raise a more narrow message ("Local configuration differs").

My only other ideas are:

I think the pull request above is the only remaining unsolved mystery -- unless you're hitting this, @cburroughs? I'll followup on the PR.

I'm not sure if D10367 makes this easy enough to debug... Other ideas on enhancements here?

Oh, don't worry about it! My thinking on this is that these (or at least, some of them) were probably all horrible bugs waiting to strike at a future date, and we're catching them now in a low-severity way where I can just say "oh check this ticket" instead of "sorry nothing works and we are the worst".

Wow, sorry about the support impact here. :/ working on this now...

@joshuaspence hit this last night, but it also had a legitimate reason (local.json differed between daemons and web).

https://github.com/phacility/phabricator/issues/713 is another case of this.

Reopening this as a catch-all since we're still seeings some users hit this and haven't improved the diagnostics yet.

There are 14 queued tasks; none of them have been processed since I originally made the task. The MetaMTA tasks are now sitting on an expires of -39245.

(Feel free to re-open this task if you want to make it a catch-all for out-of-date daemon stuff, but this issue is at least resolved for me)

This setup issue seems to have disappeared now. I think this was caused by the Docker instance not doing bin/phd stop to shutdown daemons (it just TERMs then KILLs all processes before shutdown) and hence for 6 minutes Phabricator still thinks those out-of-date daemons are still running.

T2374 might play a role here, too, in some cases.

Just to clarify for anyone else hitting this issue, the warning is about the configuration being different, not the code. T5957 the underlying issue was a different PHABRICATOR_ENV.

◀ Merged tasks: T5957.

✘ Merged into T5968.

You can use bin/mail show-outbound --id <id> to inspect a "Void" message for an explanation as to why it was voided. Usually it means the recipient declined to receive the notification as an email, so we dropped the mail and did not deliver it.

This is what the leased tasks show as well:

I ran bin/mail send-test and then bin/mail list-outbound almost immediately after; this is what I saw:

It might be helpful to add a status indicator to the daemon console, so user who report issues with this warning can go to the web UI and see which daemon(s) are running with out-of-date configuration. Then we can figure out if we have an issue in the checksumming method or an issue elsewhere.

Have you confirmed that sending mail really is a quick operation -- for example, by using bin/mail send-test? We've seen environments where this operation takes longer than one might expect.

Aaah fail. Yep the daemons and web had different phabricator_env.

You could try applying this patch to test the second theory:

Some possible causes:

Which mail adapter are you using (SendGrid, SES, Mailgun, etc)?

✘ Merged into T5501.

• In ApplicationTransactionEditor, we currently extract all mentioned objects from transactions (in expandRemarkupBlockTransactions()) and then filter out everything that's not a project.
• Add some new code to load all the mentioned objects and check which ones implement a new PhabricatorMentionableInterface.
• Write a new edge type (e.g., OBJECT_MENTIONS_OBJECT / OBJECT_MENTIONED_BY_OBJECT) against all these objects.
• Let's hide this story in the mentioning object and not generate feed stories, email, or notifications for it for now? So the mentioned object just gets a epriestley mentioned this in D123: blah blah transaction with no fanfare.
• Implement MentionableInterface on some objects, but not Users, Projects, or Files (since these already have other behaviors).
• It would also be nice to not write this edge if we're writing a Ref/Fixes edge. That is, for comment text Fixes T123, it would be nice to interpret that as not being a mention. You might have to add some getPHIDsWhichShallNotBeMentioned($xaction) to support that.

Two other related issues have cropped up since this was filed. I updated the description to reflect the state of the world.

◀ Merged tasks: T5949.

✘ Merged into T4018.

◀ Merged tasks: T4988.

(I think we will, but it might be a v2 feature because there are some security concerns.)

Assuming we use ElasticSearch, yes.

So Phacility should offer this then, if configured correctly?

This uses the fulltext search index, and MySQL has no stemming. T4818 discusses stemming in more detail.

Doh! Thanks!

I have about half a diff for this.

From the "Whitespace Changes" dropdown, select "Show All" to show all whitespace changes:

✘ Merged into T182.

Closed by commit rPd122d9ec86a8.

Closed by commit rPa2a0f002f069.

I just tested your diff on our installation and it works perfectly. Thanks!

I've attached a fix for @epriestley to review when he's online. If you could verify that it fixes the issue for your install in the mean time, that'd be appreciated.

or use systemd service:

1. create file

/lib/systemd/system/phabricator-phd.service

[Unit]
Description=phabricator-phd
After=syslog.target network.target
Before=apache2.service
User=wwwrun
Group=www

Are you using PhabricatorDraft, explicitly? Can you show me that code? I don't think we write PhabricatorDraft objects implicitly right now.

This has come up a couple of times recently. Instead of supporting this as a security policy (which will be difficult to debug) I think we'll have an easier time of and get more milage from it by supporting it as a project membership rule (which, at a minimum, can have reasonable web UI workflows and be a lot easier to debug and understand).

Closed by commit rP6669fe9e8a70.