D12549 applies the strategy recursively, although we'll probably hit shell argument length limits before we hit 65K branch heads.

This is rooted in a limitation with Mercurial. I've filed a report upstream:

The change breaks "Track Only", breaks repositories without "Track Only", breaks new imports, runs 75x slower than necessary in the cases above, etc.

Occurring on yet another repository after the latest rebase:

Thanks, I will.

Ah, yeah, if you're far behind HEAD you probably still had the 1 hour behavior. I'd expect that updating will improve things. Let us know if you run into anything else.

Hmm, it would also appear that I desperately need to update. I was unaware that the FreeBSD port maintainer hasn't updated the port since summer of last year. My apologies for probably wasting everyone's time.

I'm seeing somewhat different behaviour. I can't be 100% sure when exactly it finally figured out that those daemons weren't coming back, because I sleep during the day, but it took at least 4 hours (since I was still using it during that period). When I refreshed everything just now, the banner went away and everything appears to be chooching along like it should be.

We don't look for PID files or examine the process table because:

This appears to solve the problem for me, though Phabricator appears to ignore the 'Track Only' setting and instead imports all hundred or so branches; I don't know if that's at all related to @Jaraill's change though. And I'm also running into T7100.

Spooky problem has not occurred in the few minutes since I rebased off of e0473aa70265a222d440c7e2e2680d2d1959ab20. Will continue to watch.

We are seeing this again with both D12263 and D12253 applied.

No errors with that patch either...

I guess you could try this? Maybe?

• Shorten patch: Works
• Spooky Patch: Works
• Roll back to no patch: Confirmed still broken.

The order which we update repositories is deterministic enough that I'd expect things to usually end up locking in the same order, and they'll keep retrying regularly (every 15 seconds) after a failure, so it's not particularly surprising to me that the failures wouldn't be randomly distributed if the issue is a 64-character lock uniqueness limit, although I would expect us to see way more failures all the time since 64 characters ends before the callsign part and we haven't changed the lock name. Maybe the behavior is specific to your MySQL sub-version.

Does this patch also fix it?

With that patch the daemon log is clean and the repositories show recent updates.

Can you see if this fixes it (bringing lock name length under 64 characters):

It's also possible that this:

MySQL 5.7.5 and later enforces a maximum length on lock names of 64 characters. Previously, no limit was enforced.

Okay, that is the lock failing to acquire. Does this work?

I'm not sure I find the new log message that much more different or illuminating.

Updated log message with D12253 applied and daemons restarted

Hrrm. Maybe wait for D12253 and see if that provides any clarity. It's possible it will give us a much better and more specific message, rather than just degrading into T6958.

Specifically, if that returns an integer, it's a connection ID.

Does this show anything?

The root cause is very likely to be a stray copy of the daemon running somewhere.

D12253 "fixes" this in the sense that it likely reduces it to T6958.

Never mind, I think that's someting else.

Oh, this might not be quite related.

I'm now seeing this on a second repository. After some digging it appears to be a magical mysql GET_LOCK and not something simple on the file system I can just clear.

I'm seeing this on a build from March 12. A repository stopped updating on Friday and the daemon logs have been printing out this message ever few minutes and it is still occuring on Monday. No change after phd restart. diffusion/edit says 'Initializing Working Copy Daemons are initializing the working copy.'

It's hard to wait, so I found my way to solve this problem.

Thanks man!

We should probably also set this prior to accepting pushes via HTTP and SSH, so those writes happen under a favorable umask.

This is more possible after the centralization of the overseer and the introduction of phd reload and the SIGHUP handling. Specifically, we could do this:

You have to bin/phd stop <only the second overseer PID>.

Okay, but you still have no way of safely issuing another bin/phd stop without it stopping both overseers?

There's no way an over-overseer can know when the previous reload is finished

There's no way an over-overseer can know when the previous reload is finished; all it can do is call bin/phd stop --graceful 999999 in the background and leave it to run. In addition, a previous reload can take several hours to run; if you need to deploy out another set of code before the reload is finished, you have no safe way of doing so until the first one is entirely complete.

Queuing multiple graceful exits just isn't a use case we're going to support in the upstream. You can conceivably have your over-overseer decline to send the second SIGINT if an overseer has already been SIGINT'd once. bin/phd stop <pid> --graceful ... can send the signals selectively. I'd maaaaaybe accept a bin/phd status --json since parsing bin/phd status is a bit of a mess, but the other tools to build this already exist.

You could use ^\ / SIGQUIT to force the non-graceful exit when the daemon's in the foreground and keep ^C as graceful only.

The reason for this logic is so that you can ^C twice to get out if a daemon is hung and you're running it in the foreground. This avoids the need to bring up a second terminal, figure out the PID of the daemon running in the first terminal, kill it, hope you got the PID right, etc., or abandon the first terminal entirely.

Narratively within the system, these are still "in the queue", they just have a lease on them. They don't get taken out of the queue until they are archived.

Understood. I have the daemon logs and db dumps if you think they would be helpful.

There isn't enough information to reproduce this issue.

Yeah, that's the main aspect of it.

I think the core issue is if your daemons are failing (MySQL issues?) we should somehow tell you. The picking up where they left off when kicked is what I expect them to do already. Maybe @epriestley has better ideas than me though.

Sorry, there was a lot of background/setup story there.

Nothing here sound wrong, but maybe I'm missing some details. It doesn't sound easily reproducible unfortunately either... off the top of my head.

What's the main issue here?

Looking at the daemon logs, it looks like there were various mysql failures throughout the day (too many cnxns), but it seems like it was able to recover from those.

After T5166 we can put something useful here ("Your bulk jobs.") for normal users.

@epriestley, I'm happy to work on this if you give me some pointers

You, sir, are a gentlemen and a scholar.

For installs with phabricator.developer-mode disabled, we could conceivably fail tasks with a missing class permanently, although I'm hesitant about that (it's possible that they're important and the administrator just uninstalled an extension by accident).

I'm sure this question has been asked previously, but why can't the daemons read the updated configuration themselves? I imagine that the configuration would be cached, but surely it could be purged periodically?