Ah, thanks! Yeah, looks pretty conclusive.

Yep! It looks like that's the issue:

I'll give that a try. I think the associated diffs have enormous lines, so it seems likely that's what's going on.

This is expected.

I'm vaguely suspicious that the mail is just too big (1MB of text plus tons and tons of HTML for syntax highlighting it?), and that the error is really arising from max_allowed_packet being violated, particularly given that the INSERT took 233ms to execute.

(Following up for Greg, as I'm now the oncall.)

mysql> select @@max_allowed_packet;
+----------------------+
| @@max_allowed_packet |
+----------------------+
|             67108864 |
+----------------------+
1 row in set (0.00 sec)

metamta.differential.inline-patches is 500.
metamta.email-body-limit is still the default of 524288.

I'm so excited about this ticket :)

Somehow this didn't autoclose.

This feature request does not describe a root problem. See Describing Root Problems.

I agree that there is no sensible technical reason for the broken network policies that prompted my comment and understand your reasons for preferring to punt on a technical solution. Thanks for your thoughtful remarks.

(If the policy isn't fixable, you can personally bear the cost of this policy by tunneling the traffic over 80 or 443 locally.)

Have they asked network operations to allow outbound traffic on port 22? It seems like organizations which employ software developers should reasonably permit this traffic. Particularly, security-minded organizations should reasonably prefer asymmetric keypair authentication over HTTP password authentication in the general case.

Some GHC contributors have had some trouble with the fact that there is no choice in staging area URL. Unfortunately, some users appear to be behind corporate firewalls which block anything but http and https. We use SSH for our staging repository, which is of course problematic for these contributors. There are a few comments above from Nov 2015 which suggest that we aren't the only ones who have this issue.

Hmmm I'm not seeing those lines in that file anywhere. On the current head of master, I'm pretty sure the viewer is always set to PhabricatorUser::getOmnipotentUser() in TransactionPublishWorker. Although maybe that isn't making it all the way to the policy aware query in every case.

@jcox I just noticed this line from you:

@jcox, I dove into this a bit, but like you, I didn't come up with much. We have not used bin/remove destroy for any of our repos and there doesn't appear to be any meaningful relationship between the failing tasks that I have found yet. The script that @epriestley kindly provided was able to load any diff I threw at it. The script was helpful in determining the right queries to make to investigate further.

See Phacility Consulting if you're interested in hiring us to help troubleshoot local issues. Otherwise we need reproduction steps to be complete and reproducible in our own environments if there is a bug present. Unfortunately we're not staffed to provide one on one support like this for free. Offhand, sounds like something is wrong with how JavaScript is getting delivered. Apache config (page modspeed), proxies, anything that alters the packaging should be looked at.

unfortunately I do not know since when it stops working.

I also can't reproduce locally or on Phacility. No other installs are reporting issues. Given Phabricator was not updated and the bug appeared, I'd have to think it's a local issue to your install and not a bug in Phabricator.

Works here.

See Consulting if you'd like me to help you personally. Thanks!

Yes, this is my mistake, sorryyy

Just checked this morning and many tasks have returned and started to fail again. I'm sure there's a way to reproduce this, but our setup is a bit complex and I'm not sure where to start. @epriestley Could you tell me how to determine what these tasks are even doing? Is there a way to see what action triggered them and why they will fail indefinitely?

• To click the "Land Revision" link, you must be able to push to the repository.
• The push is performed by the user that Drydock is operating as on the automation host.
• The reproduction steps and video above are consistent with this.

It's safe to do that (set utcInstanceEpoch = utcInitialEpoch for all events with a parentPHID where utcInstanceEpoch is currently NULL), with one minor caveat.

Why are users logging into sensitive accounts on shared computers? This action seems inherently unsafe to me.

Thanks for the quick response!

Update past D16623 (Sep 28). See also T11705.

This is a screenshot of the status page. This can usually be fixed by manually rescheduling an update by either clicking "Update now" or calling "./bin/repository update" in command line.

Specific attack scenario: Phabricator is used to host proprietary code. Someone logs into Phabricator on a shared computer and forgets to log out. That shared computer is now forever logged into his account, allowing anyone using it to access said proprietary code. If the cookie would expire or at least be session-only, the risk would be significantly lower.

Would it be safe for me to write a migration that sets utcInstanceEpoch equal to utcInitialEpoch on every row in the calendar_event table?

After running the migrations, some instances of recurring events end up with null for the utcInstanceEpoch column. I'm not sure how this happened, however, as some of the instances have a value that matches utcInitialEpoch and I'm not entirely sure what the purpose of utcInstanceEpoch would be. If it matches InitialEpoch then it seems kinda useless, as does null...

@epriestley: Sounds good to me, this seems quite sensible in every way.

@epriestley: Jaime, Senior DBA at the WMF, has posted some answers for your list of questions:

@jessjohnson I still haven't had a chance to dig further into this, but I'll report back when I do (or if you figure anything out, please let me know). I don't think killing off the tasks permanently is really an option since this seems to happen for specific degenerate instances of otherwise-needed tasks. Perhaps not allowing these tasks to retry would be an okay bandaid, but I'd still like to figure out the underlying problem if we can.

Also, like @jcox, the diffs always load in that advanced space-age script like there's no problem. Perhaps these exceptions are aware of our attempts to end them. @epriestley Do you have an idea of something else we could try to see why so many exceptions are occurring?

The upstream still doesn't have a reproduction case; we won't act until we have one. See Providing Reproduction Steps.

We're also seeing about 100k of these failures in our logs over the past few weeks. I clear out all the tasks and they quickly return. Have you found the root cause?

That change doesn't specifically do anything about mail tasks, but assuming they were coming out of repository imports it should probably clear them up too. I'd expect queue behavior to generally be more reasonable around repository imports once you pick up those changes.

Nope we don't have that integrated in yet (just missed our last cutoff). Once we get that pulled in I'll confirm it clears up the issue.

I tried adding a build-step to collect the summary from jenkins, but it turns out that the next step doesn't run anyway if the first step failed (Which is the case I'm more interested in), so back to square one with this.

Did you experience these issues after D16585? My expectation is that it should be almost impossible for PhabricatorRepositoryCommitHeraldWorker tasks to stall the queue for other priority-2000 tasks after that change.

@20after4, just a heads up about this per discussion in T7944. Here's what we expose today:

Well, just Conduit for editing -- querying is already in D16592.

I think making authorizations transactional isn't too much work. They already have PHIDs, and aren't edited in very many places.

I'm comfortable bringing this stuff upstream today:

I'm not sure there's a way to do that's easier than cloning the repository. You can use git clone --depth 1 to only load the current version of each file, instead of the entire history.

Evan, do you think there is a route forward with this that would be feasible for me to implement upstream?