I believed this was semi-intentional, although the downstream report suggests it worked at one point, so maybe I just made a mistake.

In this case, use the AphrontWriteGuard::beginScopedUnguardedWrites() method to surround the block -- you can grep for usage, but it will look like this:

All queues have emptied, and repo002 is back in a healthy range again:

rP9329e6a12d9f improved the rate at which the Herald queue is draining by approximately 10x.

I've moved the largest instance from repo002 to a new empty shard, which has put both shards into fairly reasonable range. Queues still aren't draining as quickly as I'd like/expect.

I moved the largest instance from shard 002 to shard 006. This took about 35 minutes, most of which was backup/transfer time. I hit a couple of minor issues, but things appear to have worked properly overall.

I migrated a test instance, which appears to have worked properly. I'm going to run though things again, then move a large instance off repo002.

• Suspend the instance: Admin → Instance → Manage Instance → Suspend Instance
• Stop the daemons:

I've provisioned and deployed a new shard, but not yet opened it for allocations. I'm going to launch and then migrate a test instance next.

I let the queues drain for a bit overnight. One of the large queues is down from ~16K items to ~12K items in the last 8 hours, but I don't want to just let this sit for 24 hours to "resolve" it. I'm going to look at moving instances across shards instead, which is a capability we need to automate eventually anyway.

Is this reproducible using fields available in the upstream? How can I reproduce this issue against a clean, vanilla, up-to-date copy of Phabricator with no custom code?

Where is "Platform--MUST-->" coming from? That isn't a field we provide in the upstream.

Looking at improving the raw performance of bin/repository update, one fixable issue is that we run git for-each-ref three times:

I also identified an instance with a bad HTTP hook which had managed to queue up 8,000 tasks in the queue. These already have scaling backoffs and likely didn't contribute a meaningful amount of load, but we could perhaps find a more graceful approach to handling bad webhook URIs during T11330.

I disabled the keys of the user running 30 clones/minute and notified the instance administrator. This appeared to create some headroom; I reverted the daemon pool size limits to the default and we're draining the queues at a somewhat more reasonable pace now.

@swisspol, that's what this refers to ("From T11666"):

A contributing issue is that one instance appears to be making clone requests for approximately 30 repositories per minute to this shard. This is easy to rate limit, but probably not easy to rate limit in a generalized, not-a-cluster-hack way.

One remaining issue with backoff/cooldown is that mirrors aren't included: a regularly-updated repository with broken mirrors still tries mirroring. This is somewhat more involved to fix, though (see T6131).

Those changes seem stable. Unclear how much they're really helping, but load is looking at least somewhat better. I'm going to dig into things a bit more and see if I can get a better sense of whether we're on the path to recovery or not.

D16574 retains update cooldowns (so we don't immediately update everything after a restart) and sets up for the following change.

Repository retries (failed mirrors, failed fetches) have no real backoff

I could alternatively throttle down the highly active instances (or all instances on this shard) but that's not too great.

• Maybe a good time to launch new shards? Last tier size increase was 10 months ago (late November, 2015) when I fixed the allocation algorithm to be not a huge wasteful pile of garbage that required us to constantly throw hardware at it.
  • Maybe the allocation algorithm should consider active users, not just active instances? I think the 002 shard has the most active users by a hefty margin, just by luck of the draw. But this will tend to be moot at scale.
  • Maybe start moving instances between shards? This seems hard to do without downtime today, and doing all the legwork to reduce downtime through clustering is foreboding.

And you can technically:

Oh, right. That works, but I think isn't documented:

We could conceivably special-case this since setting an empty attribute to a value isn't meaningful, I suppose.

Some somewhat-good news is that Google Calendar just gives you an .ics blob of garbage to describe recurrence rules:

Things don't look promising.

I think we should be able to accommodate that fairly easily.

More generally, here's how I'm planning to model imports, which seems largely consistent with other systems:

Yeah, I fiddled with that but didn't immediately have any luck. It's possible that there's some complex condition like non-self organizer + non-self sender or something. If it only requires non-organizer, it should already work today (e.g., on this install).

Sort of. I'd like to have use cases (what are you trying to do?) first, and I'm not certain what the best representation of dates and times is for external consumption.

This probably already works, on the backend, but needs:

I think the first part of this should be fixed in HEAD now, thanks for the report!

The "multiple packages" issue may have been a symptom of the dominion/archive problem, here's a case I hit while testing that made it look like there was a second issue:

D16564 might fix this but I need to actually test it.

Also, what is the expected behavior if two Owners packages point to the same path?

Is it possible that this is slightly narrower: "package dominion rules incorrectly consider archived packages"? That seems to fit with the observations, but maybe I'm overlooking something.

See Contributing Feature Requests for help filing a feature request we can accept upstream.

Feel free to submit a diff for this, should just involve adding some stuff to HarbormasterBuild in getFieldSpecificationsForConduit() and getFieldValuesForConduit().

I'm having trouble making GMail provide a reasonable pathway forward for getting .ics events onto Google Calendar. Ostensibly this just works:

ICS primarily identifies users (organizers and attendees) by email address. Phabricator does not normally disclose email addresses, and I don't want to let someone scrape every email address out of Phabricator by inviting every user to an event and downloading the .ics.

Even if everything else defaults to public, I don't think that's an especially strong argument on its own for copying that default. There are a lot of other cases where we diverge from common practice because we think there's a stronger case for a different default or behavior.

Sorry, we aren't interested in maintaining any version of this change in the upstream at this time.

D16465 has details on how to move forward here if anyone wants to steal it before @michaeljs1990 gets back to it.

Hmm, I guess I don't have the same experience -- the defaults are what I expect and the UI doesn't seem difficult to me (rough in places, but the basic flows are what I expect/want).

This stuff is defined in PhabricatorUser->loadEditorLink(), which ain't great but probably not worth generalizing.

Of the visible rooms I'm a member of, 14 out of 15 are private (the other one is General Chat).