Pushing appears to work (as in the commits appear in Diffusion) but errors are printed:

$ git push
# Push received by "web.phacility.net", forwarding to cluster host.
# Waiting up to 120 second(s) for a cluster write lock...
# Acquired write lock immediately.
# Waiting up to 120 second(s) for a cluster read lock on "repo002.phacility.net"...
# Acquired read lock immediately.
# Device "repo002.phacility.net" is already a cluster leader and does not need to be synchronized.
# Ready to receive on cluster host "repo002.phacility.net".
Counting objects: 11, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (11/11), done.
Writing objects: 100% (11/11), 1.90 KiB | 0 bytes/s, done.
Total 11 (delta 10), reused 0 (delta 0)
# Released cluster write lock.
PHP Fatal error:  Uncaught exception 'FilesystemException' with message 'Requested path '/core/log/hyper' is not writable.' in /core/lib/libphutil/src/filesystem/Filesystem.php:1107
Stack trace:
#0 /core/lib/libphutil/src/filesystem/Filesystem.php(650): Filesystem::assertWritable('/core/log/hyper')
#1 /core/lib/libphutil/src/filesystem/PhutilDeferredLog.php(183): Filesystem::createDirectory('/core//log/hype...', 493, true)
#2 /core/lib/libphutil/src/filesystem/PhutilDeferredLog.php(155): PhutilDeferredLog->write()
#3 [internal function]: PhutilDeferredLog->__destruct()
#4 {main}
  thrown in /core/lib/libphutil/src/filesystem/Filesystem.php on line 1107
To ssh://hyper@vault.phacility.com/diffusion/HYPER/hyper.git
   d77b984..6eb6700  master -> master
error: failed to push some refs to 'ssh://hyper@vault.phacility.com/diffusion/HYPER/hyper.git'

One remaining issue with backoff/cooldown is that mirrors aren't included: a regularly-updated repository with broken mirrors still tries mirroring. This is somewhat more involved to fix, though (see T6131).

Those changes seem stable. Unclear how much they're really helping, but load is looking at least somewhat better. I'm going to dig into things a bit more and see if I can get a better sense of whether we're on the path to recovery or not.

D16574 retains update cooldowns (so we don't immediately update everything after a restart) and sets up for the following change.

Repository retries (failed mirrors, failed fetches) have no real backoff

I could alternatively throttle down the highly active instances (or all instances on this shard) but that's not too great.

• Maybe a good time to launch new shards? Last tier size increase was 10 months ago (late November, 2015) when I fixed the allocation algorithm to be not a huge wasteful pile of garbage that required us to constantly throw hardware at it.
  • Maybe the allocation algorithm should consider active users, not just active instances? I think the 002 shard has the most active users by a hefty margin, just by luck of the draw. But this will tend to be moot at scale.
  • Maybe start moving instances between shards? This seems hard to do without downtime today, and doing all the legwork to reduce downtime through clustering is foreboding.

Ohhh nice, thank you I missed that
=C124

And you can technically:

Oh, right. That works, but I think isn't documented:

It's an off-case for sure. We have a ton of data identifiers which all end with an equals-sign and a couple letters/numbers, which we abbreviate when referring to the datasets. I was actually trying to render the equals sign as part of the nav sequence and not just C124.

We could conceivably special-case this since setting an empty attribute to a value isn't meaningful, I suppose.

In case someone else ends up with my original problem, I'll mention that https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1518457/comments/69 fixed it.

Some somewhat-good news is that Google Calendar just gives you an .ics blob of garbage to describe recurrence rules:

Things don't look promising.

When you load the configuration web page, the effective configuration is highlighted yellow. If the database value is not null, it might be overriding it, but ./bin/config get should show all the configured values.

I think we should be able to accommodate that fairly easily.

@lyahdav : That looks pretty cool. It's not quite clear to me where it runs. Is this a script that one would install on the same box that phab itself is running on?

I found this link: https://secure.phabricator.com/book/phabricator/article/advanced_configuration/

Here are 2 use-cases which are both essentially the same in all the important ways:

(comment posted twice for some reason)

I thought I'd throw our use case in here in case it helps:

More generally, here's how I'm planning to model imports, which seems largely consistent with other systems:

Yeah, I fiddled with that but didn't immediately have any luck. It's possible that there's some complex condition like non-self organizer + non-self sender or something. If it only requires non-organizer, it should already work today (e.g., on this install).

Sort of. I'd like to have use cases (what are you trying to do?) first, and I'm not certain what the best representation of dates and times is for external consumption.

@epriestley: Are you open to patches that add more information to the calendar.event.search response? It currently does not even include the event time which is a pretty crucial detail to make the results useful.

For anyone trying to get Kanban statistics / reports in the meantime, you can try this project I recently started working on:
https://github.com/lyahdav/analytics-limn-analytics-data/tree/kanban_stats. It generates a CFD (Cumulative Flow Diagram) and computes cycle time and lead time.

@raincoastchris the context of this issue caught my eye because I've really wanted Phab to generate a cumulative flow chart and Kanban statistics for some time now. I took a stab at it today and modified an existing python script to generate a cumulative flow chart and Kanban statistics (https://github.com/lyahdav/analytics-limn-analytics-data/tree/kanban_stats) but configuring it is a bit painful because there is no easy way to get the names of all the columns in a workboard right now.

I think this is nearly all resolved with NUX.

This probably already works, on the backend, but needs:

Let me know if there is a conversion diff I can follow.

Awesome! Thanks for the quick fix @epriestley!

I think the first part of this should be fixed in HEAD now, thanks for the report!

The "multiple packages" issue may have been a symptom of the dominion/archive problem, here's a case I hit while testing that made it look like there was a second issue:

D16564 might fix this but I need to actually test it.

Also, what is the expected behavior if two Owners packages point to the same path?

In T11650#194907, @epriestley wrote:

Is it possible that this is slightly narrower: "package dominion rules incorrectly consider archived packages"? That seems to fit with the observations, but maybe I'm overlooking something.

Is it possible that this is slightly narrower: "package dominion rules incorrectly consider archived packages"? That seems to fit with the observations, but maybe I'm overlooking something.

Nuance is our plan for routing tasks, we don't anticipate needing this functionality in Herald.

Ok, read that. Here is a breakdown of our wishes and what we did to implement them: