It's a persistent problem. I've already tried to restart the deamons, the OS itself, but the problem persists. Phabricator was installed in a CentOS 7.2 system following the steps in this link: https://www.vultr.com/docs/how-to-install-and-configure-phabricator-on-centos-7

How can we reproduce this locally? Do you have a specific version of Outlook / OS you are using?

This isn't a bug in Phabricator, then. Please review Contributing Bug Reports and Support Resources carefully for details on what we provide support with.

On the cache:

I have run to the same problem. I have restart the daemon multiple times to fix it.

I have run to the same problem. I have restart the daemon multiple times to fix it.

We use a text field for a 'reference number' which refers to tasks imported from bugzilla reference=bz123, or RT with reference=rt456

Is this a persistent problem, or one only around for a minute after you change a config setting or update?

I think this is the CSS that tries to cut off the titles interacting weirdly with other table CSS.

I believed this was semi-intentional, although the downstream report suggests it worked at one point, so maybe I just made a mistake.

In this case, use the AphrontWriteGuard::beginScopedUnguardedWrites() method to surround the block -- you can grep for usage, but it will look like this:

I've got the endpoint and controller set up, but I get the following error when I call PhabricatorFile::newFromFileDownload:

Click "home". to see your installed dashboard. This is the expected UI.

I tried with fields available in the upstream, so it can not be reproduced with vanilla up-to-date- copy.

Alright I've been able to narrow this down a little bit more. There's some error with how external/pear-figlet/Text/Figlet.php is smushing characters. Specifically lines 339-362 as well as the _rep($r) function. The equivalent function in the C code is here. I haven't been able to find an obvious discrepancy, but I suspect there's a subtle difference in how one of the regex functions is handling things or an error in one of the regexes themselves. This error also occurs with backticks and double-quotes:

All queues have emptied, and repo002 is back in a healthy range again:

I have that problem again. There is no relation between the file and the Diff. Don't know how that can happen...

rP9329e6a12d9f improved the rate at which the Herald queue is draining by approximately 10x.

I've moved the largest instance from repo002 to a new empty shard, which has put both shards into fairly reasonable range. Queues still aren't draining as quickly as I'd like/expect.

I moved the largest instance from shard 002 to shard 006. This took about 35 minutes, most of which was backup/transfer time. I hit a couple of minor issues, but things appear to have worked properly overall.

PHP 7.1 RC2 is now available, and the patch for async_signals was merged into what is presently the 7.1 branch.

I migrated a test instance, which appears to have worked properly. I'm going to run though things again, then move a large instance off repo002.

• Suspend the instance: Admin → Instance → Manage Instance → Suspend Instance
• Stop the daemons:

I've provisioned and deployed a new shard, but not yet opened it for allocations. I'm going to launch and then migrate a test instance next.

I let the queues drain for a bit overnight. One of the large queues is down from ~16K items to ~12K items in the last 8 hours, but I don't want to just let this sit for 24 hours to "resolve" it. I'm going to look at moving instances across shards instead, which is a capability we need to automate eventually anyway.

Is this reproducible using fields available in the upstream? How can I reproduce this issue against a clean, vanilla, up-to-date copy of Phabricator with no custom code?

We have defined that one. It is like our teams with the company. When new bug is created for example maps team, herald automatically sets the right person/team for the bug.

Where is "Platform--MUST-->" coming from? That isn't a field we provide in the upstream.

phabricator 51f8ec4487be02e290b1182a7cd5b4069fe7d49f (Tue, Sep 20) arcanist 9e82ef979e8148c43b9b8439025d505b1219e213 (Thu, Aug 25) phutil fa6519f4cea1c059bd381c20971cac413cc4760e (Thu, Sep 15)

@ksmith it's a Python script that you would run on your local machine. If you have arcanist setup for development it should be pretty simple to get it working. Ideally it would just be a plugin, but I didn't look into that approach yet.

Ok I have observed this now on this install, however only in the second haunt mode Z+Z:
https://secure.phabricator.com/D15067

Looking at improving the raw performance of bin/repository update, one fixable issue is that we run git for-each-ref three times:

I also identified an instance with a bad HTTP hook which had managed to queue up 8,000 tasks in the queue. These already have scaling backoffs and likely didn't contribute a meaningful amount of load, but we could perhaps find a more graceful approach to handling bad webhook URIs during T11330.

I disabled the keys of the user running 30 clones/minute and notified the instance administrator. This appeared to create some headroom; I reverted the daemon pool size limits to the default and we're draining the queues at a somewhat more reasonable pace now.

@swisspol, that's what this refers to ("From T11666"):

I don't know if this is related but since an hour or so, whenever I push I get an error even though the push appear to work (the commits show up in Diffusion):

$ git push
# Push received by "web.phacility.net", forwarding to cluster host.
# Waiting up to 120 second(s) for a cluster write lock...
# Acquired write lock immediately.
# Waiting up to 120 second(s) for a cluster read lock on "repo002.phacility.net"...
# Acquired read lock immediately.
# Device "repo002.phacility.net" is already a cluster leader and does not need to be synchronized.
# Ready to receive on cluster host "repo002.phacility.net".
Counting objects: 56, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (50/50), done.
Writing objects: 100% (56/56), 9.30 MiB | 1.84 MiB/s, done.
Total 56 (delta 32), reused 1 (delta 0)
# Released cluster write lock.
AphrontQueryException: #1364: Field 'messageCount' doesn't have a default value
To ssh://hyper@vault.phacility.com/diffusion/HYPER/hyper.git
   015051e..67b218b  master -> master
error: failed to push some refs to 'ssh://hyper@vault.phacility.com/diffusion/HYPER/hyper.git'

A contributing issue is that one instance appears to be making clone requests for approximately 30 repositories per minute to this shard. This is easy to rate limit, but probably not easy to rate limit in a generalized, not-a-cluster-hack way.

@epriestley Did you see the AphrontQueryException: #1364: Field 'messageCount' doesn't have a default value problem too?

Hummm I don't know if it was a transient error or I just happened to magically fix it by force pushing to the repo, but git pulls now work again on my Mac and in CI.

Pushing appears to work (as in the commits appear in Diffusion) but errors are printed:

$ git push
# Push received by "web.phacility.net", forwarding to cluster host.
# Waiting up to 120 second(s) for a cluster write lock...
# Acquired write lock immediately.
# Waiting up to 120 second(s) for a cluster read lock on "repo002.phacility.net"...
# Acquired read lock immediately.
# Device "repo002.phacility.net" is already a cluster leader and does not need to be synchronized.
# Ready to receive on cluster host "repo002.phacility.net".
Counting objects: 11, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (11/11), done.
Writing objects: 100% (11/11), 1.90 KiB | 0 bytes/s, done.
Total 11 (delta 10), reused 0 (delta 0)
# Released cluster write lock.
PHP Fatal error:  Uncaught exception 'FilesystemException' with message 'Requested path '/core/log/hyper' is not writable.' in /core/lib/libphutil/src/filesystem/Filesystem.php:1107
Stack trace:
#0 /core/lib/libphutil/src/filesystem/Filesystem.php(650): Filesystem::assertWritable('/core/log/hyper')
#1 /core/lib/libphutil/src/filesystem/PhutilDeferredLog.php(183): Filesystem::createDirectory('/core//log/hype...', 493, true)
#2 /core/lib/libphutil/src/filesystem/PhutilDeferredLog.php(155): PhutilDeferredLog->write()
#3 [internal function]: PhutilDeferredLog->__destruct()
#4 {main}
  thrown in /core/lib/libphutil/src/filesystem/Filesystem.php on line 1107
To ssh://hyper@vault.phacility.com/diffusion/HYPER/hyper.git
   d77b984..6eb6700  master -> master
error: failed to push some refs to 'ssh://hyper@vault.phacility.com/diffusion/HYPER/hyper.git'

One remaining issue with backoff/cooldown is that mirrors aren't included: a regularly-updated repository with broken mirrors still tries mirroring. This is somewhat more involved to fix, though (see T6131).

Those changes seem stable. Unclear how much they're really helping, but load is looking at least somewhat better. I'm going to dig into things a bit more and see if I can get a better sense of whether we're on the path to recovery or not.

D16574 retains update cooldowns (so we don't immediately update everything after a restart) and sets up for the following change.

Repository retries (failed mirrors, failed fetches) have no real backoff

I could alternatively throttle down the highly active instances (or all instances on this shard) but that's not too great.

• Maybe a good time to launch new shards? Last tier size increase was 10 months ago (late November, 2015) when I fixed the allocation algorithm to be not a huge wasteful pile of garbage that required us to constantly throw hardware at it.
  • Maybe the allocation algorithm should consider active users, not just active instances? I think the 002 shard has the most active users by a hefty margin, just by luck of the draw. But this will tend to be moot at scale.
  • Maybe start moving instances between shards? This seems hard to do without downtime today, and doing all the legwork to reduce downtime through clustering is foreboding.

Ohhh nice, thank you I missed that
=C124

And you can technically:

Oh, right. That works, but I think isn't documented:

It's an off-case for sure. We have a ton of data identifiers which all end with an equals-sign and a couple letters/numbers, which we abbreviate when referring to the datasets. I was actually trying to render the equals sign as part of the nav sequence and not just C124.

We could conceivably special-case this since setting an empty attribute to a value isn't meaningful, I suppose.