The best place for this kind of question is now your organization's support pact -- @tammybutow or @alexmv should be able to help you get access.

@epriestley so maybe this isn't the right place to leave this comment (let me know if I should create a new task?) but I'm finding this is acting in a way that seems a bit inconsistent. In particular, I added a new boolean custom field --

Another thing we can do here which will generally reduce the operational load is to be more selective about how we dump/export tables from MySQL. Broadly, we have three general classes of tables today:

I've pushed the MySQL configuration tweaks here, and manually built the common ngrams table with a 0.15 threshold.

(Basically, a notification is just a copy of a feed story which we deliver directly to an individual at the same time we publish the feed story, so feed and notifications are intertwined.)

Mostly right: each node server process has both a "client" service and an "admin" service on different ports. Everything else is pretty much accurate, except that "admin server -> client server" is generally inside the same process.

I realized I don't really understand the architecture of the notifications system or our deployment of it. Tell me what I got right:

I'm also going to increase the db tier innodb_buffer_pool_size from 4GB to 6GB. The hosts have 7.5GB of RAM and 8GB of swap, and are pretty much sitting there with a little under 3GB of RAM mostly unused and no other load.

The thing I'm mostly looking at is how many queries are pushed down to 0 ngrams, i.e. they can't use the ngram index.

I looked at how the last 1,000 queries on this instance would be affected by different "common" thresholds. Note that these charts are all sort of garbage (X axis is nonlinear) because I couldn't figure out how basic spreadsheet software works.

@epriestley I've pinged you via pm about prioritization.

@epriestley
After i upgraded my application to the newest version，which swap the search engine to ‘Ferret’，I reindexed several times，but I can not get any result.These is my version information：
phabricator33756bcf1d70ea5579dff1ab276bbe660d10494c (Tue, Oct 3) (branched from f9110b87abf337dd1e7714d755775e53cffd4db9 on origin)arcanist0a7f403333fe9082b39bd007b9d5f9e765c8b9ce (Tue, Oct 3) (branched from c804c5026011f27614a7bbdb2bb32cab590d68ca on origin)phutilb400c6b04bb247a3e0f1941390bc450f36ac2ccd (Tue, Oct 3) (branched from 9f9c33797a3ebbf1c4dcaa474a0c4e0b32d5396a on origin)diff2.8.1 at /usr/bin/diffgit1.7.1 at /usr/bin/githgNot Availablepygmentize2.0.2 at /usr/bin/pygmentizesvn1.6.11 at /usr/bin/svn

Not a problem - thanks for fixing that so quickly.
I can confirm that fixed the issue for us.

Here are some possible structural changes we can make to the table, using the Maniphest ngrams table on this install as an example dataset (this is roughly 50x smaller than the target dataset):

There's still a big element of mystery here: why did MySQL sustain 80MB/sec of write I/O without daemons running? There "should" have been no queries against the fngrams table, and a large table "should" not require huge I/O volumes if it isn't being used. I don't think it's totally implausible that this was some kind of general thrash/swap/memory management issue in MySQL (the innodb_buffer_pool_size on these hosts if 4GB, and mysql was using all of it) but I would expect "paging-like" activities to cause a large read volume, not a large write volume.

Sorry to leave this hanging, but the issue was with our server configuration. We did not have the correct sudo rights confgured

Looks to be caused by this commit rPb75a4151c8996c5dfb1c8c14378fe3259666eac2

https://discourse.phabricator-community.org/t/daemons-tasks-crashing-in-a-loop-during-reindex/506

Running the reindex led to a couple crash looping tasks on my end:

(I'll fix this if you don't get there first, but have used up all my brain energy for today. And thanks for the report!)

Would it be acceptable to change PhutilRemarkupHyperlinkRule to catch this exception and behave as if it were a non-whitelisted protocol? (Happy to draft the patch, just want to check before I do so)

This change creates a slight problem for us at KDE as we have some historical commits in our Subversion repository which have URLs in them which are invalid. This exception means that:

• Herald spins forever trying to process these commits, failing every single time (currently up to 855 failures).
• The commit can't be viewed in the browser (See https://phabricator.kde.org/R883:271607)

Well, it half-works in email, so that's something!

You can do this

Ahhh, so you can't have multiple adjacent macros on the same line; that was my problem. I guess that could be special-cased, which would be linear with the number of "tokens that are macros coming before a non-macro" appearing on a line, but I'm probably the only one clamoring for multiple horizontally aligned parrots.

They need to be on their own line. I think this rule got added in 2011, shortly after I left FB.

Drive-by mention in PHI91.

It should also work with Mercurial (D18652 and D18653 were my Mercurial tests).

I missed your question in (1). A typical scenario for this would be when working on a feature which requires some refactoring work to be done. A revision is created with just the refactoring work and then a dependent revision is made which includes the actual feature work. During feature work additional refactoring might be made and need to update to that changeset and either add/amend changes, rebase the dependent revision back on top. Having the bookmarks auto created would help especially when using arc:bookmark when updating revisions back to phab.

Is the experimental fix git-only or can we test with mercurial as well?

• arc diff --cache (which was a passthrough for arc lint --cache) has been removed.

D18651 is marked as fixing this. The "fix" is very rough and will land in the experimental branch, see T11911 (and perhaps T12996) for followups.

• arc diff --no-diff (an internal workflow) has been removed.
• The internal "async lint" flag has been removed. This was an ancient, Facebook-specific flag for extensions which is obsoleted by Harbormaster.
• arc lint --only-new has been removed. This relied on upstream lint support which was never fully developed.
• arc lint --cache has been removed.
• The arc.lint.cache config option has been removed.
• arc lint --only-changed has been removed. arc now detects whether you want warnings on all lines or only changed lines by examining the other arguments (paths, --everything, --lintall, and --rev).

I think I'm going to land this stuff on the experimental branch since there's also some other related work which is better targeted there, and things are probably going to get worse before they get better.

Can we avoid filling the output buffer in subprocesses by testing if writes to stdout would block? (This should also be a "yes".)

See T12996 for spiritual followup.

This is now in stable, and deployed here and to the Phacility production cluster.

For record-keeping, this was asked again in https://discourse.phabricator-community.org/t/469

We had to write some code to make phage work with exec and fix the phd stop flow to use --force, but this deploy completed successfully.