Both bak volumes are now swapped. The old volumes are detached as dbak001.phacility.net-old and dbak002.phacility.net-old. I'll delete them after the deployment on Saturday if no issues arise before then.

I haven't gotten any emails yet so I may have to go muck with the WHOIS stuff and make sure there's a valid email somewhere -- phurl.io itself has no mail or MX records.

Upgrade dbak001 to 128GB (from 64GB).
Upgrade dbak002 to 128GB (from 64GB).

Swap notify001 to notify002 in the LB.

I think this is basically "node is bananas" and our AMI is Ubuntu 14 which ships with "Node for DOS".

Just sent a cert request for phurl.io. (Actually it should be two, one for each region).

I'm bumping into T12171 when bringing up the new host. I'm going to take another stab at figuring out what's going on there because the workaround I found in that task is ridiculous.

Switch all SSL to AWS.

Test that moving SSL termination to nlb001 works.

Oh, sorry, misread -- that makes more sense. phurl.io is the only one we serve anything from ourselves right now, and I think the only one we have plans to serve anything from.

Yeah I'm just talking about requesting the SSL certs, not moving the domains. Unless I'm missing something, I don't think there's any way to get stuck just by getting the certs ready.

Let's make sure all the SSL is swapped first just so we don't get into trouble if we make it halfway, run into issues, and something expires, but that's be helpful once SSL is in the clear. I'm not sure if nlb was the last case of SSL terminating somewhere other than LBs or not, but I think there's one left that I just don't remember offhand.

I can go request these certs via the AWS cert manager. We can also do it via CloudFormation, which will reduce the number of clicks significantly (and make it trivial to request all the same certs in multiple regions.

Replace notify001, which is scheduled for AWS downtime on September 20th.

This is resolved by the Ferret engine, which can execute all parts of the query logic in MySQL.

This is resolved by the Ferret engine, using title:...:

This is resolved by the Ferret engine:

See also PHI64.

Per above, not planning to actually go forward with the GC step since the impact isn't ultimately very large.

From T12932, I don't think it's especially promising as a way forward, so I'm just going take an operations pathway instead (see T12978), so I expect to just move to unprototype the Ferret engine here.

kind of sad this isn't called pherret

Looking at the actual data, I'm less sure this is actually a good strategy. Here's the data for this install, considering the production configuration of storage.mysql-engine.max-size as 65535:

The actual indexes are fairly large (3GB for ~500MB of task data, although lispum tasks have an exceptionally large amount of text).

I expect T12819 will ultimately remove all tables with FULLTEXT indexes, which should moot this.

bin/storage adjust and bin/storage upgrade now run ANALYZE, unless users opt-out with flags.

We generally stopped seeing this after moving to InnoDB FULLTEXT, which seems less prone to table crashes than MyISAM FULLTEXT was. See also T12819 for the fate of InnoDB FULLTEXT.

All the search which was previously driven by InnoDB FULLTEXT is now driven by the Ferret engine on this install.

Calling this resolved since there's no remaining upstream action, feel free to send diffs for lexers if you build them.

Here's a closer look at what's probably happening:

I briefly hit a bizarre case where a Ferret engine query took 10 seconds to find a document in 163 projects. However, running ANALYZE TABLE on the ngrams table resolved this completely. I suspect the ngrams join may require some tweaking (and maybe a bin/storage analyze). Analyzing the Maniphest table actually seems to improve performance by ~50% too (???) although that's not a very scientific measurement.

Yeah -- we'll take the patch in the latter case (porting lexers to PHP) but may not have time to review them for a while. We aren't interested in any version of options/support for using HTTP to talk to Pygments in the upstream.

Writing, deploying, and supporting an additional service is an enormous amount of work for us and not a solution I'm interested in pursuing for this problem

The only way forward on Pygments performance with upstream support is porting language lexers to PHP. Writing, deploying, and supporting an additional service is an enormous amount of work for us and not a solution I'm interested in pursuing for this problem: it burdens us with a large ongoing support cost instead of very small mostly-one-time lexer porting costs.

(As a side note, the way we'd support this on our ubuntu phabricator machine is to write an upstart script to start the pygments server on startup -- it would also restart the server if it died -- and to run the server under pypy for even more performance happiness.)