This is resolved by the Ferret engine, which can execute all parts of the query logic in MySQL.

This is resolved by the Ferret engine, using title:...:

This is resolved by the Ferret engine:

See also PHI64.

Per above, not planning to actually go forward with the GC step since the impact isn't ultimately very large.

From T12932, I don't think it's especially promising as a way forward, so I'm just going take an operations pathway instead (see T12978), so I expect to just move to unprototype the Ferret engine here.

Looking at the actual data, I'm less sure this is actually a good strategy. Here's the data for this install, considering the production configuration of storage.mysql-engine.max-size as 65535:

The actual indexes are fairly large (3GB for ~500MB of task data, although lispum tasks have an exceptionally large amount of text).

I expect T12819 will ultimately remove all tables with FULLTEXT indexes, which should moot this.

bin/storage adjust and bin/storage upgrade now run ANALYZE, unless users opt-out with flags.

We generally stopped seeing this after moving to InnoDB FULLTEXT, which seems less prone to table crashes than MyISAM FULLTEXT was. See also T12819 for the fate of InnoDB FULLTEXT.

All the search which was previously driven by InnoDB FULLTEXT is now driven by the Ferret engine on this install.

Calling this resolved since there's no remaining upstream action, feel free to send diffs for lexers if you build them.

Here's a closer look at what's probably happening:

I briefly hit a bizarre case where a Ferret engine query took 10 seconds to find a document in 163 projects. However, running ANALYZE TABLE on the ngrams table resolved this completely. I suspect the ngrams join may require some tweaking (and maybe a bin/storage analyze). Analyzing the Maniphest table actually seems to improve performance by ~50% too (???) although that's not a very scientific measurement.

Yeah -- we'll take the patch in the latter case (porting lexers to PHP) but may not have time to review them for a while. We aren't interested in any version of options/support for using HTTP to talk to Pygments in the upstream.

The only way forward on Pygments performance with upstream support is porting language lexers to PHP. Writing, deploying, and supporting an additional service is an enormous amount of work for us and not a solution I'm interested in pursuing for this problem: it burdens us with a large ongoing support cost instead of very small mostly-one-time lexer porting costs.

See PHI54 for some additional discussion.

Unclear if building a search engine by just doing a lot of JOINs actually scales or not. Seems OK here.

See also PHI45.

See also PHI52.

No, we don't want to bring anything upstream if we don't support or maintain it.