Basically, the idea here is that if you search for:

• "the": It's pointless to use the ngrams table anyway since it has very little power to constrain this query, and falling back to LIKE on the ffield table is perfectly fine, since most documents match anyway.
• "the xylophone": Rare ngrams like "xyl", "ylo", and "oph" will be selected, which is correct and desirable. This kind of query should actually perform better.
• "the of in and to it is a my": This query is probably fine as a LIKE since all the terms are very common, but it's possible it performs a little worse.
• "source code program bug": This sort of query, where all terms are common words but not incredibly common words, might get worse. If 2+ of the ngrams occur in a small number of documents, performance should be pretty great. If only 1, performance should be reasonable. If 0, this may be the worst case (we fall back to LIKE and need to grep a lot of text). The choice of a "common" threshold will determine how many words fall into this awkward spot before the ngrams index kicks in.

This change doesn't alter search results, it just changes the situations in which we try to use the ngrams table to constrain the amount of text we eventually grep through with LIKE. So the worst case is just that we fall back to a big LIKE grep, which isn't great, but isn't horrible either.

Couple of bugfixes coming; MySQL does not retain trailing whitespace (???) on CHAR(...) columns so we have to fake it a bit.

Do we we need distinct, per-app tables for tracking common ngrams? I'm sure there's some variance between which ngrams are "common", but I'm not sure I can see the distribution affecting search result quality.

We don't strictly need separate tables, but it gives us more flexibility: we could change the thresholds for each document type independently, or scale the threshold to the size of the corpus, or add exceptions for particular document types, etc. We can also drop and rebuild one document type's index without having to rebuild everything else.

I don't know if we'll need to do any of that, but the only advantage I can come up with for a single table is that we need slightly fewer queries when running searches, but the cost of those queries is trivial and we could easily cache the results if it ever showed up on a profile.