Page MenuHomePhabricator

Implement ngram search for File objects
ClosedPublic

Authored by amckinley on Apr 17 2017, 7:14 PM.
Tags
None
Referenced Files
Unknown Object (File)
Sun, Jun 26, 12:37 AM
Unknown Object (File)
Fri, Jun 17, 4:30 PM
Unknown Object (File)
Thu, Jun 16, 8:58 PM
Unknown Object (File)
Tue, Jun 14, 3:19 PM
Unknown Object (File)
Jun 6 2022, 9:35 PM
Unknown Object (File)
Jun 4 2022, 3:50 AM
Unknown Object (File)
Jun 4 2022, 12:31 AM
Unknown Object (File)
May 28 2022, 2:33 PM
Subscribers

Details

Summary

Follows the outline in D15656 for implementing ngram search for names of File objects. Also created FileFullTextEngine, because without implementing PhabricatorFulltextInterface, ./bin/search complains that File is not an indexable type.

Test Plan
  • ran ./bin/storage upgrade to apply the schema change
  • confirmed the presence of a new file_filename_ngrams table
  • added a couple file objects
  • ran bin/search index --type file --force
  • confirmed the presence of rows in file_filename_ngrams
  • did a few keyword searches and saw expected results

Diff Detail

Repository
rP Phabricator
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

resources/sql/autopatches/20170417.files.ngrams.php
5 ↗(On Diff #42576)

Need to remove this unless we want to force a reindex of all documents during storage upgrade.

src/applications/files/search/FileFulltextEngine.php
3 ↗(On Diff #42576)

Not sure if this class needs to exist, but I couldn't force reindexing without implementing PhabricatorFulltextInterface.

Note that this also doesn't clean up ngrams once files are deleted. I'm going to follow along with D14846: Implement basic ngram search for Owners Package names to fix that.

This is right, except that we probably should't implement FulltextInterface or fulltext-index these. bin/search index was written before the ngrams stuff was written, let me see if I can send you a quick patch to let it work without needing fulltext.

The cleanup should happen automatically as a side effect of DestructionEngine, but neither PhabricatorFileTemporaryGarbageCollector nor PhabricatorFileDeleteController use the destruction engine properly (bin/remove destroy will). Swapping those to use a destruction engine should fix the ngrams stuff, maybe as part of T10828.

To use the destruction engine, replace this:

$file->delete();

...with this:

$engine = new PhabricatorDestructionEngine();
$engine->destroyObject($file);

D17704 should fix the bin/search index stuff -- it ended up easier than I thought, assuming it works.

  • removing FileFulltextEngine
  • removing migration to force file reindex
  • update File deletion logic to use DestructionEngine
This revision is now accepted and ready to land.Apr 18 2017, 12:35 AM
This revision was automatically updated to reflect the committed changes.