Page MenuHomePhabricator

Aggregate corpus data in Ferret field rows
ClosedPublic

Authored by epriestley on Aug 30 2017, 2:11 PM.
Tags
None
Referenced Files
F18110164: D18497.id44429.diff
Mon, Aug 11, 4:55 PM
F18093567: D18497.id44443.diff
Thu, Aug 7, 7:25 PM
F18024295: D18497.id.diff
Sat, Aug 2, 9:18 PM
F17959748: D18497.diff
Fri, Aug 1, 8:34 AM
F17702926: D18497.diff
Jul 16 2025, 12:05 AM
Unknown Object (File)
Jun 21 2025, 4:00 PM
Unknown Object (File)
Jun 1 2025, 6:37 PM
Unknown Object (File)
May 31 2025, 4:59 AM
Subscribers
None

Details

Summary

Ref T12819. This addresses two issues:

  • One practical issue is that right now, if you search for "dog cat", and they appear in different fields (for example, "dog" appears ONLY in the title, while "cat" appears ONLY in a comment) we won't find the document. This is somewhat rare -- usually, if "dog" appears in the title, it's also repeated in the description -- but I think clearly a bug. To attack this, start automatically creating a virtual "ALL" field with the full document text which we'll use as the primary thing we match against.
  • For fields which may occur more than once -- today, only comments -- aggregate them all into one big "all of the text" row instead of writing one row per comment. This partly addresses the first point ("dog" in one comment and "cat" in a different comment won't be found) and partly makes some of the query gymnastics easier.
Test Plan

Ran bin/storage upgrade, ran bin/search index <Txxx>, saw sensible corpus values in the database:

mysql> select * from maniphest_task_ffield\G
*************************** 1. row ***************************
          id: 3
  documentID: 1981
    fieldKey: full
   rawCorpus: This is the task title
This is the task description.
normalCorpus: thi the task titl
thi the task descript
*************************** 2. row ***************************
          id: 4
  documentID: 1981
    fieldKey: titl
   rawCorpus: This is the task title
normalCorpus: thi the task titl
*************************** 3. row ***************************
          id: 5
  documentID: 1981
    fieldKey: body
   rawCorpus: This is the task description.
normalCorpus: thi the task descript
3 rows in set (0.00 sec)

Diff Detail

Repository
rP Phabricator
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

This revision is now accepted and ready to land.Aug 30 2017, 2:26 PM
This revision was automatically updated to reflect the committed changes.