Page MenuHomePhabricator

Raise the effective corpus size limit in "PhutilProseDifferenceEngine" by using "diff" for coarse passes
Closed, ResolvedPublic

Description

See PHI1408. An install has a large Phriction document which splits into about 3,000 blocks in the outermost pass of PhutilProseDifferenceEngine. This exceeds the effective limit of 128 blocks between the outermost changes in EditDistanceMatrix.

We can remedy this by doing coarse prose difference passes with diff instead of with PhutilEditDistanceMatrix. Specifically:

  • Split the corpus into blocks.
  • Hash each block, possibly normalizing them first.
  • Generate an edit string by using diff on the hashed intermediates.

This should scale to any realistic block count (e.g., ~100K).

Revisions and Commits

Restricted Differential Revision
Restricted Differential Revision
Restricted Differential Revision
Restricted Differential Revision

Event Timeline

epriestley created this task.
epriestley added a revision: Restricted Differential Revision.Sep 25 2019, 9:05 PM
epriestley added a revision: Restricted Differential Revision.Sep 25 2019, 9:36 PM
epriestley added a revision: Restricted Differential Revision.Sep 25 2019, 9:40 PM
epriestley added a revision: Restricted Differential Revision.Sep 25 2019, 10:12 PM
epriestley added a commit: Restricted Diffusion Commit.Sep 25 2019, 11:41 PM
epriestley added a commit: Restricted Diffusion Commit.Sep 25 2019, 11:47 PM
epriestley closed this task as Resolved by committing Restricted Diffusion Commit.Sep 25 2019, 11:50 PM
epriestley added a commit: Restricted Diffusion Commit.
epriestley added a commit: Restricted Diffusion Commit.