Add an upstream JSON fragment lexer
ClosedPublic
Actions

Authored by epriestley on Oct 16 2015, 2:54 PM.

Details

Reviewers

chad
gd

Maniphest Tasks

T9566: Timeouts when highlighting source with very long lines using pygmentize

Commits

rPHU5e2297f15880: (stable) Promote 2015 Week 43
rPHUdb3667a62f66: Add an upstream JSON fragment lexer

Summary

Ref T9566. This seems to be >100x faster than an older version of pygmentize for JSON inputs with a single long line. See discussion in D14273.

Test Plan

Verified runtime improvement by comparing scripts/highlight.php on the one-line.json test case from D14273 with the new lexer enabled and commented out. Saw >10x local improvement. My local base runtime is also >10x better than the runtime from D14273. My laptop might be a little faster than whatever that hardware was, but this is probably mostly pygmentize improvements in my most recent version.
Added a test case.
Used the highlighter to produce reasonable-looking results:

Screen Shot 2015-10-16 at 7.51.42 AM.png (230×350 px, 19 KB)

Diff Detail

Repository

rPHU libphutil

Branch

json1

Lint

Lint Warnings

Severity	Location	Code	Message
Warning	src/markup/syntax/highlighter/__tests__/jsonfragment/basics.expect:2	TXT3	Line Too Long
Warning	src/markup/syntax/highlighter/__tests__/jsonfragment/basics.expect:3	TXT3	Line Too Long
Warning	src/markup/syntax/highlighter/__tests__/jsonfragment/basics.expect:4	TXT3	Line Too Long
Warning	src/markup/syntax/highlighter/__tests__/jsonfragment/basics.expect:5	TXT3	Line Too Long
Warning	src/markup/syntax/highlighter/__tests__/jsonfragment/basics.expect:6	TXT3	Line Too Long
Warning	src/markup/syntax/highlighter/__tests__/jsonfragment/basics.expect:7	TXT3	Line Too Long
Warning	src/markup/syntax/highlighter/__tests__/jsonfragment/basics.expect:8	TXT3	Line Too Long
Warning	src/markup/syntax/highlighter/__tests__/jsonfragment/basics.expect:10	TXT3	Line Too Long
Warning	src/markup/syntax/highlighter/__tests__/jsonfragment/basics.expect:11	TXT3	Line Too Long

Unit

Tests Passed

Build Status

Buildable 8308
Build 9526: Run Core Tests
Build 9525: arc lint + arc unit

Event Timeline

epriestley updated this revision to Diff 34496.Oct 16 2015, 2:54 PM

epriestley retitled this revision from to Add an upstream JSON fragment lexer.

epriestley updated this object.

epriestley edited the test plan for this revision. (Show Details)

epriestley added reviewers: gd, chad.

epriestley added a task: T9566: Timeouts when highlighting source with very long lines using pygmentize.

Actually, just fix lint by excluding __tests__/data/, which seems reasonable for all similar cases.

Just exclude every /__tests__/data/ directory from all linters.

chad accepted this revision.Oct 16 2015, 3:20 PM

chad edited edge metadata.

This revision is now accepted and ready to land.Oct 16 2015, 3:20 PM

gd accepted this revision.Oct 16 2015, 3:32 PM

gd edited edge metadata.

(I'm going to hold this until the release cut tomorrow morning since some issues might crop up once it gets a little more use and it sounds like "upgrade pygments" fixed most of this anyway.)

Closed by commit rPHUdb3667a62f66: Add an upstream JSON fragment lexer (authored by epriestley, committed by epriestley). · Explain WhyOct 17 2015, 11:56 AM

This revision was automatically updated to reflect the committed changes.

epriestley mentioned this in T10165: Differential syntax highlighting can't handle ES6 backtick strings.Jan 16 2016, 2:44 AM