Page MenuHomePhabricator
Create Task
Maniphest T9566

Timeouts when highlighting source with very long lines using pygmentize
Closed, DuplicatePublic
Actions

Description

pygmentize is really slow and expensive for large source but especially for long lines.

We had cases where users pasted large (almost 1MB) JSON one-liner in Paste application and pygmentize caused not only that particular paste page to display nginx "504 Gateway Time-out" error but also for Paste application home page rendering Paste application unaccessible for all users.

Since pages get killed and fail to cache the result Paste application is unaccessible until someone goes directly to paste edit page (../paste/edit/<id>) and changes language to "Plain Text".

Steps to reproduce:

  1. Have Pygments installed and enabled.
  2. Open Paste → Create Paste.
  3. Paste large JSON oneliner (e.g. ~1MB).
  4. Select Language: JSON (or JavaScript).
  5. Click Save.

Expected result:

  • Paste is saved and paste view page opens.

Actual result:

  • Paste is saved but pygmentize process causes view page to timeout.

I can assume this problem is not local to Paste application but anywhere where pygmentize source highlighting is involved.

I'll try to post a diff for this myself but any fix in upstream would be appreciated.

Revisions and Commits

rPHU libphutil
AbandonedD14273 Skip pygmentize for large source and too long lines
D14292rPHUdb3667a62f66 Add an upstream JSON fragment lexer

Related Objects

Event Timeline

gd created this task.Oct 14 2015, 11:57 AM
gd updated the task description. (Show Details)
gd added a project: Paste.
gd added a subscriber: gd.
gd mentioned this in D14273: Skip pygmentize for large source and too long lines.Oct 14 2015, 12:14 PM
gd added a revision: D14273: Skip pygmentize for large source and too long lines.Oct 14 2015, 12:16 PM
epriestley added a revision: D14292: Add an upstream JSON fragment lexer.Oct 16 2015, 2:54 PM
epriestley mentioned this in T9587: Detect old Pygments versions and encourage an upgrade.Oct 16 2015, 4:13 PM
joshuaspence added a subscriber: joshuaspence.Oct 17 2015, 5:15 AM
• ricky.liddard closed this task as a duplicate of T9415: PhabricatorApplicationTransactionPublishWorker requires unreasonable amount of memory for large numbers of mail recipients?.Oct 17 2015, 7:10 AM
gd added a comment.Oct 17 2015, 9:21 AM

I'm not sure this is related to "T9415: PhabricatorApplicationTransactionPublishWorker requires unreasonable amount of memory for large numbers of mail recipients?".

Anyway this task can be closed in favour of "T9587: Detect old Pygments versions and encourage an upgrade".

epriestley added a subscriber: epriestley.Oct 17 2015, 10:24 AM

That was just some random demo user clicking buttons. I'll merge 'em.

epriestley reopened this task as Open.Oct 17 2015, 10:24 AM
epriestley closed this task as a duplicate of T9587: Detect old Pygments versions and encourage an upgrade.
epriestley added a commit: rPHUdb3667a62f66: Add an upstream JSON fragment lexer.Oct 17 2015, 11:56 AM
epriestley mentioned this in Blog Post: Development Notes (2015 Week 42).Oct 17 2015, 1:46 PM
epriestley mentioned this in T9600: Paste should truncate very long lines in ApplicationSearch result list snippets.Oct 19 2015, 3:20 PM
Phabricator · Built by Phacility