Can you walk me through what behaviors you saw/expected, what you searched for, what you found, what you're currently seeing, what you haven't figured out yet, etc?

For example, on the first part, are you asking that question because some (say, ".c") file is not highlighting at all, or because some file is not highlighting like you expect it to?

It was (and still is, partially) unclear to me under what conditions a file would be syntax highlighted with pygmentize. It seems that the following logic is applied:

1. If the filename matches an entry in syntax.filemap, then the file is syntax highlighted as specified.
2. The file is syntax highlighted based on its extension (although I'm not certain of this... is this something provided by pygmentize or does Phabricator do something extra?)
3. AFAIK, no other logic will be applied (so, an unknown file starting with a #!/bin/bash shebang will not necessarily be highlighted as a shell script)?

It is unclear whether file extensions need to be explicitly specified in the syntax.filemap. For example (and admittedly I haven't tested this), but will *.go files be highlighted as Golang OOTB or do I need to add "@\\.go$@" => "Golang" to syntax.filemap.

I am also unclear on the effects of the pygments.dropdown-choices option. I haven't seen such a dropdown before.

pygments.dropdown-choices is poorly named. It currently is used by the "Language" dropdown in Paste (e.g., on /paste/create/). In the future, it might be used by {T832} once that gets build. In both cases, it's desirable to show users something like a dropdown menu (so they have a better sense of what their options are and how to access them) but not to show every language (since Pygments supports like ten thousand obscure languages). Possibly, replacing the dropdown with a typeahead/tokenizer would be better, although the amount of garbage on the full list is fairly overwhelming.

The logic is fairly complex and will probably change a lot in the future (for example, see T3626). I worry that documenting it isn't very useful, since it will probably fall out of date and generally won't help users answer real questions.

Here's how it works today, in rough terms:

• We infer the language from the filename:
  1. If the path matches something in syntax.filemap, we use that result.
  2. If not, and the path has a suffix, we use the suffix.
  3. Otherwise, we fail to guess the language based on the filename.
• If we didn't get a guess based on the filename, we guess based on the file content:
  1. We look for various common patterns (like #!/bin, <?php, etc) to produce guesses.
• Based on the guess we have at this stage, we choose a highlighter:
  • If the guess is "php" and XHPAST is built, we use XHPAST.
  • If the guess is "php" and XHPAST isn't built, we use the builtin fragment highlighter.
  • If the guess is "diviner", "rainbow", "remarkup", "console", or "invisible", we use a builtin highlighter.
  • If the guess is anything else and we have pygments, we use pygments.
    • We pass the guess to pygments explicitly, so it ignores the file name and content.
  • If there's no pygments, we don't highlight.