Sanitize UTF8 more aggressively to satisfy json_encode()

Authored by epriestley on Aug 24 2016, 2:52 PM.


Sanitize UTF8 more aggressively to satisfy json_encode()

Fixes T11525. Currently, there are some strings such that:


...fails. I encountered this with DarkConsole trying to JSON encode queries that inserted encrypted file data into the MySQL blob store, so basically random data.

There appear to be two cases we aren't handling well:

  • Overlong representations: Shorter characters can be written in an invalid way with more bytes. We previously allowed these -- sometimes -- but json_encode() does not. Instead, reject them. We already rejected overlong 2-character codes.
  • Surrogate characters: There is a range of surrogate characters reserved for use in UTF16 which json_encode() rejects. Just reject these ourselves, too.

Test Plan:
Wrote a bunch of test cases to cover this stuff, all of which now pass.

Fuzzed json_encode(phutil_utf8ize($string)) on random strings in a loop. Before these changes it would fail after a handful of attempts, in less than a second. After these changes, I ran it for several minutes and didn't see any failures.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11525

Differential Revision: