Anything programmatically detectable should be raised through lint. Asking humans to detect it is still likely to not solve the core problem since they'll always be less reliable. .

well script-and-regex is NOT anything complex... or you can try and add zero-widht-space as a rule to arcanist Text linter or make own linter extension... anything is good ;)

I think that ArcanistTextLinter actually checks for ASCII right now, so it should be easy to enable for this case. See also D6050, which discusses selecting encoding for it.

Per above, these are already detected by the Text linter:

$ arc diff --conduit-uri=http://local.phacility.com
...
Linting...
>>> Lint for zws.txt:


   Error  (TXT5) Bad Charset
    Source code should contain only ASCII bytes with ordinal decimal values
    between 32 and 126 inclusive, plus linefeed. Do not use UTF-8 or other
    multibyte charsets.

    >>>        1 Here are some zero-width spaces: >>>​​<<<
 LINT ERRORS  Lint raised errors!
...

On the CLI on my system, at least, the colorization makes the location of the problem more clear than the raw text conveys:

There is also an "Invsible" syntax highlighter which visually shows invisible characters (like tabs and newlines). You can find this in View Options → Highlight As... → Invisible.

Currently, this only shows ASCII invisible characters (bytes \x00 through \x1F). We could reasonably expand this to include Unicode space characters as well, although there are a substantial number of them outside of ZWS (e.g., one result on Google: https://www.cs.tut.fi/~jkorpela/chars/spaces.html).

This list excludes some marks which should probably be rendered by a unicode-aware invisibles mode, like the various RTL markers and Byte Order Marks.

I think rounding these up and improving the "Invisible" highlighter would strictly be an improvement with no costs or downsides.

But I think this is probably not a good fit for the described problem (users accidentally inserting ZWS characters into source code). I think this is a good tool if lint raises an error, a developer isn't sure how to fix it and asks for help, and you can use this mode to see what the problem is. But it's not a good tool to find these characters in the first place, since the implied workflow (switch every file to "Invisibles" and look at it just in case a user made a mistake) is entirely unreasonable.

To step back, how did ZWS characters get there in the first place? This seems like a mistake which is normally very difficult to make.

However, we use ZWS characters in Differential to improve copy/paste behavior (so that when a user selects a chunk of text on the right hand side of a diff and copies it, they get only the right-hand side on their clipboard, instead of a big mishmash of left, right, and line numbers). Layered on top of this, we intercept browser "copy" events and strip the ZWS characters.

It is possible that this code has a bug (or, for example, doesn't work properly anymore on a newer version of some browse), and that the ZWS characters came from copy/pasting code out of Differential in the first place. If so, we should either try to fix the bug or consider removing all the ZWS magic entirely (I think the utility is very marginal, and not worth even a small number of users actually ending up with ZWS characters on their clipboards).

Upshot:

• The text linter we ship with enforces ASCII-only source code by default. It is what we use in this project, and it appears to completely solve this problem for us. If that sounds like a good approach, let us know if you need help configuring it. The most common reason this isn't a good approach for installs is that they prefer to write their source code in UTF8 (usually because non-English text appears in the source for some reason).
• If you want a UTF8 mode for the Text linter, we can note your interest on T11150 and give it some more weight for prioritization because you're a customer, but note that this is probably a bit complicated (i.e., I suspect users are likely to disagree on which characters are and are not allowed in "UTF8 source code" and that this mode will be less of a distinct mode and more of a collection of filter lists) and largely entangled with T10038, a large task.
• If the ZWS characters came from Differential in the first place, it would be helpful to know that (and as much information about how it happened as possible) so we can either prevent it or re-evaluate our behavior.

Finally, I don't think it's totally unreasonable to consider displaying ZWS and other characters in this class in Differential by default. I don't think it's the best solution to any problem here, but it shouldn't normally hurt anything.

However, it conflicts with the existing copy/paste behavior. As long as we're supporting copy/paste, I don't think we should transform the source for display: copy/paste should actually create a copy of the source that was selected. If we replace ZWS, etc., with visible text, this won't be true. (We could "highlight" these characters in a bright color instead of actually replacing them, but I worry this would be unclear. We could "highlight" + tooltip, maybe. Any approach in this vein will be complex because we can not integrate this directly into syntax highlighters in the general case, and run into HTML safety issues if we try to layer it on top.)

I don't think the copy/paste behavior is very important and I'm not opposed to removing it in favor of visible ZWS characters -- especially if it's causing ZWS characters to leak into source somehow -- but at least some users use it. This would be an easier call to make with evidence that the ZWS strategy is causing problems.

We could also look at a switch away from the ZWS strategy. For example, instead of ZWS, I believe we could use <img src="..." alt="🐐" /> or similar. If the oncopy magic failed, at least we'd get a super obvious failure: a herd of goats appear in your source code. However, this could break source that legitimately used goat characters (which are highly questionable, but I think less questionable than ZWS characters) and might cause other problems (at a minimum, we'd need to hide it from screenreaders).

• T2495: Improve whitespace expansion of tab literals is somewhat adjacent here.
• T11632: Display invisible characters on erroneous user input is a little related: if we generate canonical lists of "invisible" characters in UTF8, they should probably be shared by all three of: the code that detects when input in forms is invalid because of invisibles; the future UTF8-aware invisible mode; and the UTF8 text linting mode described in T11150.

FWIW I only code via copy/paste.

me too but not everyone is on our level

Two questions pick at me:

1. Which editor did the user use to add the ZWS in? Any editor I remember does something to warn you about it.
2. Why are there Unicode characters for "goat" (🐐) and for "cooked rice" (🍚)?

In Firefox, selecting a range which crosses an inline comment currently drops the line after the comment. Here, return arra does not copy:

The following line, after the problem line, copies fine:

This currently copies as:

rchAttachments() {
      id(new Differen

(This is possibly some kind of TextFragment / .textContent issue.)

I'm probably not going to fix this since it's pretty minor and moderately complicated to fix, but this interaction is not ideal:

• Begin selecting text on either side of the diff.
• With the mouse button held down, move the cursor over the line numbers.

Desired behavior: nothing special happens.
Actual behavior: the inline comment reticle is shown.