Wrong unicode char reporting.

[lunny]

[wolfogre]

It is a Chinese unicode ’, my typo.

[silverwind]

I agree, this detection is still far too agressive. It should only warn on actual suspicious stuff, like RTL characters and such, not "uncommon" punctations.

[wxiaoguang]

Also agree that "the detection is too aggressive"

At least, I think we can make the detection skip the Markdown/HTML/XML/etc types. That detection only makes sense for source code files.

[silverwind]

I would prefer a solution that adds a config option ui.CHARACTER_HIGHLIGHT=bidi,ambiguous that triggers on certain character classes:

1. bidi: Nefarious BIDI characters like RTL Override character. I think this list on Wikipedia is accurate.
2. ambiguous: Also highlight ambiguous characters like non-standard whitespace and puncation.

The default should be bidi to cover CVE-2021-42574, but admins could also set it to empty string to disable the checking completely.

[silverwind]

At least, I think we can make the detection skip the Markdown/HTML/XML/etc types. That detection only makes sense for source code files.

No, don't skip any files. BIDI exploits can also be in the filename. From the linked page:

when using it in the file name my-text.'U+202E'cod.exe, the file name is actually displayed as my-text.exe.doc

[wxiaoguang]

I think it worries too much.

https://github.com/wxiaoguang/playground/wiki/test-202e

$ xxd playground.wiki/test-202e.md
00000000: 6162 63e2 80ae 7879 7a                   abc...xyz

[silverwind]

GitHub only does it for code. I guess that is where it matters the most. So we could do two options with these defaults:

[ui]
CHARACTER_HIGHLIGHT=bidi
CHARACTER_HIGHLIGHT_INCLUDE_MARKUP=false

[wxiaoguang]

GitHub only does it for code.

Yup, for "source code" only. For markdown in code repo:

GitHub only does it for "source code", not for rendered contents.

---

Update: it seems that the "display invisible char" button on the github page is broken .... at least it doesn't work for me 😂

[silverwind]

Yeah, that's what I meant, source code, e.g. stuff that renders in a monospace font 😉.