Related: #369

micromark/micromark#190 (comment)

The current specification is thought to treat an isolated surrogate code unit (e.g. "\ud800") as non-punctuation:

• The general category of all surrogate code points is Cs, not started with P or S.
• The character in the specifications is an Unicode Code Point, not an Unicode Scalar Value (surrogate code points are not included in scalar values)

However, once it is replaced with U+FFFD by e.g. String.prototype.toWellFormed, a conflict occurs: U+FFFD is a punctuation because its general category is So.

Due to this conflict, implementations cannot make the input string well-formed (replace invalid code unit sequences with U+FFFD) in advance.

Example:

• Input is: "a*\ud800*" in the UTF-16 form (e.g. JS)

• An implementation will do:

  1. Convert the input to HTML
  2. Replace isolated surrogate code units (\ud800 for the above example) with U+FFFD

• If the implementation do i. first, it will get: <p>a<strong>&#xFFFD</strong></p>.

• If the implementation do ii. first, it will get: <p>a*&#xFFFD*</p>.

To fix this conflict, I think that it is needed to define that whether ill-formed code unit sequences are treated as punctuations or not is unspecified behavior. This change will reduce the burden to create an implementation and will not bring breaking changes on any existing normal Markdown document files.

The following screenshot shows that an escape like &#xD800 in HTML is replaced with U+FFFD when rendered by Firefox:

It shows that a code point escape for an isolated surrogate pair in HTML (and Markdown) is ultimately equivalent to &#xFFFD.