universal inline code block syntax for nim doc+rst

[timotheecour]

/cc @a-mr

proposal

add a syntax for doc comments and rst files allowing writing arbitrary inline code blocks with optional syntax indicator, as follows:

hello1 ``` abc ``` rest of comment # simplest cases, matches {abc}
hello2 ```py abc ``` rest of comment # matches {abc} syntax colored with py language (ditto below)
hello3 ```py abc  ``` rest of comment # matches {abc }
hello4 ```py  abc  ``` rest of comment # matches { abc }
hello5 ```py   a bc ``` rest of comment # matches {  a bc}
hello6 ```py `abc` ``` rest of comment # matches {`abc`}
hello7 ````py ```abc ```` rest of comment # matches {```abc} (just increase backticks as needed)

the syntax is:

prefix = N backticks (N>=3)
1 optional alphanumeric identifier indicating language
1 space
S: arbitrary string that cannot contain `prefix` but can contain spaces and backticks
1 space
prefix

benefits

• this can encode anything, including beginning/trailing whitespace, beginning/trailing backticks
• this can have optional language indicating how to syntax highlight
• this is non-ambiguous and easy to understand, no complex escaping is needed

note

• markdown cannot handle beginning/trailing whitespace, and, depending on the markdown interpreter, consecutive inner spaces inside will be collapsed to 1 whitespace
• rst also cannot handle beginning/trailing whitespace or arbitrary sequences of backticks, neither with interpreted literals (single backtick) nor with inline literals (double backtick), and the escaping rules are messy and complex to implement (nim's implementation is still broken, and there are multiple standards anyways)
• backward compatibility: triple backticks are broken (see inline triple backtick foo.bar in doc comment incorrectly identifies foo as lang, stripping it in docs Nim#17314) so this isn't an issue; instead we're giving new meaning to something that's currently broken
• this would not be the go-to inline code syntax (which would remain single backticks) but would be the sytnax to use for cases not well handled by single backticks.

this proposal is simple (both to understand and to implement), allows encoding anything, and has 0 edge cases that I know of.

open question

• I think we can also make this proposal work with N>=2 or even N>=1 instead of N>=3 given that the proposed syntax is currently illegal in those cases, so there would be no collusion with existing semantics, thanks to trailing space before last backtick(s)

[narimiran]

How important/crucial is this?
Shouldn't we focus on more important stuff than spending our time on implementing this and then fixing future bugs?

Thumb down from me, sorry.

[Araq]

I like the idea but ideally we should copy from some existing, well-known markdown or rst extension. If there are any...

[timotheecour]

maybe such extensions exist but I'm not awareof it, and people often complain about markdown/rst lacking such a feature. I don't like any of the workarounds mentioned in those links, which is why I ended up suggesting the simple solution in this RFC.

markdown:

• https://meta.stackexchange.com/questions/93581/how-can-we-display-additional-spaces-using-markdown
• https://stackoverflow.com/questions/15721373/how-do-i-ensure-that-whitespace-is-preserved-in-markdown

Use   instead of space characters.
 

• https://meta.stackexchange.com/questions/123584/markdown-code-snippets-dont-allow-trailing-spaces
  suggests using unicode Alt+0160, but typing it can be a pain and not obvious if you're reading some code and are unawaire it's using unicode, or if you copy paste it from the source instead of rendered output you'll get, well, unicode

• https://meta.stackexchange.com/a/105363/223860
  this one recommends zero-width unicode character, and has same caveats:

And then people copy-past zero-width characters in their code and wonder why it doesn't work... –

• https://meta.stackoverflow.com/questions/363471/wrong-parsing-of-in-comments

• https://meta.stackexchange.com/questions/105316/my-comment-code-block-was-not-converted-properly-is-this-a-bug

• https://meta.stackexchange.com/questions/79254/recent-change-of-inline-code-in-comments-leads-to-mistakes-in-old-comments/79344#79344

Spaces are not allowed as the start or end character in inline code blocks now. Do not include them, or it won't work.

• https://talk.commonmark.org/t/leading-and-trailing-white-spaces-in-code-blocks/628

rst:

• https://stackoverflow.com/questions/45534757/how-to-use-inline-code-with-a-trailing-whitespace

the preservation of whitespace cannot be guaranteed. If the preservation of line breaks and/or other whitespace is important, literal blocks should be used

The end-string must be immediately preceded by non-whitespace.". So it looks like it's not possible per definition. This answer has introduced a custom role for accomplishing a similar task however it requires modifying the Sphinx code

• https://stackoverflow.com/questions/39151571/how-to-add-leading-trailing-spaces-to-monospace-text-in-sphinx-restructuredtext

[a-mr]

There is already too much confusing syntax for adding even more.

Trailing/leading whitespace use cases are too marginal, let us leave it be.

For inline syntax highlighting we'd better focus on implementation of existing RST syntax:

See `def f(): return 3`:python:
 or `def f(): return 3`:py:

(In rst2html.py it already works if such roles are defined:

.. role:: python(code)
   :language: python
.. role:: py(code)
   :language: python

)

[timotheecour]

@a-mr the existing syntax is the one that's confusing and limited, with lots of edge cases it cannot handle, and people complain about it (see links in #355 (comment)).

In rst2html.py it already works [...]
Trailing/leading whitespace use cases are too marginal, let us leave it be.

are these too marginal too?

• rendering trailiing backslash \ in syntax highlighted code
• rendering backticks in syntax highlighted code (see rst2html)
• preserving multiple consecutive spaces
• rendering double/triple backticks

Instead of copying bad design, this RFC replaces it with something that's:

• trivial to understand
• requires no escaping (WYSIWYG) in particular easy to copy-paste
• is universal, and has no edge cases
• is easier to implement (because no escaping needed)

ie, it replaces something that sucks by something that doesn't suck.

eg, say I want inline block to represent these syntax-highlighted snippets for bash:

text before ```bash export MYPWD=`pwd`; ``` text after

or even this: ```bash export MYPWD=`pwd` ```

or this: ```cmd cd\``` (eg for windows cmd prompt to cd to root)

with rst2html how do you do that? say even the 1st case:

.. role:: bash(code)
   :language: bash

can't work: backtick needs escaping:
`export MYPWD=`pwd`;`:bash:

can't work with double backticks: roles are for single backticks:
``export MYPWD=`pwd`;``:bash:

no syntax highlighting:
``export MYPWD=`pwd```

this doesn't render correctly in rst2html:
`export MYPWD=\`pwd\`;`:bash:

this doesn't render correctly in rst2html:
`export MYPWD=\`pwd\`;\\`:bash:

gives:

with this RFC, this is simply:

text before ```bash export MYPWD=`pwd`; ``` text after
text before ```bash export MYPWD=`pwd` ``` text after
text before ```cmd cd\ ``` text after

[a-mr]

@timotheecour ,

Yes,

those are very minor use cases.

For them we can just advice users to use full .. code-block:: nim or its normal Markdown counterpart.

[a-mr]

A problem with trailing space looks completely artificial.

If you really have it then currently just explain it by a few English words :-)