[api-documenter] Officially support MDX in addition to Markdown

[matteematt]

Using api-documenter markdown -i dist -o docs/api is occasionally producing markdown files with random html comment sections in them.

When running the previous command it is seeming to produce markdown files with random html comments in them

For example, my colleuge recently created a PR where she was changing the variable name in a part of the monorepo, and now in an unrelated part the api-documenter is creating this change:

The changes she's made does not effect the tsdoc which is being used to generate the markdown files.

Details

I have also run into this issue multiple times myself. This is causing issues because we pull these markdown files into docusaurus and these are causing syntax errors there

There error appears to always be related to some symbols such as [] or <> - in the above screenshot it seems to be related to the [] array syntax of the permissions, and I have previously seen it related to the <> generic syntax of a class.

Standard questions

Please answer these questions to help us investigate your issue more quickly:

Question	Answer
@microsoft/api-documenter version?	^7.15.3
Operating system?	Reproduced in our repo on Mac and Windows
Documentation target?	Markdown
Would you consider contributing a PR?	Yes
TypeScript compiler version?	Version 4.7.4
Node.js version (node -v)?	v18.7.0

[emensch]

Dealing with the same issue and it's breaking my integration with docusaurus.

[aju]

We have same issue for our workflow, @emensch @matteematt did you find any workaround ?

[matteematt]

After some more digging I think the issue is something to do when there is different line endings (windows vs linux style) but it was a while ago so I can't fully remember.

We ended up doing this workaround as a local plugin in docusouras

/**
 * Run this on required files
 * Replacing html tags like <b> with their markdown equivalent `**` fixes the issue, as the line starts with markdown.
 */
function cleanseMarkdownContent(input) {
  return input.replace(/<!-- -->/g, "").replace(/<b>|<\/b>/g, "**");
}

[aju]

Thank you! We used https://www.npmjs.com/package/replace-in-files-cli to replace those comments.

[octogonz]

Dealing with the same issue and it's breaking my integration with docusaurus.

The <!-- --> comments are an escaping mechanism. The official escaping mechanism of CommonMark is the backslash \, but it doesn't work reliably across real world Markdown implementations. (For example the backslash may be visible on the final web page.) The problem is that most Markdown engines support proprietary syntaxes implemented as clumsy postprocessors that disregard the grammar. (As one example, your regular expression hack above will wrongly replace <!-- --> inside a ``` code block. But how to do it correctly? You'd need to implement a parser plugin.) We found that <!-- --> is pretty reliable for ensuring that characters are literally rendered by Markdown.

Docusaurus is a special case because it's NOT actually Markdown. Docusaurus uses a different language MDX which is JSX with some Markdown features retrofitted onto it. A lot of interoperability issues were fixed in MDX 2.0, but last I checked Docusaurus has been stuck on MDX 1.0 for a while.

That said, we use API Documenter + Docusaurus for the Rush Stack documentation project, which you can find here: rushstack-websites - /websites/api.rushstack.io.

I remember we found+fixed some MDX compatibility issues, but I don't remember the fixes offhand. You might take a look at that project.

[octogonz]

😆 Looks like we're doing the same thing in api-documenter-docusaurus-plugin:

api-documenter-docusaurus-plugin/src/DocusaurusMarkdownFeature.ts

    // Requires more investigation. HTML comments are ok, but the little empty
    // comments (<!-- -->) that are inserted in between links break the MDX parser
    // in Docusuarus.
    eventArgs.pageContent = eventArgs.pageContent.replace(/<!-- -->/g, ' ');

    this._apiItemsWithPages.add(eventArgs.apiItem);

Since Docusaurus has grown to be one of the most popular targets for API Documenter, let's see if we can come up with a better solution. I think there's two bits of work needed:

1. Add a setting to the api-documenter.json config file where you can specify your Markdown dialect, so that MarkdownEmitter doesn't have to be one-size-fits-all
2. Figure out a better set of escape sequences for Docusaurus

(Ideally someone would also upgrade the Docusaurus project to use MDX 2.0, but if it still hasn't happened by now, it's probably more involved than it seems.)

[octogonz]

We could also improve the plugin APIs so that API Documenter plugins can override MarkdownEmitter behaviors (rather than relying on String.replace()). But you shouldn't have to author a plugin to use Docusaurus -- it should be officially supported by API Documenter.

[aju]

@octogonz Thank you for great explanation and sharing your plugin :)