RFC: CommonMark compatibility, supporting multiple markdown/content parsers

[slorber]

💥 Proposal

People using Docusaurus don't always like the MDX parser:

• If you come from an existing Markdown docs base (like v1), you need to make it compatible with MDX, despite that you actually don't plan to embed any JSX components in the markdown
• You might want to keep compatibility to CommonMark, to stay compatible with existing ecosystem (Github md viewer, markdownlint etc...)
• It creates more "lock-in", because to leave MDX you have to convert back to CommonMark
• It can be confusing to not be able to use CommonMark (ie html tags, not jsx) in .md files, and to learn that even .md files are parsed with MDX

Related discussions:

• How to use plain Markdown (CommonMark) in Docusaurus v2? #3009
• Better compatibility with CommonMark mdx-js/mdx#1125

---

Solution ?

• https://github.com/rexxars/react-markdown
• https://github.com/probablyup/markdown-to-jsx

These libs:

• is also based on UnifiedJS ecosystem
• allows to pass custom React elements to replace existing tags

We may be able to build some shared abstraction on top of react-markdown + MDX.

If this works, we could switch from one parser to another with a simple switch/setting, that could be:

• .md -> common-mark compatible parser
• .mdx -> MDX
• global default D2 parser setting
• parser frontmatter

The idea would be that, if a doc does not embed any html/jsx, we could switch from one parser to the other, and shouldn't notice any change.

--

Feedbacks welcome

[slorber]

Sidenote: <!--truncate--> marker used for blog summaries will likely not work in MDX 2:

• https://v2.docusaurus.io/docs/blog/#summary-truncation
• Rewrite how MDX is parsed mdx-js/mdx#1039

Edit: I think it will still work because it's processed before mdx compilation

[borekb]

That sounds excellent!

Thinking about the common abstraction you mentioned, and assuming that this is still the overarching goal:

Beyond that, Docusaurus 2 is a performant static site generator and can be used to create common content-driven websites (e.g. Documentation, Blogs, Product Landing and Marketing Pages, etc) extremely quickly.

I think that Docusaurus could document an interface for plugins / formats / loaders (I don't know how to call them) that could possibly look like this:

• At the base level, the format should be able to produce an HTML output, i.e., an HTML string. For example, if I have a .txt file, I'd be able to write a "format" that produces <pre>... contents of the txt file ...</pre>. Since this is just a string, Docusaurus wouldn't operate on it in any way, just display it.

• Smarter formats would return some sort of AST (or JSX or whatever would be suitable). For example, if I wanted to implement .md that turns code blocks to live playgrounds, like React Styleguidist does, I'd be able to do that.

Some wilders use cases this would cover (I actually had them in the past):

• A marketing team uses headless WordPress to maintain the contents of landing pages.
• Feature comparison / grid is maintained in a Google Sheet.
• .md files use some sort of Markdown dialect, for example, the site used to be powered by MkDocs and uses Python-Markdown plus a couple of custom extensions.

For this RFC, I think it's more than enough to support CommonMark but since I've now spent some time thinking about how we'd use Docusaurus and what I'd love it to allow me to do, I thought I'd post it here.

Thanks a lot for this RFC and all the work that goes into Docusaurus!

[borekb]

@slorber I'd like to create a prototype of CommonMark support but am unfamiliar with Docusaurus codebase so would really appreciate high-level guidelines if you will.

Roughly speaking, if I wanted to parse .md files as CommonMark, which parts of the codebase I'd need to touch? I can overwrite the code for now in a fork, i.e., it's not my ambition yet to make this a general solution supporting both MDX and CommonMark, I just want to see what's the minimal set of changes to swap the MDX parser for something like remark.

Any hints appreciated 🙏 .

[slorber]

Hi,

My first intuition would be to modify "docusaurus-mdx-loader", and provide a loader option to tell it to load the files as md or mdx. In the end, we need a React component anyway, using MDX, but in md mode, we could convert the html elements to JSX elements just before feeding mdx, so that mdx is happy?

Not sure, this would require some experiments to see if this is possible

[borekb]

Thanks a lot, I'll give it a go later this week or the next one.

[borekb]

plugin-content-docs that supports CommonMark for .md files

We've experimented with plain Markdown support in an internal prototype and wanted to post the key results here.

Summary

It's doable and not that complex – about 200 LoC. There's currently some ugliness like to get the ToC, we're converting AST to React components and then parsing it back to a string for which we didn't find a better solution yet but I'm sure there should be, e.g. something like hast-util-to-jsx if it was maintained.

How it's done

The rewritten plugin-content-docs-2/index.ts customizes loaders:

• For .mdx files, use @docusaurus/mdx-loader
• For .md files, use a custom loader (see below).

In our prototype, we first duplicate ~20 LoC from the base implementation and then customize the loaders. The entire file (certainly with opportunities for further cleanup) looks like this:

import path from 'path';

import admonitions from 'remark-admonitions';
import {STATIC_DIR_NAME} from '@docusaurus/core/lib/constants';
import {
  docuHash,
  aliasedSitePath,
} from '@docusaurus/utils';
import {
  LoadContext,
  Plugin,
  OptionValidationContext,
  ValidationResult,
} from '@docusaurus/types';

import loadEnv from '@docusaurus/plugin-content-docs/lib/env';

import {
  PluginOptions,
  LoadedContent,
  SourceToPermalink,
} from '@docusaurus/plugin-content-docs/lib/types';
import {Configuration} from 'webpack';
import {VERSIONS_JSON_FILE} from '@docusaurus/plugin-content-docs/lib/constants';
import {PluginOptionSchema} from '@docusaurus/plugin-content-docs/lib/pluginOptionSchema';
import {ValidationError} from '@hapi/joi';

import * as originalPluginContentDocs from '@docusaurus/plugin-content-docs';

export default function pluginContentDocs(
  context: LoadContext,
  options: PluginOptions,
): Plugin<LoadedContent | null, typeof PluginOptionSchema> {

  if (options.admonitions) {
    options.remarkPlugins = options.remarkPlugins.concat([
      [admonitions, options.admonitions],
    ]);
  }

  const {siteDir, generatedFilesDir} = context;
  const docsDir = path.resolve(siteDir, options.path);
  const sourceToPermalink: SourceToPermalink = {};

  const dataDir = path.join(
    generatedFilesDir,
    'docusaurus-plugin-content-docs',
    // options.id ?? 'default', // TODO support multi-instance
  );

  // Versioning.
  const env = loadEnv(siteDir, {disableVersioning: options.disableVersioning});
  const {versioning} = env;
  const {
    docsDir: versionedDir,
  } = versioning;

  const result = originalPluginContentDocs.default(context, options);
  result.configureWebpack = function (_config, isServer, utils) {
    const {getBabelLoader, getCacheLoader} = utils;
    const {rehypePlugins, remarkPlugins} = options;
    // Suppress warnings about non-existing of versions file.
    const stats = {
      warningsFilter: [VERSIONS_JSON_FILE],
    };

    return {
      stats,
      devServer: {
        stats,
      },
      resolve: {
        alias: {
          '~docs': dataDir,
        },
      },
      module: {
        rules: [
          {
            test: /(\.mdx)$/,
            include: [docsDir, versionedDir].filter(Boolean),
            use: [
              getCacheLoader(isServer),
              getBabelLoader(isServer),
              {
                loader: require.resolve('@docusaurus/mdx-loader'),
                options: {
                  remarkPlugins,
                  rehypePlugins,
                  staticDir: path.join(siteDir, STATIC_DIR_NAME),
                  metadataPath: (mdxPath: string) => {
                    // Note that metadataPath must be the same/in-sync as
                    // the path from createData for each MDX.
                    const aliasedSource = aliasedSitePath(mdxPath, siteDir);
                    return path.join(
                      dataDir,
                      `${docuHash(aliasedSource)}.json`,
                    );
                  },
                },
              },
              {
                loader: path.resolve(__dirname, './markdown/index.js'),
                options: {
                  siteDir,
                  docsDir,
                  sourceToPermalink,
                  versionedDir,
                },
              },
            ].filter(Boolean),
          },
          {
            test: /(\.md)$/,
            include: [docsDir, versionedDir].filter(Boolean),
            use: [
              getCacheLoader(isServer),
              getBabelLoader(isServer),
              {
                loader: path.resolve(__dirname, './custom-md-loader/index.js'),
                options: {
                  remarkPlugins,
                  rehypePlugins,
                  staticDir: path.join(siteDir, STATIC_DIR_NAME),
                  metadataPath: (mdxPath: string) => {
                    // Note that metadataPath must be the same/in-sync as
                    // the path from createData for each MDX.
                    const aliasedSource = aliasedSitePath(mdxPath, siteDir);
                    return path.join(
                      dataDir,
                      `${docuHash(aliasedSource)}.json`,
                    );
                  },
                },
              },
              {
                loader: path.resolve(__dirname, './markdown/index.js'),
                options: {
                  siteDir,
                  docsDir,
                  sourceToPermalink,
                  versionedDir,
                },
              },
            ].filter(Boolean),
          },
        ],
      },
    } as Configuration;
  }

  return result;
}

export function validateOptions({
  validate,
  options,
}: OptionValidationContext<PluginOptions, ValidationError>): ValidationResult<
  PluginOptions,
  ValidationError
> {
  return originalPluginContentDocs.validateOptions({validate, options});
}

The there's a custom loader – plugin-content-docs-2/src/custom-md-loader/index.ts. It looks like this in full:

import {loader} from 'webpack';
import {getOptions} from 'loader-utils';
import {readFileSync} from 'fs-extra';
import matter from 'gray-matter';
import stringifyObject from 'stringify-object';
import unified from 'unified';
import parse from 'remark-parse';
import remark2rehype from 'remark-rehype';
import rehype2react from 'rehype-react';
import React from 'react';
import rightToc from '@docusaurus/mdx-loader/src/remark/rightToc';
import slug from 'remark-slug';
import raw from 'rehype-raw';
import emoji from 'remark-emoji';
import admonitions from 'remark-admonitions';
import headings from 'rehype-autolink-headings';
import highlight from '@mapbox/rehype-prism';
import reactElementToJSXString from 'react-element-to-jsx-string';

const mdLoader: loader.Loader = function (fileString) {
  const callback = this.async();

  const {data, content} = matter(fileString);

  const options = getOptions(this) || {};

  let exportStr = `export const frontMatter = ${stringifyObject(data)};`;
  // Read metadata for this MDX and export it.
  if (options.metadataPath && typeof options.metadataPath === 'function') {
    const metadataPath = options.metadataPath(this.resourcePath);
    if (metadataPath) {
      // Add as dependency of this loader result so that we can
      // recompile if metadata is changed.
      this.addDependency(metadataPath);
      const metadata = readFileSync(metadataPath, 'utf8');
      exportStr += `\nexport const metadata = ${metadata};`;
    }
  }

  const processedMd = unified()
    .use(parse, {commonmark: true})
    .use(slug)
    .use(emoji)
    .use(admonitions)
    .use(rightToc)
    .use(remark2rehype, {allowDangerousHtml: true})
    .use(raw)
    .use(headings)
    .use(highlight)
    .use(rehype2react, {createElement: React.createElement, Fragment: React.Fragment})
    .processSync(content);

  const jsxString = reactElementToJSXString((processedMd as any).result);

  // I don't like this at all, but it's a prototype...
  // We need to get 'rightToc' data from the JSX string, so following lines
  // are about getting the info and then replacing it, along with escaping unwanted chars.
  const rightTocString = jsxString
    .match(/(export const rightToc = \[[\s\S.]*\];)/)![1]
    .replace(/(\\n)|(\\t)|(\\)/g, '');

  const escapedJsxString = jsxString
    .replace(/{\`[\S\s.]*?export const rightToc = \[[\s\S.]*\];[\S\s.]*?\`}/, '')
    .replace(/{'[\s\S]*?'}/g, `{' '}`)
    .replace(/`/g, '\`');

  const code = `
  import React from 'react';

  ${rightTocString}
  ${exportStr}

  export default function MDLoader() {
    return (${escapedJsxString});
  }
  `;

  return callback && callback(null, code);
};

export default mdLoader;

If there wasn't the ugly React to string parsing code, it would actually be quite simple.

The downside from the maintenance point of view is that the MD loader is explicit about its unified.js plugins while the MDX loader is a bit more indirect / obscure, so there would be two places to maintain this configuration. But I think this could be refactored to be more aligned, and even in the worst case, it's like 15 lines of code and the default set of plugins probably isn't changing that often.

Overall, it seems feasible to me.

[borekb]

An alternative approach would be to convert MD to MDX first and then just let the mdx-loader to its thing. But there probably isn't currently a convertor from MD to MDX in the unified ecosystem, though many pieces are in place: unifiedjs/ideas#9.

[slorber]

thanks for those details, that looks interesting. If MDX provided a converter that would be great, also would helpful for v1->v2 migrations

I don't have much time to explore these ideas but we'll come back to it someday.

Note, not sure it's related, but there's a large docs plugin refactor here: #3245