Support the usage of Markdown-based notebooks with "Lite" directives (#221)

* Add Jupytext as a dependency

* Add ability to convert MyST-MD notebooks on-the-fly

* Document ability to use MyST notebooks

* Add a sample MyST notebook

* Showcase `.md` example for NotebookLite, Voici

* Add `jupytext` for RTD builds

* Drop UUID suffix, cache based on timestamps

* Clarify MyST notebook in notebook contents

* Handle notebook collision

* Refactor stripping logic + strip MyST notebooks

* Add docs on how to strip from Markdown notebooks

* Avoid conflict: rename Markdown notebook

* Handle collisions among included files in docs build

Author: agriyakhetarpal

dev-environment.yml

@@ -6,6 +6,7 @@ dependencies:
     - jupyter_server
     - jupyterlab_server
     - jupyterlite-core >=0.3,<0.4
+    - jupytext
     - pydata-sphinx-theme
     - micromamba
     - myst-parser

docs/configuration.md

@@ -96,7 +96,7 @@ This behaviour can be enabled by setting the following config:
 strip_tagged_cells = True
 ```
 
-and then by tagging the cells you want to strip with the tag `jupyterlite_sphinx_strip` in the JSON metadata
+and then by tagging the cells you want to strip with the `jupyterlite_sphinx_strip` tag in the JSON metadata
 of the cell, like this:
 
 ```json
@@ -109,7 +109,7 @@ of the cell, like this:
 
 This is useful when you want to remove some cells from the rendered notebook in the JupyterLite
 console, for example, cells that are used for adding reST-based directives or other
-Sphinx-specific content.
+Sphinx-specific content. It can be used to remove either code cells or Markdown cells.
 
 For example, you can use this feature to remove the `toctree` directive from the rendered notebook
 in the JupyterLite console:
@@ -147,9 +147,43 @@ in the JupyterLite console:
 where the cell with the `toctree` directive will be removed from the rendered notebook in
 the JupyterLite console.
 
+In the case of a MyST notebook, you can use the following syntax to tag the cells:
+
+````markdown
+
++++ {"tags": ["jupyterlite_sphinx_strip"]}
+
+# Heading 1
+
+This is a Markdown cell that will be stripped from the rendered notebook in the
+JupyterLite console.
+
++++
+
+```{code-cell} ipython3
+:tags: [jupyterlite_sphinx_strip]
+
+# This is a code cell that will be stripped from the rendered notebook in the
+# JupyterLite console.
+def foo():
+    print(3)
+```
+
+```{code-cell} ipython3
+# This cell will not be stripped
+def bar():
+    print(4)
+```
+````
+
+The Markdown cells are not wrapped, and hence the `+++` and `+++` markers are used to
+indicate where the cells start and end. For more details around writing and customising
+MyST-flavoured notebooks, please refer to the
+[MyST Markdown overview](https://jupyterbook.org/en/stable/content/myst.html).
+
 Note that this feature is only available for the `NotebookLite`, `JupyterLite`, and the
-`Voici` directives and works with the `.ipynb` files passed to them. It is not implemented
-for the `TryExamples` directive.
+`Voici` directives and works with the `.md` (MyST) or `.ipynb` files passed to them. It
+is not implemented for the `TryExamples` directive.
 
 ## Disable the `.ipynb` docs source binding
 

docs/directives/jupyterlite.md

@@ -36,6 +36,24 @@ You can also pass a Notebook file to open automatically:
    :prompt_color: #00aa42
 ```
 
+The notebook can also be a MyST-flavoured Markdown file that will be converted to a Jupyter Notebook before being opened.
+
+```rst
+.. jupyterlite:: my_markdown_notebook.md
+   :width: 100%
+   :height: 600px
+   :prompt: Try JupyterLite!
+   :prompt_color: #00aa42
+```
+
+```{eval-rst}
+.. jupyterlite:: my_markdown_notebook.md
+   :width: 100%
+   :height: 600px
+   :prompt: Try JupyterLite!
+   :prompt_color: #00aa42
+```
+
 If you use the `:new_tab:` option in the directive, the Notebook will be opened in a new browser tab.
 The tab will render the full-fledged Lab interface, which is more complete and showcases all features
 of JupyterLite.

docs/directives/my_markdown_notebook.md

@@ -0,0 +1,23 @@
+---
+jupytext:
+  text_representation:
+    extension: .md
+    format_name: myst
+    format_version: 0.13
+    jupytext_version: 1.16.4
+kernelspec:
+  display_name: Python 3 (ipykernel)
+  language: python
+  name: python3
+---
+
+# This is a MyST Markdown-flavoured notebook
+
+```{code-cell} ipython3
+def foo():
+    print(3)
+```
+
+```{code-cell} ipython3
+foo()
+```

docs/directives/notebooklite.md

@@ -16,7 +16,9 @@
    :prompt: Try classic Notebook!
 ```
 
-You can also pass a Notebook file to open:
+You can provide a notebook (either Jupyter-based or MyST-Markdown flavoured) to open:
+
+1. Jupyter Notebook
 
 ```rst
 .. notebooklite:: my_notebook.ipynb
@@ -32,6 +34,22 @@ You can also pass a Notebook file to open:
    :prompt: Try classic Notebook!
 ```
 
+2. MyST Markdown
+
+```rst
+.. notebooklite:: my_markdown_notebook.md
+   :width: 100%
+   :height: 600px
+   :prompt: Try classic Notebook!
+```
+
+```{eval-rst}
+.. notebooklite:: my_markdown_notebook.md
+   :width: 100%
+   :height: 600px
+   :prompt: Try classic Notebook!
+```
+
 If you use the `:new_tab:` option in the directive, the Notebook will be opened in a new browser tab.
 The tab will render the classic Notebook UI, which is more minimal and does not showcase the entire
 Lab interface.

docs/directives/voici.md

@@ -12,7 +12,10 @@
    :height: 600px
 ```
 
-You can provide a notebook that will be rendered with Voici:
+You can provide a notebook file (either Jupyter-based or MyST-Markdown flavoured) that will be
+rendered with Voici:
+
+1. Jupyter Notebook
 
 ```rst
 .. voici:: my_notebook.ipynb
@@ -28,6 +31,22 @@ You can provide a notebook that will be rendered with Voici:
    :prompt_color: #dc3545
 ```
 
+2. MyST Markdown
+
+```rst
+.. voici:: my_markdown_notebook.md
+   :height: 600px
+   :prompt: Try Voici!
+   :prompt_color: `#dc3545`
+```
+
+```{eval-rst}
+.. voici:: my_markdown_notebook.md
+   :height: 600px
+   :prompt: Try Voici!
+   :prompt_color: `#dc3545`
+```
+
 If you use the `:new_tab:` option in the directive, the Voici dashboard will execute and render
 the notebook in a new browser tab, instead of in the current page.
 

jupyterlite_sphinx/jupyterlite_sphinx.py

@@ -24,6 +24,7 @@
 
 from ._try_examples import examples_to_notebook, insert_try_examples_directive
 
+import jupytext
 import nbformat
 
 try:
@@ -478,6 +479,72 @@ class _LiteDirective(SphinxDirective):
         "new_tab_button_text": directives.unchanged,
     }
 
+    def _target_is_stale(self, source_path: Path, target_path: Path) -> bool:
+        # Used as a heuristic to determine if a markdown notebook needs to be
+        # converted or reconverted to ipynb.
+        if not target_path.exists():
+            return True
+
+        return source_path.stat().st_mtime > target_path.stat().st_mtime
+
+    # TODO: Jupytext support many more formats for conversion, but we only
+    # consider Markdown and IPyNB for now. If we add more formats someday,
+    # we should also consider them here.
+    def _assert_no_conflicting_nb_names(
+        self, source_path: Path, notebooks_dir: Path
+    ) -> None:
+        """Check for duplicate notebook names in the documentation sources.
+        Raises if any notebooks would conflict when converted to IPyNB."""
+        target_stem = source_path.stem
+        target_ipynb = f"{target_stem}.ipynb"
+
+        # Only look for conflicts in source directories and among referenced notebooks.
+        # We do this to prevent conflicts with other files, say, in the "_contents/"
+        # directory as a result of a previous failed/interrupted build.
+        if source_path.parent != notebooks_dir:
+
+            # We only consider conflicts if notebooks are actually referenced in
+            # a directive, to prevent false posiitves from being raised.
+            if hasattr(self.env, "jupyterlite_notebooks"):
+                for existing_nb in self.env.jupyterlite_notebooks:
+                    existing_path = Path(existing_nb)
+                    if (
+                        existing_path.stem == target_stem
+                        and existing_path != source_path
+                    ):
+
+                        raise RuntimeError(
+                            "All notebooks marked for inclusion with JupyterLite must have a "
+                            f"unique file basename. Found conflict between {source_path} and {existing_path}."
+                        )
+
+        return target_ipynb
+
+    def _strip_notebook_cells(
+        self, nb: nbformat.NotebookNode
+    ) -> List[nbformat.NotebookNode]:
+        """Strip cells based on the presence of the "jupyterlite_sphinx_strip" tag
+        in the metadata. The content meant to be stripped must be inside its own cell
+        cell so that the cell itself gets removed from the notebooks. This is so that
+        we don't end up removing useful data or directives that are not meant to be
+        removed.
+
+        Parameters
+        ----------
+        nb : nbformat.NotebookNode
+            The notebook object to be stripped.
+
+        Returns
+        -------
+        List[nbformat.NotebookNode]
+            A list of cells that are not meant to be stripped.
+        """
+        return [
+            cell
+            for cell in nb.cells
+            if "jupyterlite_sphinx_strip" not in cell.metadata.get("tags", [])
+        ]
+
     def run(self):
         width = self.options.pop("width", "100%")
         height = self.options.pop("height", "1000px")
@@ -498,43 +565,59 @@ def run(self):
         )
 
         if self.arguments:
+            # Keep track of the notebooks we are going through, so that we don't
+            # operate on notebooks that are not meant to be included in the built
+            # docs, i.e., those that have not been referenced in the docs via our
+            # directives anywhere.
+            if not hasattr(self.env, "jupyterlite_notebooks"):
+                self.env.jupyterlite_notebooks = set()
+
             # As with other directives like literalinclude, an absolute path is
             # assumed to be relative to the document root, and a relative path
             # is assumed to be relative to the source file
             rel_filename, notebook = self.env.relfn2path(self.arguments[0])
             self.env.note_dependency(rel_filename)
 
-            notebook_name = os.path.basename(notebook)
+            notebook_path = Path(notebook)
+
+            self.env.jupyterlite_notebooks.add(str(notebook_path))
 
-            notebooks_dir = Path(self.env.app.srcdir) / CONTENT_DIR / notebook_name
+            notebooks_dir = Path(self.env.app.srcdir) / CONTENT_DIR
+            os.makedirs(notebooks_dir, exist_ok=True)
+
+            self._assert_no_conflicting_nb_names(notebook_path, notebooks_dir)
+            target_name = f"{notebook_path.stem}.ipynb"
+            target_path = notebooks_dir / target_name
 
             notebook_is_stripped: bool = self.env.config.strip_tagged_cells
 
-            # Create a folder to copy the notebooks to and for NotebookLite to find
-            os.makedirs(os.path.dirname(notebooks_dir), exist_ok=True)
-
-            if notebook_is_stripped:
-                # Note: the directives meant to be stripped must be inside their own
-                # cell so that the cell itself gets removed from the notebook. This
-                # is so that we don't end up removing useful data or directives that
-                # are not meant to be removed.
-
-                nb = nbformat.read(notebook, as_version=4)
-                nb.cells = [
-                    cell
-                    for cell in nb.cells
-                    if "jupyterlite_sphinx_strip" not in cell.metadata.get("tags", [])
-                ]
-                nbformat.write(nb, notebooks_dir, version=4)
-
-            # If notebook_is_stripped is False, then copy the notebook(s) to notebooks_dir.
-            # If it is True, then they have already been copied to notebooks_dir by the
-            # nbformat.write() function above.
+            if notebook_path.suffix.lower() == ".md":
+                if self._target_is_stale(notebook_path, target_path):
+                    nb = jupytext.read(str(notebook_path))
+                    if notebook_is_stripped:
+                        nb.cells = self._strip_notebook_cells(nb)
+                    with open(target_path, "w", encoding="utf-8") as f:
+                        nbformat.write(nb, f, version=4)
+
+                notebook = str(target_path)
+                notebook_name = target_name
             else:
-                try:
-                    shutil.copy(notebook, notebooks_dir)
-                except shutil.SameFileError:
-                    pass
+                notebook_name = notebook_path.name
+                target_path = notebooks_dir / notebook_name
+
+                if notebook_is_stripped:
+                    nb = nbformat.read(notebook, as_version=4)
+                    nb.cells = self._strip_notebook_cells(nb)
+                    nbformat.write(nb, target_path, version=4)
+                # If notebook_is_stripped is False, then copy the notebook(s) to notebooks_dir.
+                # If it is True, then they have already been copied to notebooks_dir by the
+                # nbformat.write() function above.
+                else:
+                    try:
+                        shutil.copy(notebook, target_path)
+                    except shutil.SameFileError:
+                        pass
+
         else:
             notebook_name = None
 

pyproject.toml

@@ -30,6 +30,7 @@ docs = [
     "myst_parser",
     "pydata-sphinx-theme",
     "jupyterlite-xeus>=0.1.8,<0.3.0",
+    "jupytext",
 ]
 
 [tool.hatch.version]