What to do about Jinja2 dependency & supporting old sphinx versions

[rossbar]

I just wanted to float this out there given that there have been several issues (#376 #373) in addition to many more from libraries that use numpydoc (e.g. scipy/scipy#15895, many more linked examples in the parent sphinx issue: sphinx-doc/sphinx#10291).

The latest minor release of Jinja2 (3.1) is incompatible with all sphinx versions earlier than 4.0.2 (see sphinx-doc/sphinx#10291). Since numpydoc currently supports sphinx versions all the way back to 1.8, the Jinja2 incompatibility has been a bit disruptive. #369 added a pin for Jinja2, but this is inconvenient for projects that use numpydoc but have the latest sphinx installed and want to use the latest jinja2 as well. Essentially the (Jinja2<=3.1) pin is only in numpydoc because of the commitment to support sphinx < 4.0.2.

I wanted to collect opinions on what could be done here. Are there strong motivations for supporting sphinx<4 and is there a timeline for when numpydoc drops support for older sphinx versions? FWIW the sphinx devs have stated they are not patching sphinx 3. If numpydoc is going to keep supporting older sphinx versions, then I think an alternative to pinning Jinja2 should be investigated e.g. refactoring or vendoring the problematic code.

[mattip]

Refactoring/vendoring sounds like a nice way to work around this. Any idea how hard it would be?

[rossbar]

Any idea how hard it would be?

Unfortunately it doesn't seem that straightforward. I tried vendoring only the sphinx.jinja2glue module (essentially copying it into numpydoc/numpydoc/_sphinx_compat_jinja2glue and importing BuiltinTemplateLoader from the vendored module. However this is not sufficient to get the test suite working with Sphinx 3.5.4 as there are other instances of the deprecated (now removed) Jinja2 functions throughout Sphinx. It really doesn't seem like vendoring is an option. Writing a custom replacement for BuiltinTemplateLoader also doesn't seem very straightforward as the original inherits from both Sphinx and Jinja2 classes.

[mattip]

Maybe there is nothing to be done from numpydoc's side.

What if instead of pinning we detect the use of a bad combination of versions and emit a more helpful warning/error: "If you wish to continue using sphinx<4.0.2 you need to pin Jinja2<3.1". It seems there are a lot of projects broken by this change.

[rgommers]

What if instead of pinning we detect the use of a bad combination of versions and emit a more helpful warning/error: "If you wish to continue using sphinx<4.0.2 you need to pin Jinja2<3.1".

This seems reasonable - but make it an error rather than a warning, given that it doesn't actually work and Sphinx warnings are too easy to miss.

Users who need older Sphinx can always pin to an older numpydoc version.

[rossbar]

Maybe there is nothing to be done from numpydoc's side.

I tend to agree here - realistically any dependency pinning should be in Sphinx, but they've already stated that they won't backport to unsupported sphinx versions (i.e. <4). This is the part that makes me wonder - why is numpydoc committed to supporting unsupported versions of sphinx?

It seems there are a lot of projects broken by this change.

Indeed - so much so that it will be evermore difficult for projects to rely on sphinx <4. Even testing the vendored solution was difficult because it turns out sphinx 3.5.4 (the latest in the pre-4 series) doesn't work with Python 3.10.

I guess ultimately I'm interested in a more general discussion about the support commitments for numpydoc; I'll go ahead and raise that on another forum.

Re: how best to handle the current situation...

What if instead of pinning we detect the use of a bad combination of versions and emit a more helpful warning/error

I'll give this a try 👍

[jarrodmillman]

See https://pypi.org/project/numpydoc/1.3/