Error when importing Cython modules

[rth]

I'm trying to fix mypy issues in scikit-learn (scikit-learn/scikit-learn#16726) and have a question/observation about C and Cython modules.

Say package has a _lib.pyx Cython extension, which defines a function _lib_func. The following passes mypy without errors,

from package._lib import _lib_func

however,

from package import _lib

produces the following error,

error: Module 'package' has no attribute '_lib'

Are there any workaround for importing the complete C extension module and not only functions/classes it contains (short of using type: ignore at the import line)?

Example

Sorry I don't have a minimal reproducible example, I can make one if necessary. An actual example I run into,

• sklearn/manifold/_barnes_hut_tsne.pyx Cython extension module.
• built as a C extension here
• that produces mypy error in import sklearn/manifold/tests/test_t_sne.py using,

  from sklearn.manifold import _barnes_hut_tsne

  with,

  error: Module 'sklearn.manifold' has no attribute '_barnes_hut_tsne'
  

  while importing individual functions from this C extension module works fine. And of course running this Python code also works fine.

Versions

• cpython: 3.8.0
• mypy : 0.770
• cython: 0.29.14

[emmatyping]

Hi! Mypy doesn't actually import or interact with C extension code. It can only parse and gain type information from .py and .pyi files. The error

error: Module 'sklearn.manifold' has no attribute '_barnes_hut_tsne'

is because mypy cannot find neither _barnes_hut_tsne.py nor a function or class etc. named that in the sklearn.manifold __init__.py[i]. I do think this could be reworded to mention a missing module as a possibility.

The following should not pass typecheck if _lib is a C extension, do you have an example of that case?

from package._lib import _lib_func

error: Module 'sklearn.manifold' has no attribute '_barnes_hut_tsne'

while importing individual functions from this C extension module works fine.

When I try mypy -c "from sklearn.manifold._barnes_hut_tsne import compute_gradient mypy does not find the module, as expected.

[rth]

Thanks for the explanations @ethanhs!

When I try mypy -c "from sklearn.manifold._barnes_hut_tsne import compute_gradient mypy does not find the module, as expected.

Yes, you are right. One important information that I forgot to mention is that I was running mypy with the --ignore-missing-import option. In which case one of those produces and error but not the other,

$ mypy --ignore-missing-import -c "from sklearn.manifold import _barnes_hut_tsne"
<string>:1: error: Module 'sklearn.manifold' has no attribute '_barnes_hut_tsne'
Found 1 error in 1 file (checked 1 source file)

$ mypy --ignore-missing-import -c "from sklearn.manifold._barnes_hut_tsne import gradient"
Success: no issues found in 1 source file

without the flag, they both error indeed,

$ mypy -c "from sklearn.manifold._barnes_hut_tsne import gradient"
<string>:1: error: Skipping analyzing 'sklearn.manifold._barnes_hut_tsne': found module but no type hints or library stubs
<string>:1: note: See https://mypy.readthedocs.io/en/latest/running_mypy.html#missing-imports
Found 1 error in 1 file (checked 1 source file)

So it looks like it's expected behavior. Closing. Thanks again!