namespace packages broken because pkgutil shims not generated by default

[philsc]

🐞 bug report

Affected Rule

I'm assuming there's some issue in py_binary or py_library. Not sure.

Is this a regression?

Yes, 1.4.1 appears to work, but 1.5.0-rc4 does not.

Description

I can no longer import the kfp.compiler module.

🔬 Minimal Reproduction

Check out the reproduction here: https://github.com/philsc/rules_python-repro/tree/repro-kfp-1.5.0-rc4

The README showcases the issue I'm running into.

🔥 Exception or Error

Doing this:

import kfp
print(kfp)

import kfp.compiler
print(kfp.compiler)

And having this BUILD target:

py_binary(
    name = "bin",
    srcs = ["bin.py"],
    deps = [
        requirement("kfp"),
    ],
)

results in this error:

<module 'kfp' from '/data/phil_cache/bazel/_bazel_philipp.schrader/29d62092ea113224b51636420449a2d3/execroot/__main__/bazel-out/k8-fastbuild/bin/bin.runfiles/my_deps_kfp_pipeline_spec/site-packages/kfp/__init__.py'>
Traceback (most recent call last):
  File "/data/phil_cache/bazel/_bazel_philipp.schrader/29d62092ea113224b51636420449a2d3/execroot/__main__/bazel-out/k8-fastbuild/bin/bin.runfiles/__main__/bin.py", line 4, in <module>
    import kfp.compiler
ModuleNotFoundError: No module named 'kfp.compiler'

Rules_python version:

1.5.0

Workaround

Set pip.parse.enable_implicit_namespace_pkgs = True in 1.5

Note that this will have to be changed to False in subsequent versions; the bug is an inverted logic check after some refactoring.

[philsc]

Actually, setting enable_implicit_namespace_pkgs = True, fixes the issue in the reproduction that I pushed, but not for our internal repo. I will take a closer look.

[philsc]

I think this is an issue in one of our internal packages making some invalid assumptions around imports.

Sorry for the noise!

[michael-quinlan]

I'm not sure this is unique to your setup, after upgrading to 1.5 i'm getting

ImportError: cannot import name 'genai' from 'google' (/home/shared/.cache/bazel/_bazel_shared/d87484021b443b9cd35511f6aa54a7de/execroot/_main/bazel-out/k8-fastbuild/bin/tool_backend/tool_backend.runfiles/rules_python++pip+pip_311_google_auth/site-packages/google/init.py)

which is very similar in that the two google namespaces are not resolving. 1.4.1 worked fine.

enable_implicit_namespace_pkgs = True seems to fix it on 1.5

[aaronmaxlevy]

@philsc would you be willing to re-open this issue?

[rickeylev]

fly-by guess: Where and how the pkgutil-style namespace package shims were generated changed from being done in the repo phase to the build phase. #2882

[rickeylev]

I poked this a bit

• kfp is a distribution, a wheel
• kfp is a namespace package.
• kfp's wheel is well formed: kfp/__init__.py is in the wheel and uses pkgutil-style extend_path()
• kfp-pipeline-spec is a distribution, a wheel
• kfp-pipeline-spec is a well formed. It does not contain kfp/__init__.py
• kfp-pipeline-spec adds the kfp.pipeline_spec package to the kfp namespace.

What's happening is incompatible_default_to_explicit_init_py=false (default) is mucking things up by creating empty init.py files where there shouldn't be any init.py files. What happens is the empty init.py file turns the package into a regular package, not namespace package, and then the search for other locations doesn't work. Setting enable_implicit_namespace_pkg=True fixes it because it generates an init.py with the pkgutil extend_path code.

But then...why did it ever work in 1.4...?

Ah, I think we accidentally changed the default. In #2882:

# python/private/pypi/whl_installer/wheel_installer.py
if not enable_implicit_namespace_pkgs:
    _setup_namespace_pkg_compatibility(installation_dir)

# meanwhile, new logic
# python/private/pypi/whl_library_targets.bzl
if enable_implicit_namespace_pkgs:
    <create pkgutil init shims>

Should be a simple fix we can backport and release as 1.5.1

We can also directly test these types of wheels with some new test helpers we have.