System image compilation and --compiled-modules=no is broken in Python >= 3.9

[tkf]

In Python >= 3.9, Julia(compiled_modules=False) fails with

JuliaError: Exception 'LoadError' occurred while calling julia code:
const PyCall = Base.require(Base.PkgId(Base.UUID("438e738f-606a-5dbb-bf0a-cddfbfd45ab0"), "PyCall"))

With Julia(debug=True, compiled_modules=False), it shows

┌ Error: Failed to import PyCall
│   exception =
│    LoadError: could not load symbol "main":
│    /.../bin/python: undefined symbol: main
│    Stacktrace:
│      [1] #dlsym#1
│        @ ./libdl.jl:59 [inlined]
│      [2] dlsym(hnd::Ptr{Nothing}, s::Symbol)
│        @ Base.Libc.Libdl ./libdl.jl:57
│      [3] top-level scope
│        @ ~/.julia/packages/PyCall/ScKXc/src/startup.jl:76
│      [4] include(mod::Module, _path::String)
│        @ Base ./Base.jl:420
│      [5] include(x::String)
│        @ PyCall ~/.julia/packages/PyCall/ScKXc/src/PyCall.jl:1
│      [6] top-level scope
│        @ ~/.julia/packages/PyCall/ScKXc/src/PyCall.jl:38
│      [7] include
│        @ ./Base.jl:420 [inlined]
│      [8] _require(pkg::Base.PkgId)
│        @ Base ./loading.jl:1149
│      [9] require(uuidkey::Base.PkgId)
│        @ Base ./loading.jl:1013
│     [10] top-level scope
│        @ none:3
│    in expression starting at /home/runner/.julia/packages/PyCall/ScKXc/src/startup.jl:41
│    in expression starting at /home/runner/.julia/packages/PyCall/ScKXc/src/PyCall.jl:1
└ @ Main none:5

Note that lines 41 to 76 as mentioned in the stack trace have this code:

    41  if !symbols_present
    42      # Python not present. Use deps.jl
...
    58  else
    59      @static if Sys.iswindows()
...
    72      else
    73          libpy_handle = proc_handle
    74          # Now determine the name of the python library that these symbols are from
    75          some_address_in_libpython = Libdl.dlsym(libpy_handle, :Py_GetVersion)
    76          some_address_in_main_exe = Libdl.dlsym(proc_handle, Sys.isapple() ? :_mh_execute_header : :main)

--- https://github.com/JuliaPy/PyCall.jl/blob/4e5c12cfc597a0e911d42fe661aabd4f1cd74a5f/src/startup.jl#L41-L76

Since the system image compilation script uses --compiled-modules=no for Julia >= 1.5, this also breaks python3 -m julia.sysimage sys.so.

For full traces of this failure mode in CI, see: #458 (comment)

[tkf]

Hmm... It does seem that Python < 3.9 and >= 3.9 are different.

In Python 3.8, main exists in the main process among other symbols:

Python 3.8.12 (default, Oct 24 2021, 03:28:30)
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes
>>> proc = ctypes.CDLL(None)
>>> proc.main
<_FuncPtr object at 0x7fd729264700>
>>> proc._start
<_FuncPtr object at 0x7fd729264880>
>>> proc.Py_Main
<_FuncPtr object at 0x7fd729264940>

However, it is missing in Python 3.9:

Python 3.9.7 (default, Oct 24 2021, 00:30:21)
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes
>>> proc = ctypes.CDLL(None)
>>> proc.main
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/arakaki/.pyenv/versions/3.9.7/lib/python3.9/ctypes/__init__.py", line 387, in __getattr__
    func = self.__getitem__(name)
  File "/home/arakaki/.pyenv/versions/3.9.7/lib/python3.9/ctypes/__init__.py", line 392, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: /home/arakaki/.pyenv/versions/3.9.7/bin/python3.9: undefined symbol: main
>>> proc._start
<_FuncPtr object at 0x7f686ff83040>
>>> proc.Py_Main
<_FuncPtr object at 0x7f686ff83640>

and in Python 3.10

Python 3.10.0 (default, Oct 24 2021, 00:31:24) [GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes
>>> proc = ctypes.CDLL(None)
>>> proc.main
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/arakaki/.pyenv/versions/3.10.0/lib/python3.10/ctypes/__init__.py", line 387, in __getattr__
    func = self.__getitem__(name)
  File "/home/arakaki/.pyenv/versions/3.10.0/lib/python3.10/ctypes/__init__.py", line 392, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: /home/arakaki/.pyenv/versions/3.10.0/bin/python3.10: undefined symbol: main
>>> proc._start
<_FuncPtr object at 0x7fe19e328940>
>>> proc.Py_Main
<_FuncPtr object at 0x7fe19e328a00>

Python 3.10 installed via conda also have the same error.

[tkf]

python3.9 binary seems to have main, though:

$ objdump -dC ~/.pyenv/versions/3.9.7/bin/python3.9 | grep -e '<main>' -e '<_start>' -A10
0000000000000720 <main>:
 720:   e9 db ff ff ff          jmpq   700 <Py_BytesMain@plt>
 725:   66 2e 0f 1f 84 00 00    nopw   %cs:0x0(%rax,%rax,1)
 72c:   00 00 00
 72f:   90                      nop

0000000000000730 <_start>:
 730:   31 ed                   xor    %ebp,%ebp
 732:   49 89 d1                mov    %rdx,%r9
 735:   5e                      pop    %rsi
 736:   48 89 e2                mov    %rsp,%rdx
 739:   48 83 e4 f0             and    $0xfffffffffffffff0,%rsp
 73d:   50                      push   %rax
 73e:   54                      push   %rsp
 73f:   4c 8d 05 6a 01 00 00    lea    0x16a(%rip),%r8        # 8b0 <__libc_csu_fini>
 746:   48 8d 0d f3 00 00 00    lea    0xf3(%rip),%rcx        # 840 <__libc_csu_init>
 74d:   48 8d 3d cc ff ff ff    lea    -0x34(%rip),%rdi        # 720 <main>
 754:   ff 15 86 08 20 00       callq  *0x200886(%rip)        # 200fe0 <__libc_start_main@GLIBC_2.2.5>
 75a:   f4                      hlt
 75b:   0f 1f 44 00 00          nopl   0x0(%rax,%rax,1)