How to access Python variables from %%Julia in ipython?

[schlichtanders]

Hello,
after successfully pip installing juliacall and loading its ipython extension, I cannot find a way to access variables defined on the python side.

Say I have a normal python cell

python_variable = "find me"

I now would like to access this from julia

%%julia
using PythonCall
# pyeval(...) ?
# pybuiltins... ?

I haven't found a way. Can you help and provide some documentation?

[schlichtanders]

it seems that when starting a Python Jupyter notebook and activating juliacall.ipython, the Python process run by juliacall is not the same python as the notebook itself.

This leads to numerous problems that Python objects cannot be successfully transferred to julia. Many are represented lazily by references (which is PythonCall's style), however those references are invalid in the julia's python process.

[schlichtanders]

After further research, I've found that PyCall has also some problems interfacing python&julia back and forth on some python installations (including my one). They have this recent issue which seems to be their way to go forward. Maybe that can also be helpful for PythonCall

[cjdoris]

It's important to note that %%julia executes code in the Julia module Main whereas your Python code executes in __main__, and there is no mechanism to synchronise them. The existing solution to your problem is to

from juliacall import Main as jl

and set/get variables on jl.

However I have just pushed a change (undocumented and experimental) to the main branch which allows you to specify a set of variables to synchronise between Julia and Python:

In [1]: %load_ext juliacall.ipython

In [2]: x = 2

In [3]: y = 8

In [4]: %%julia x y z
   ...: z = "$x^$y = $(x^y)";
   ...:
   ...:

In [5]: z
Out[5]: '2^8 = 256'

Try it out and let me know what you think. You can install it with

pip install git+https://github.com/cjdoris/PythonCall.jl

[schlichtanders]

Looks awesome!

Can you check whether the new experimental feature can also transfer python lists? That was failing completely for me

[cjdoris]

Can you be more specific in what didn't work, e.g. show some code+output?

[schlichtanders]

I found again time looking into PythonCall/JuliaCall (the motivation being more consistent and better benchmark timings of julia code when compared to PyCall/PyJulia)

I tried to install dev, however it fails during import of juliacall:

ERROR: InitError: AssertionError: CTX.which == :PyCall
Stacktrace:
  [1] (::PythonCall.C.var"#35#43"{PythonCall.C.var"#python_cmd#41"})()
    @ PythonCall.C ~/.julia/packages/PythonCall/dsECZ/src/cpython/context.jl:148
  [2] with_gil
    @ ~/.julia/packages/PythonCall/dsECZ/src/cpython/gil.jl:10 [inlined]
  [3] with_gil
    @ ~/.julia/packages/PythonCall/dsECZ/src/cpython/gil.jl:9 [inlined]
  [4] init_context()
    @ PythonCall.C ~/.julia/packages/PythonCall/dsECZ/src/cpython/context.jl:145
  [5] __init__()
    @ PythonCall.C ~/.julia/packages/PythonCall/dsECZ/src/cpython/CPython.jl:21
  [6] _include_from_serialized(pkg::Base.PkgId, path::String, depmods::Vector{Any})
    @ Base ./loading.jl:831
  [7] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt64)
    @ Base ./loading.jl:1039
  [8] _require(pkg::Base.PkgId)
    @ Base ./loading.jl:1315
  [9] _require_prelocked(uuidkey::Base.PkgId)
    @ Base ./loading.jl:1200
 [10] macro expansion
    @ ./loading.jl:1180 [inlined]
 [11] macro expansion
    @ ./lock.jl:223 [inlined]
 [12] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1144
 [13] top-level scope
    @ none:8
during initialization of module C

---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
Cell In[2], line 1
----> 1 from juliacall import Main as jl
      2 get_ipython().run_line_magic('load_ext', 'juliacall')

File ~/Projects/Jolin.io/workshop-accelerate-Python-with-Julia/.venv/lib/python3.10/site-packages/juliacall/__init__.py:218
    214         raise Exception('PythonCall.jl did not start properly')
    216     CONFIG['inited'] = True
--> 218 init()
    220 def load_ipython_extension(ip):
    221     import juliacall.ipython

File ~/Projects/Jolin.io/workshop-accelerate-Python-with-Julia/.venv/lib/python3.10/site-packages/juliacall/__init__.py:214, in init()
    212 res = jl_eval(script.encode('utf8'))
    213 if res is None:
--> 214     raise Exception('PythonCall.jl did not start properly')
    216 CONFIG['inited'] = True

Exception: PythonCall.jl did not start properly

help is appreciated

[cjdoris]

I have made a release, so if you update to JuliaCall v0.9.13 then it should work.

[schlichtanders]

A new try, and again a failure. I am running v0.9.13 on both sites

In jupyter with python 3.10.9 kernel doing

from juliacall import Main as jl
%load_ext juliacall.ipython

# JuliaCall comes with its own Julia dependency file juliapkg.json
# however for binder it is much simpler to just reuse binder's installation mechanism
%julia Pkg.activate(Base.current_project())
%julia using PythonCall

x = 4

%%julia x
println(x)

gives

---------------------------------------------------------------------------
JuliaError                                Traceback (most recent call last)
Cell In[3], line 1
----> 1 get_ipython().run_cell_magic('julia', 'x', 'println(x)\n')

File [~/Projects/Jolin.io/workshop-accelerate-Python-with-Julia/.venv/lib/python3.10/site-packages/IPython/core/interactiveshell.py:2475](https://file+.vscode-resource.vscode-cdn.net/home/ssahm/Projects/fall-in-love-with-julia/~/Projects/Jolin.io/workshop-accelerate-Python-with-Julia/.venv/lib/python3.10/site-packages/IPython/core/interactiveshell.py:2475), in InteractiveShell.run_cell_magic(self, magic_name, line, cell)
   2473 with self.builtin_trap:
   2474     args = (magic_arg_s, cell)
-> 2475     result = fn(*args, **kwargs)
   2477 # The code below prevents the output from being displayed
   2478 # when using magics with decodator @output_can_be_silenced
   2479 # when the last Python token in the expression is a ';'.
   2480 if getattr(fn, magic.MAGIC_OUTPUT_CAN_BE_SILENCED, False):

File [~/Projects/Jolin.io/workshop-accelerate-Python-with-Julia/.venv/lib/python3.10/site-packages/juliacall/ipython.py:24](https://file+.vscode-resource.vscode-cdn.net/home/ssahm/Projects/fall-in-love-with-julia/~/Projects/Jolin.io/workshop-accelerate-Python-with-Julia/.venv/lib/python3.10/site-packages/juliacall/ipython.py:24), in JuliaMagics.julia(self, line, cell)
     21 @line_cell_magic
     22 def julia(self, line, cell=None):
     23     code = line if cell is None else cell
---> 24     ans = Main.seval('begin\n' + code + '\nend')
     25     PythonCall._flush_stdio()
     26     if not code.strip().endswith(';'):

File [~/.julia/packages/PythonCall/dsECZ/src/jlwrap/module.jl:25](https://file+.vscode-resource.vscode-cdn.net/home/ssahm/Projects/fall-in-love-with-julia/~/.julia/packages/PythonCall/dsECZ/src/jlwrap/module.jl:25), in seval(self, expr)
     23         return ValueBase.__dir__(self) + self._jl_callmethod($(pyjl_methodnum(pyjlmodule_dir)))
     24     def seval(self, expr):
---> 25         return self._jl_callmethod($(pyjl_methodnum(pyjlmodule_seval)), expr)
     26 """, @__FILE__(), "exec"), jl.__dict__)
     27 pycopy!(pyjlmoduletype, jl.ModuleValue)

JuliaError: UndefVarError: `x` not defined
Stacktrace:
 [1] top-level scope
   @ none:2
 [2] eval
   @ [./boot.jl:370](https://file+.vscode-resource.vscode-cdn.net/home/ssahm/Projects/fall-in-love-with-julia/boot.jl:370) [inlined]
 [3] eval
   @ [./Base.jl:68](https://file+.vscode-resource.vscode-cdn.net/home/ssahm/Projects/fall-in-love-with-julia/Base.jl:68) [inlined]
 [4] pyjlmodule_seval(self::Module, expr::Py)
   @ PythonCall [~/.julia/packages/PythonCall/dsECZ/src/jlwrap/module.jl:13](https://file+.vscode-resource.vscode-cdn.net/home/ssahm/Projects/fall-in-love-with-julia/~/.julia/packages/PythonCall/dsECZ/src/jlwrap/module.jl:13)
 [5] _pyjl_callmethod(f::Any, self_::Ptr{PythonCall.C.PyObject}, args_::Ptr{PythonCall.C.PyObject}, nargs::Int64)
   @ PythonCall [~/.julia/packages/PythonCall/dsECZ/src/jlwrap/base.jl:62](https://file+.vscode-resource.vscode-cdn.net/home/ssahm/Projects/fall-in-love-with-julia/~/.julia/packages/PythonCall/dsECZ/src/jlwrap/base.jl:62)
 [6] _pyjl_callmethod(o::Ptr{PythonCall.C.PyObject}, args::Ptr{PythonCall.C.PyObject})
   @ PythonCall.C [~/.julia/packages/PythonCall/dsECZ/src/cpython/jlwrap.jl:47](https://file+.vscode-resource.vscode-cdn.net/home/ssahm/Projects/fall-in-love-with-julia/~/.julia/packages/PythonCall/dsECZ/src/cpython/jlwrap.jl:47)

[cjdoris]

I can tell from the stacktrace that you are not on JuliaCall 0.9.13.

Note however that this bit of your code:

from juliacall import Main as jl
%load_ext juliacall.ipython

# JuliaCall comes with its own Julia dependency file juliapkg.json
# however for binder it is much simpler to just reuse binder's installation mechanism
%julia Pkg.activate(Base.current_project())
%julia using PythonCall

is not doing what you think. That very first import juliacall line imports both JuliaCall and PythonCall. This means that the using PythonCall line later is using PythonCall from JuliaCall's default project and not from Base.current_project().

In general, it is not recommended to switch projects in Julia mid-workflow because you can end up with incompatible packages loaded.

If you want to use an existing Julia project, you should just do this:

import os
os.environ['PYTHON_JULIAPKG_PROJECT'] = '/path/to/project'
os.environ['PYTHON_JULIAPKG_OFFLINE'] = 'yes'
%load_ext juliacall

[schlichtanders]

I owe you two beers, at least 🙂 . Thank you so much for your help. Yeah the confusion of the juliacall versions was kicking me massively and without your help I wouldn't have noticed.

This works lovely with the new juliacall version. I am very happy that I still can present it today evening.

Thank you also for the hint with the environments - I will include that right away.

[cjdoris]

I'm glad you got it working. I don't suppose you have a recording of your presentation?

[schlichtanders]

indeed the session was recorded 👍 , it was a 1.5 hours meetup, presenting both PyCall.jl and PythonCall.jl as well as a final benchmark demo Julia vs cpp

here the youtube link