PyJulia does not work with Julia 1.0 & 0.7 with Python installed via Conda/Ubuntu/(what else?)

[tkf]

I'm writing down the issue so that it is easy to find by the users.

At the moment, for PyJulia to load Julia >= 0.7, two conditions have to be met:

1. Python interpreter configured for PyCall.jl and Python interpreter used to import PyJulia have to use identical libpython (once Support virtualenv/venv without separate cache #190 is merged) have exactly identical paths.

2. The Python interpreter have to be dynamically linked (which is usually the case in Windows and macOS according to this comment)

To check if condition 2 is the case, run:

$ ldd /usr/bin/python | grep libpython
        libpython3.7m.so.1.0 => /usr/lib/libpython3.7m.so.1.0 (0x00007f17c12c4000)

where /usr/bin/python is the appropriate Python interpreter.

If it does not print anything, you can't use PyJulia with this Python interpreter.

[mprogram]

This was very helpful; thank you, @tkf

I must say I was very-very surprised by not finding libpython3 in my ldd $(which python3) output.

Debian/Ubuntu packagers are the known strong advocates against linking things statically; but doing so in python/3?!?!

[tkf]

Yeah, I was surprised too. I thought it was just conda (in which static compilation maybe makes sense).

[Hong-Xiang]

@tkf Thanks for your reply. But there are still some questions:

• Currently we do not have any method to call Julia in conda based/Ubuntu/Debian python3, right? Even dirty/tricky method would be helpful.
• Could we use custom compiled python for it? If true, is this a practical solution?
• Will this issue be solved in near future or we should not expect that?

Thanks.

[tkf]

I think there are three ways to use Python with Julia 1.0 on Ubuntu/Debian/etc. at the moment:

• If you just want to use your Python module, you can import it from PyCall.jl and call the functions in it. See, e.g., https://github.com/tkf/IPython.jl/blob/d5b27ac6e38f1b96ba42099da1098975cdd8b9be/src/core.jl#L31 and https://github.com/tkf/IPython.jl/blob/d5b27ac6e38f1b96ba42099da1098975cdd8b9be/src/core.jl#L20 in IPython.jl/src/core.jl.
• If you just want an interactive Python session, you can use my IPython.jl to drop into IPython through Julia REPL.
• If you want fully functional PyJulia in Python, one way to do it is though docker. Check out: https://github.com/tkf/docker-ipyjulia

Will this issue be solved in near future

I think I know how to fix it #173. But likely not in next weeks. I can't grant anything :) If I were to do it I want to do it after dropping Julia 0.6 (which requires PyJulia to be released soon).

[tkf]

I made an experimental package julia-venv: https://github.com/tkf/julia-venv

It implements the private DEPOT_PATH hack mentioned in #173 but does so per-virtualenv basis. This let us use different Python executables without recompiling PyCall.jl all the time. On top of creating a private DEPOT_PATH, it installs a CLI julia-venv which acts like the normal julia interpreter (by invoking it via C API from Python). This way, we can run PyCall.jl always in "PyJulia-mode" so that the issue with Python in Ubuntu/Debian and Conda can be avoided (https://github.com/tkf/julia-venv#how-it-works). As a nice side-effect, you can also use PyCall.jl with different Python executable and different set of Python libraries.

Edit: Actually, julia-venv probably still doesn't work with Conda (and probably also with Ubuntu): tkf/julia-venv#2 Edit^2: It works now.

[Hong-Xiang]

@tkf ，thanks for your awesome work. The new package works on Ubuntu 16.04 with anaconda. It really helps.

There are some minor suggestions (or some misunderstanding of me).

third-party packages need to be re-compiled/re-installed

for test purpose, I test from julia import NPZ, while NPZ is already added in original julia environment. But it failed.

and the following code works:

from julia import Pkg
Pkg.add("NPZ")
#... some install outputs
from julia import NPZ
data = NPZ.npzread("path")

It seems that pyjulia and julia of system is using different package cache? I think re-install packages is not a problem and it might be a valuable feature. I only suggest add some doc for this

convenient broadcast call

broadcast is commonly used in julia, but Main.abs.([1,2,3]) is a SyntaxError in python.
in README.md of pyjulia, there is

Main.eval("sin.(xs)")

which seems that pyjulia will not expose broadcast method to python level. I'm not sure if I am right, but if it is true, I'm thinking is it possible to adding some special syntax like Main.abs.broadcast([1,2,3]) or in short Main.abs.b([1,2,3]) (although may introduce some confusing to sum_b for python version sum!).

InterruptException() handling

When I press ctrl-C in python REPL, there would be a small problem.
If I'm using a pure python REPL, it should work like this

>>> # presss ctrl-C here
KeyboardInterrupt
>>> 

But in a python REPL which loaded julia, it would work like this:

>>> # press ctrl-C here

and nothing happens, after that if I enter next command

>>> # Ctrl - C, and nothing happens
>>> Main.xs = [1,2,3] # Enter, change line and type new command and Enter I got:
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "(path_anaconda)/envs/juliaenv/lib/python3.6/site-packages/julia/core.py", line 140, in __setattr__
    self._julia.eval(setter)(value)
  File "(path_anaconda)/envs/juliaenv/lib/python3.6/site-packages/julia/core.py", line 606, in eval
    ans = self._call(src)
  File "(path_anaconda)/envs/juliaenv/lib/python3.6/site-packages/julia/core.py", line 562, in _call
    self.check_exception(src)
  File "(path_anaconda)/envs/juliaenv/lib/python3.6/site-packages/julia/core.py", line 589, in check_exception
    .format(exception, src))
julia.core.JuliaError: Exception 'InterruptException' occurred while calling julia code:

            PyCall.pyfunctionret(
                (x) -> eval(Main, :(xs = $x)),
                Any,
                PyCall.PyAny)

It seems the InterruptException is send to julia in one delayed statement?

[tkf]

@Hong-Xiang Hey, thanks for the feedback. Are you using julia-venv in Ubuntu or Debian? Did it actually work?

I think re-install packages is not a problem and it might be a valuable feature.

Yeah, something like virtualenv's --system-site-packages would be nice. I actually did that in first versions but stopped doing that since I encountered with some quirks in julia's package handling. It it actually as "easy" as replacing depots = filter(!isequal(default_depot1), __py_julia_venv_orig_DEPOT_PATH) with depots = copy(__py_julia_venv_orig_DEPOT_PATH).

convenient broadcast call

I started experimenting more numpy-like interface on top of PyJulia (i.e., broadcast operations on arrays by default):

https://ipyjulia-hacks.readthedocs.io/en/latest/

Not sure if it is suitable for PyJulia, though.

But if you want to leverage Julia, I think it's better to just use Julia syntax. If you don't want to use global variable (of course), you can do:

Main.eval("(xs) -> sin.(exp.(xs) .+ 1)")(xs)

or equivalently

Main.eval("(xs) -> @. sin(exp(xs) + 1)")(xs)

Unlike numpy.exp etc., it doesn't allocate intermediate arrays.

InterruptException

Hmm... I didn't notice that. But yes, I can reproduce it. Actually, I just noticed that it's reported in #189. Let's track the issue there.

It looks like IPython 7 (development version) handle ctrl-C differently and it doesn't have this problem. I've been using it so I didn't notice the problem.

[Hong-Xiang]

@tkf my system/environment info is:

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.5 LTS
Release:	16.04
Codename:	xenial

and actually it's Ubuntu Mate.

conda:

$ conda --version
conda 4.3.30

python(in conda virutal env, with julia-venv)

$ source activate juliaenv

(juliaenv) $ which python
(home path)/anaconda3/envs/juliaenv/bin/python

(juliaenv)  $ python
Python 3.6.2 |Continuum Analytics, Inc.| (default, Jul 20 2017, 13:51:32) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from julia_venv import get_julia
>>> get_julia()
<julia.core.Julia object at 0x7f66c6cc7f28>
>>> from julia import Main

it worked, without error.

python(in conda virtual env, without julia-venv)

(juliaenv)  $ python
Python 3.6.2 |Continuum Analytics, Inc.| (default, Jul 20 2017, 13:51:32) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from julia import Main
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load
  File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 646, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 616, in _load_backward_compatible
  File "(home path)/anaconda3/envs/juliaenv/lib/python3.6/site-packages/julia/core.py", line 172, in load_module
    JuliaMainModule(self, fullname))
  File "(home path)/anaconda3/envs/juliaenv/lib/python3.6/site-packages/julia/core.py", line 77, in __init__
    self._julia = loader.julia
  File "(home path)/anaconda3/envs/juliaenv/lib/python3.6/site-packages/julia/core.py", line 164, in julia
    self.__class__.julia = julia = Julia()
  File "(home path)/anaconda3/envs/juliaenv/lib/python3.6/site-packages/julia/core.py", line 496, in __init__
    "\"lib\", \"pyjulia%s-v$(VERSION.major).$(VERSION.minor)\"))" % sys.version_info[0])
  File "(home path)/anaconda3/envs/juliaenv/lib/python3.6/site-packages/julia/core.py", line 562, in _call
    self.check_exception(src)
  File "(home path)/anaconda3/envs/juliaenv/lib/python3.6/site-packages/julia/core.py", line 589, in check_exception
    .format(exception, src))
julia.core.JuliaError: Exception 'UndefVarError' occurred while calling julia code:
unshift!(Base.LOAD_CACHE_PATH, abspath(Pkg.Dir._pkgroot(),"lib", "pyjulia3-v$(VERSION.major).$(VERSION.minor)"))
>>> 

tests like using standard libraries, basic array operations and even third-party packages like NPZ is performed, and it seems just worked, as mentioned above.
Is there any further tests I can do for this?

[tkf]

Thanks. Interesting... Actually, the problem I thought I had was already fixed tkf/julia-venv#2. My guess was that it was some kind of IO caching issue (compilation file was not properly written before exiting the problem). It then maybe explains why it's machine-dependent.

Anyway, I'm glad that it worked in your machine!