Importing local module fails

[mlubin]

pyimport doesn't seem to be able to find python modules that are present in the local directory. Any reason for this? What's the best way to import user modules?

[stevengj]

python prepends '' to sys.path, whereas initializing libpython does not, by default.

More technically, we are passing 0 for the updatepath parameter of PySys_SetArgvEx as recommended in the Python library (see here) and as recommended for security reasons.

I'm not sure of the best option here. One option would be to just prepend '' to sys.path automatically all of the time (similar to the python executable?). Another would be to only do this in interactive mode, somehow. Or....?

@loladiro, how does Julia decide whether to import modules from the current directory?

[mlubin]

Maybe a custom option to pyinitialize? This should be available from both interactive and noninteractive modes, as python users would expect.

[Keno]

@stevengj Not sure what you mean. If you're talking about include it depends the most recently included file in the current task (the directory of that, or, if none cwd).

[stevengj]

@loladiro, if you type using Foo at the Julia prompt, it will load the module from a Foo.jl file in the current directory if it exists. This seems like the equivalent of having '' in the Python path. I'm not sure where this behavior is specified in the Julia source...

Furthermore, it looks like Julia has a similar security issue to Python... it looks like the current directory is searched before the other standard package directories.

[Keno]

@stevengj I think it just does require which falls back to include which uses the mechanism mentioned above.

[stevengj]

@loladiro, see my amended comment above... it looks like we may have the same security issue as in Python after all. e.g. if I add a PyPlot.jl to my current directory and do using PyPlot, Julia gets confused.

Or maybe because it is task-dependent this is okay?

[Keno]

It think this is different because using PyPlot in a package will search package directory for a PyPlot.jl. cwd is only applicable at the prompt (I guess we could say the source directory of the prompt is implicitly cwd). I think the issue in python is that the search path gets set globally.

[stevengj]

@mlubin, regarding adding an option to pyinitialize, I don't see why this is better than just telling people to modify sys.path, e.g.

unshift!(PyVector(pyimport("sys")["path"]), "")

although one might provide a convenience function for this. (e.g. I could just define PyCall.path as a PyVector mirror of sys.path.) (Note that this only works with the latest version of PyCall, which adds unshift! to PyVectors.)

However, I'd like to do something sensible by default. If Python users are used to being able to load things in the current directory in python, they should be able to do that in PyCall without manually modifying the path. The question is, how can I do this in a way that is reasonably secure?

[stevengj]

I could mimic Julia's behavior: every pyimport call could modify sys.path based on the (Julia) directory of the current task. This seems reasonable to me, but I'm not sure whether it might have unexpected consequences in Python.

[malmaud]

What if we called Python's os.chdir whenever Julia's cd was called (as well as in pyinitialize)?
EDIT: NM, it's necessary and sufficient to modify sys.path.

[stevengj]

How would we modify Julia's cd?

[malmaud]

I think I'm not understanding the security issue with always prepending sys.path with "" in pyinitialize. If I execute a file test.jl with python like

import numpy
...

and a file called numpy.py is in the same directory as test.py, then that local numpy.py will be imported. PyCall would then have the same behavior. Is that what you're trying to avoid?

[stevengj]

It's not where test.jl is located that is the problem. What we want to avoid is having the current directory in the default path, independent of the location of the file that is being executed. You don't want unexpected behavior to result just from executing Julia from the wrong path.