Better integration with Pluto

[fonsp]

Hi!!! I love this package!

Is there something we can do to provide better integration with Pluto?

For example: take a look at https://github.com/fonsp/Pluto.jl/wiki/%F0%9F%8E%81-Package-management and fonsp/Pluto.jl#844 . Perhaps we can also store the contents of CondaPkg.toml in the notebook, just like Project.toml and Manifest.toml?

-fons, cc @icweaver @dralletje

[icweaver]

Yea, this package feels like magic. I am loving all of the symmetries between Pkg and CondaPkg, especially the REPL mode and updated printing for status that you just added.

I was wondering if CondaPkg.status was planning to get a similar treatment? Being able to combine it with PlutoUI's @with_terminal has been really handy for quickly checking the current environment from directly within the notebook =D

Edit:

I was wondering if CondaPkg.status was planning to get a similar treatment?

It did, and it's beautiful

[cjdoris]

Thanks very much!

I've just been trying out PythonCall + Pluto and it already works OK.

One thing is that you need to do

using CondaPkg
x = CondaPkg.add("somepackage")
x; using PythonCall

to force the CondaPkg.add to happen before using PythonCall. So I guess one thing would be to have special rules to give CondaPkg.add and friends higher priority than most imports, so you don't need to fudge the DAG by hand like this.

Of course all the CondaPkg logging happens on the server - it would be neat for some of this to show in Pluto too but I'm not sure how you could do this in a nice generic way. Maybe Pluto could have a rule that when the project depends on CondaPkg, then Pluto will call CondaPkg.resolve() after using CondaPkg and CondaPkg.add(...) but before any other imports.

AFAICT Pluto starts a brand new temporary project each time you (re-)open a notebook. This is a bit of a killer for CondaPkg because Conda environments are not cheap to create like Julia ones - especially on Windows. So a big plus for the user might be to re-use the same temporary projects - maybe you could record the location of the project in the notebook file, and put a hash of the notebook into the project somewhere, then re-use the project if these things agree. You'd still have to wait for Conda to install stuff each day though, since the temporary projects are deleted whenever you close Julia.

[cjdoris]

On that last point, maybe you shouldn't use the temporary project mechanism at all. Instead Pluto could manage its own pool of projects, and periodically delete projects whose notebook file has disappeared. Then a notebook would have a longer-lived project and the Conda environment could stay unchanged.

[icweaver]

Really interesting!

AFAICT Pluto starts a brand new temporary project each time you (re-)open a notebook. This is a bit of a killer for CondaPkg because Conda environments are not cheap to create like Julia ones - especially on Windows. So a big plus for the user might be to re-use the same temporary projects

Please correct me if I'm wrong Fons and Michiel, but I think this is already avoided when the Pluto package manager is disabled. I've sort of fallen into the habit of placing a cell like this at the top of a notebook (the shared env workflow) whenever I want to re-use an env, but maybe there is a more ergonomic way:

begin
    import Pkg
    Pkg.activate(Base.current_project())
	
    using CondaPkg
    CondaPkg.add("numpy"); CondaPkg.resolve()

    using PythonCall
end

So the first time a notebook is run, it does the usual:

    Updating registry at `~/.julia/registries/General.toml`
      From worker 2:	  Activating project at `~/Desktop/scratch`
      From worker 2:	    CondaPkg Found dependencies: /home/mango/Desktop/scratch/CondaPkg.toml
      From worker 2:	    CondaPkg Found dependencies: /home/mango/.julia/packages/PythonCall/B5irB/CondaPkg.toml
      From worker 2:	    CondaPkg Creating environment
      From worker 2:	             │ /home/mango/.julia/artifacts/278eb11bd219794ab18506d6b4cbdf91aa259bff/bin/micromamba
      From worker 2:	             │ -r /home/mango/.julia/scratchspaces/0b3b1443-0f03-428d-bdfb-f27f9c1191ea/root
      From worker 2:	             │ create
      From worker 2:	             │ -y
      From worker 2:	             │ -p /home/mango/Desktop/scratch/.CondaPkg/env
      From worker 2:	             │ --no-channel-priority
      From worker 2:	             │ numpy
      From worker 2:	             │ python >=3.5,<4
      From worker 2:	             └ -c conda-forge
      From worker 2:	
      From worker 2:	                                           __
      From worker 2:	          __  ______ ___  ____ _____ ___  / /_  ____ _
      From worker 2:	         / / / / __ `__ \/ __ `/ __ `__ \/ __ \/ __ `/
      From worker 2:	        / /_/ / / / / / / /_/ / / / / / / /_/ / /_/ /
      From worker 2:	       / .___/_/ /_/ /_/\__,_/_/ /_/ /_/_.___/\__,_/
      From worker 2:	      /_/
      From worker 2:	
      From worker 2:	conda-forge/noarch        
      From worker 2:	conda-forge/linux-64      
      From worker 2:	
      From worker 2:	Transaction
      From worker 2:	
      From worker 2:	  Prefix: /home/mango/Desktop/scratch/.CondaPkg/env
      From worker 2:	
      From worker 2:	  Updating specs:
      From worker 2:	
      From worker 2:	   - numpy
      From worker 2:	   - python[version='>=3.5,<4']
      From worker 2:	
      From worker 2:	
      From worker 2:	  Package               Version  Build                Channel                    Size
      From worker 2:	───────────────────────────────────────────────────────────────────────────────────────
      From worker 2:	  Install:
      From worker 2:	───────────────────────────────────────────────────────────────────────────────────────
...

And then closing the notebook and coming back to it again works beautifully with PythonCall.jl and friends:

Press Ctrl+C in this terminal to stop Pluto

    Updating registry at `~/.julia/registries/General.toml`
      From worker 2:	  Activating project at `~/Desktop/scratch`
      From worker 2:	    CondaPkg Found dependencies: /home/mango/Desktop/scratch/CondaPkg.toml
      From worker 2:	    CondaPkg Found dependencies: /home/mango/.julia/packages/PythonCall/B5irB/CondaPkg.toml
      From worker 2:	    CondaPkg Dependencies already up to date

So I guess it just comes down to what particular workflow happens to be used for a given project (fully reproducible workflow with Pluto's package manager vs. "BYOP" (bring your own project)), but it sounds like your last suggestion:

On that last point, maybe you shouldn't use the temporary project mechanism at all. Instead Pluto could manage its own pool of projects, and periodically delete projects whose notebook file has disappeared. Then a notebook would have a longer-lived project and the Conda environment could stay unchanged.

might be a way to combine the best of both worlds?

[icweaver]

to force the CondaPkg.add to happen before using PythonCall

Oh, thanks for pointing this out! Updating my examples now, haha

[cjdoris]

Closing as I think PythonCall and Pluto already work really well together :). Feel free to reopen.