Explicit backend use in case of LOCAL run

[raphaelauv]

When we run MODIN in local whitout providing any dask or .. cluster does it run with the multiprocessing package of the lib python? This could be explicited in the documentation and README.md

At first I was looking how and why would modin run a "fake" dask cluster in local.

What do you think about it ? Thank you

[devin-petersohn]

Hi @raphaelauv, we do not currently have support for a Python multiprocessing engine. Did you see that in the documentation? Is that something you want?

Modin doesn't use a "fake" dask cluster, it creates a Dask Client which uses a worker per CPU core. Does that make sense?

Please let me know if I didn't answer your question!

[raphaelauv]

Thank for your answer

I made this assumption from the diagram on this page https://modin.readthedocs.io/en/latest/developer/architecture.html , I throught by PYTHON backend this mean using the multiprocessing package.

( Also this video talk about a multiprocessing backend https://youtu.be/yhjzMk5Cmvk?t=399 ... )

So it create a local dask cluster. Should the documentation talk about a fake simulated dask cluster in case of local run ?

[devin-petersohn]

I see, the Python backend is a pure python engine with no parallelism. We use it for mostly debugging, and it is not very fast at larger datasets.

The video you link has added some future work plans, but not labeled them. MPI and Python multiprocessing are plans for future work.

Should the documentation talk about a fake simulated dask cluster in case of local run ?

Yes! Our documentation needs updating, and we have had many outside contributors add improvements to the documentation during our recent Pydata Global sprint. Would you like to contribute this to the documentation?