[WIP] - Mixed thread/processes evaluation

[mrocklin]

Fixes #329

This starts an attempt to run tasks in processes even though we use the multiprocessing scheduler. We wrap tasks in run_in_process function which calls out to an external pool.

Some issues:

1. This doesn't support nested tasks, we may want some form of quoting for this
2. I suspect that creating the pool may cause issues in windows
3. This doesn't actually use this functionality anywhere. Good places to experiment are Bag.to_dataframe and dask.array.from_hdf5

[mrocklin]

@shoyer your thoughts on this would be appreciated.

[shoyer]

What about a user facing API (e.g., the context manager)? So far I have successfully avoided writing dask graphs directly in xray :).

For us, I suppose all we need is a parameter we could pass to da.from_array and da.store (e.g., multiprocess=True) that signals that each call to getitem/setitem should be wrapped in run_in_process.

[mrocklin]

I'm concerned about using these functions in processes with unserializable hdf5 dataset objects. Thoughts on how best to integrate run_in_processes to these top-level functions?

[mrocklin]

Or rather, for this to work with processes and xray I expect to have to build a task that has a function that opens up a file using the netCDF4 library. I was hoping to avoid explicitly having from_h5py and from_netcdf4 functions.

[shoyer]

I agree that dask should not have from_h5py and from_netcdf4 functions.

My thought was that I would write an object like the following in xray, to be passed off to da.from_array:

from contextlib import contextmanager

class NetCDFArray(object):
    def __init__(self, filename, variable_name):
        self.filename = filename
        self.variable_name = variable_name
        with self._open() as var:
             self._dtype = var.dtype
             self._shape = var.dtype

    @contextmanager
    def _open(self):
        with netCDF4.Dataset(self.filename) as nc:
            var = nc.variables[self.variable_name]
            yield var

    @property
    def dtype(self):
        return self._dtype

    @property
    def shape(self):
        return self._shape

    def __getitem__(self, key):
        with self._open() as var:
            return var[key]

    def __setitem__(self, key, value):
        with self._open() as var:
            var[key] = value

[mrocklin]

OK, well, see 820784f . It's totally untested but may do what we need.

[mrocklin]

@shoyer can I ask you to give this a spin? Possibly with something like the NetCDFArray object that you have above.

[shoyer]

Sorry, I've been busy this week and have not been able to get to this. Now I'm headed into the woods for the next 6 days....

On Tue, Aug 4, 2015 at 8:31 AM, Matthew Rocklin [email protected]
wrote:

@shoyer can I ask you to give this a spin? Possibly with something like the NetCDFArray object that you have above.

Reply to this email directly or view it on GitHub:
#457 (comment)

[mrocklin]

Have fun!

[mrocklin]

Closing this as stale. Happy to reopen on renewed interest.