Reading Dataset from memory

[m-novikov]

Can netcdf4 read Dataset from memory?
Something like this:

from netCDF4 import Dataset
f = open('netcdf_file.nc', 'rb')
df = Dataset(f)

It would be quite convenient and useful API.
As for now to read netCDF4 Dataset from URL I need explicitly save it as named temporary file.

[lesserwhirls]

Greetings!

We are working on this on the C side at Unidata, and hopefully it will be
finished soon. Once it's in the C lib, there should be very little work to
expose it in the netcdf4-python lib.

Cheers!

Sean

On Tuesday, May 5, 2015, Maxim Novikov notifications@github.com wrote:

Can netcdf4 read Dataset from memory?
Something like this:

from netCDF4 import Dataset
f = open('netcdf_file.nc', 'rb')
df = Dataset(f)

It would be quite convenient and useful API.
As for now to read netCDF4 Dataset from URL I need explicitly save it as
named temporary file.

—
Reply to this email directly or view it on GitHub
#406.

[dopplershift]

To be clear, though, this would involve passing the entire "string" of data, not a file-like object.

[jswhit]

you can create a Dataset in memory, using 'diskless=True'.

[lesserwhirls]

Create, yes, but read - no.

Let's say you use the NetcdfSubset service from the THREDDS Data Server.
The server will return a netCDF file. When you use urllib2 to make the
request, you end up reading the server response, which is the bytes of a
netcdf file already in memory. The idea is to read in memory to remove the
need to write a temporary file to disk.

Sean

On Tue, May 5, 2015 at 1:31 PM, Jeff Whitaker notifications@github.com
wrote:

you can create a Dataset in memory, using 'diskless=True'.

—
Reply to this email directly or view it on GitHub
#406 (comment)
.

[jswhit]

you could copy the data directly to a diskless file (without first writing to disk) couldn't you?

[dopplershift]

Define "diskless file".

NetCDF requires a filename to read data from.

[jswhit]

from netCDF4 import Dataset
nc = Dataset(URL)
ncm = Dataset('inmemory.nc',diskless=True,mode='w')
.....logic to copy data from nc to ncm...

then you have a 'diskless' or in-memory version of the dataset at URL

[dopplershift]

Is the nc = Dataset(URL) supposed to work for anything besides opendap? It didn't for me when I just tried to hit a THREDDS server using NCSS. What I'm picturing is:

from netCDF4 import Dataset
from urllib.requests import urlopen
url = urlopen(URL)
ncdata = url.read()
nc = Dataset(ncdata, diskless=True, mode='r')

[m-novikov]

Passing string seems like fine idea, in my case of usage this will be convenient enough (I read 10mb NetCDF files which is not problem to store in python memory).
But reading whole data in memory for python will increase memory consumption and this string will need to be garbage collected which is not always happens fast in CPython.
Why not stick to file interface? This will give developer possibility to implement most convenient way handling the source of data, for string only it can be wrapped with cStringIO/StringIO.

PS. I dont really know binary structure of NetCDF4 format, event if it cannot be read/written incrementally, handling buffer consumption on C side of python extension can be good for large files.

[dopplershift]

Well right now, the C netcdf library only takes a filename or an opendap URL; there's not even the option of taking any kind of file pointer. Even if the latter was possible, you still wouldn't be able to turn a StringIO instance into such a thing for standard C.

What they're adding to the netCDF C library is an API to point to an existing in-memory buffer and eliminate all file I/O; HDF5 already has such an API. It would be possible to add a Python API to netcdf4-python to take a file-like object, but at some level here all of the data needs to be read into a buffer, with a single pointer to be handed to the C-library. This is likely not to actually be a str, but I'm not sure if it's bytes, bytearray, memoryview or what (direct conversion to non-Python buffer?). I still need to learn a bit more Cython...

[shoyer]

There's been some discussion about this over on the h5py issue tracker (h5py/h5py#552). It sounds like some changes to the HDF5 libraries may be necessary to make this worth entirely smoothly.

In the meantime, if you're working with netCDF3 files, using a file like object is already possible with the scipy.io.netcdf interface.

[m-novikov]

Thank you. As I read NetCDF4 files, for now I settle with NamedTemporaryFile workaround.

[thehesiod]

looked into using a local OpenDAP server but couldn't find anything that easily worked serving local netCDF files. This would be an option if anyone can get it to work.