Sparse arrays

[mrocklin]

I would like to have an XArray that has scipy.sparse arrays rather than numpy arrays. Is this in scope?

What would need to happen within XArray to support this?

[shoyer]

Yes, I would say this is in scope, as long as we can keep most of the data-type specific logic out of xarray's core (which seems doable).

Currently, we define most of our operations on duck arrays in https://github.com/pydata/xarray/blob/master/xarray/core/duck_array_ops.py

There are a few other hacks throughout the codebase, which can find by searching for "dask_array_type": https://github.com/pydata/xarray/search?p=1&q=dask_array_type&type=&utf8=%E2%9C%93

It's pretty crude, but basically this would need to be extended to implement many of these methods on for sparse arrays, too. Ideally we would define xarray's adapter logic into more cleanly separated submodules, perhaps using multiple dispatch. Even better, we would make this public API, so you can write something like xarray.register_data_type(MySparseArray) to register a type as valid for xarray's .data attribute.

It looks like __array_ufunc__ will actually finally land in NumPy 1.13, which might make this easier.

See also #1118

[rabernat]

👍 to the scipy.sparse array suggestion

[While we are discussing supporting other array types, we should keep gpu arrays on the radar]

[benbovy]

Although I don't know much about SciDB, it seems to be another possible application for xarray.register_data_type.

[mrocklin]

Here is a brief attempt at a multi-dimensional sparse array: https://github.com/mrocklin/sparse

It depends on numpy and scipy.sparse and, with the exception of a bit of in-memory data movement and copies, should run at scipy speeds (though I haven't done any benchmarking).

@rabernat do you have an application that we could use to drive this?

[rabernat]

@rabernat do you have an application that we could use to drive this?

Nothing comes to mind immediately. My data are unfortunately quite dense! 😜

[olgabot]

In case you're still looking for an application, gene expression from single cells (see data/00_original/GSM162679$i_P14Retina_$j.digital_expression.txt.gz) is very sparse due to high gene dropout. The shape is expression.shape (49300, 24760) and it's mostly zeros or nans. A plain csv from this data was 2.5 gigs, which gzipped to 300 megs.

Here is an example of using xarray to combine these files but my kernel keeps dying when I do ds.to_netcdf() :(

Hope this is a good example for sparse arrays!

[rth]

do you have an application that we could use to drive this?

Other examples where labeled sparse arrays would be useful are,

• one-hot encoding that are widely used in machine learning.
• tokenizing textual data produces large sparse matrices where the column labels correspond to the vocabulary, while row labels correspond to document ids. Here is a minimal example using scikit-learn,

  import os.path
  from sklearn.datasets import fetch_20newsgroups
  from sklearn.feature_extraction.text import CountVectorizer
  
  ds = fetch_20newsgroups()
  vect = CountVectorizer()
  X = vect.fit_transform(ds.data)
  print(X)  # Extracted tokens
  # Returns:
  # <11314x130107 sparse matrix of type '<class 'numpy.int64'>'
  #	with 1787565 stored elements in Compressed Sparse Row format>
  
  column_labels = vect.get_feature_names()
  print(np.asarray(column_labels))
  # Returns:
  # array(['00', '000', '0000', ..., 'íålittin', 'ñaustin', 'ýé'],   dtype='<U180')
  
  row_labels = [int(os.path.split(el)[1]) for el in ds.filenames]
  print(np.asarray(row_labels))
  # Returns:
  # array([102994,  51861,  51879, ...,  60695,  38319, 104440])

[rabernat]

Sparse Xarray DataArrays would be useful for the linear regridding operations discussed in JiaweiZhuang/xESMF#3.

[lbybee]

I'm interested to see if there have been any developments on this. I currently have an application where I'm working with multiple dask arrays, some of which are sparse (text data). It'd be worth my time to move my project to xarray, so I'm be interested in contributing something here if there is a need.

[Hoeze]

I'd know a project which could make perfect use of xarray, if it would support sparse tensors:
https://github.com/theislab/anndata

Currently I have to work with both xarray and anndata to store counts in sparse arrays separate from other depending data which is a little bit annoying :)

[shoyer]

See also: #1938

The major challenge now is the dispatching mechanism, which hopefully http://www.numpy.org/neps/nep-0018-array-function-protocol.html will solve.