[Proposal] Expose Variable without Pandas dependency

[jhamman]

This issue proposes exposing Xarray's Variable class as a stand-alone array class with named axes (dims) and arbitrary metadata (attrs) but without coordinates (indexes). Yes, this already exists but the Variable class in currently inseparable from our Pandas dependency, despite not utilizing any of its functionality. What would this entail?

The biggest change would be in making Pandas an optional dependency and isolating any imports. This change could be confined to the Variable object or could be propagated further as the Explicit Indexes work proceeds (#1603).

Why?

Within Xarray, the Variable class is a vital building block for many of our internal data structures. Recently, the utility of a simple array with named dimensions has been highlighted by a few potential user communities:

• Scikit-learn: SLEP 8: Propagating feature names scikit-learn/enhancement_proposals#18
• PyTorch: (https://pytorch.org/tutorials/intermediate/named_tensor_tutorial.html, http://nlp.seas.harvard.edu/NamedTensor)

An example from the above linked SLEP as to why users may not want Pandas a dependency in Xarray:

@amueller: ...If we go this route, I think we need to make xarray, and therefore pandas, a mandatory dependency...
...
@adrinjalali: ...And we still do have the option of making a NamedArray. xarray uses the pandas' index classes for the indexing and stuff, which is something we really don't need...

Since we already have a class developed that meets these applications' use cases, its seems only prudent to evaluate the feasibility in exposing the Variable as a low-level api object.

In conclusion, I'm not sure this is currently worth the effort but its probably worth exploring at this point.

[amueller]

FYI the conversation on sklearn is far from resolved, and at this point I think the added pandas dependency is not what will keep us from using xarray. I think right now we're most concerned about sparse data representations (and I was considering asking you folks if you'd support scipy.sparse ;)

[jhamman]

Thanks @amueller! You're not the only group that has voiced interest in an index-free labeled array so I think its still worth discussing.

I think right now we're most concerned about sparse data representations (and I was considering asking you folks if you'd support scipy.sparse ;

We have recently added support for pydata/sparse. We need to document this better (#3484). This work is not 100% complete (#3213) and is part of Xarray's larger goal of supporting duck arrays. Please chime in on one of those issues so we can go a bit deeper.

[effigies]

To add another use case, the NiBabel package has considered how to label axes (https://github.com/nipy/nibabel/wiki/BIAP6, nipy/nibabel#412), but it's fallen by the wayside. We considered xarray when it was still xray, but the pandas dependency has always been a sticking point. This is partially due to a desire to keep dependencies minimal, and partly due to the size of pandas causing significant overhead at import time for what is a relatively small component.

If it's useful to go further into our use case, we can, but this is just to put in a vote for making pandas an optional dependency, if possible.

[matthew-brett]

Just to add - we at Nibabel are very interested in adding labelled arrays with Xarray, but for us, the Pandas dependency is a serious problem. We're a base library for reading brain imaging formats, and we sit at the bottom of several imaging stacks, so it is very important to us that we don't introduce heavy dependencies - because we pass these on to all the libraries that depend on us. We've looked enviously at Xarray for a while, but the Pandas dependency is a serious-enough problem that we've held off from using it. Just for example, the Pandas dependency of Xarray was the reason that I was working on Datarray, to see if we could use that instead (we couldn't).

[max-sixty]

What would the options be for moving this forward?

• Try / Except import blocks, like we do for other libraries
• A separate package (could be in the same repo though)
• ?

[dcherian]

Another possible outcome is that pandas becomes optional as part of the indexing refactor, and we expose Variable as public API.

BUT @shoyer has been consistent against making Variable public, so maybe the best option is to fork it out to xarray-contrib/variable?

EDIT: The git log for variable.py shows really minor changes over the past year or two (https://github.com/pydata/xarray/commits/master/xarray/core/variable.py) — mostly adding new method (e.g. pad & quantile) and some NEP-18 work. So it seems pretty stable and perhaps not much effort to exchange future patches between xarray and the fork.

[shoyer]

BUT @shoyer has been consistent against making Variable public, so maybe the best option is to fork it out to xarray-contrib/variable?

Variable is a public API, which just don't encourage it for (most) users.

I think it would be relatively straightforward to expose xarray.Variable without the pandas dependency. We currently use pandas a little bit for input normalization in Variable and methods like isnull(), but we could likely add slower fallback option (or just not expose these features unless pandas is installed).

[shoyer]

Thinking about this a little more, I would lean towards the separate package that xarray could depend upon. Perhaps we could all this xarr to denote a minimal version of xarray, unless that's too punny?

[andersy005]

Perhaps we could all this xarr to denote a minimal version of xarray, unless that's too punny?

I've heard folks referring to this proposal as xarray-lite, and I think it'd be a good name, too.

[matthew-brett]

We were just talking about this over at https://github.com/nipy/nibabel - because we are about commit ourselves to an array-axis-labelling API. Is xarray-lite on the near or the distant horizon? Should we wait, to make our decisions? Mentioning @effigies because he reminded me about this thread.

[shoyer]

We wrote a proposal to build an "xarray-lite" as part of a recent CZI application. We're still waiting to hear back, but if that's funded we'd definitely love to work with you on this.

…

On Fri, Jul 9, 2021 at 12:16 PM Matthew Brett ***@***.***> wrote: We were just talking about this over at https://github.com/nipy/nibabel - because we are about commit ourselves to an array-axis-labelling API. Is xarray-lite on the near or the distant horizon? Should we wait, to make our decisions? Mentioning @effigies <https://github.com/effigies> because he reminded me about this thread. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3981 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJJFVRJCOTD7TEROEW5RHDTW5DHHANCNFSM4MLCDJQQ> .