Support for multidimensional dtypes

[alexbw]

With 0.11 out, Pandas supports more dtypes than before, which is very useful to us science folks. However, some data is intrinsically multi-dimensional, high enough dimensional so that using labels on columns is impractical (for instance, images).

I understand DataFrames or Panels are usually the recommended panacea for this problem. This works if the datatype doesn't have any annotation. For instance, for each frame of a video, I have electrophysiology traces, timestamps, and environmental variables measured.

I have a working solution where I explicitly separate out the non-scalar data from the scalar data. I use Pandas exclusively for the scalar data, and then a dictionary of multi-D arrays for the array data.

What is the work and overhead involved in supporting multi-D data types? I would love to keep my entire ecosystem in Pandas, as it's much faster and richer than just NumPy data wrangling.

See below for the code that I hope is possible to run, with fixes.

If you can point me to a place in the codebase where I can tinker, that would also be much appreciated.

import pandas as pd
mydtype=np.dtype('(3,3)f4')
pd.Series(np.zeros(3,), dtype=mydtype)

Exception: Data must be 1-dimensional

My suspicion is that you're going for more complex instead of simplifying.
However, there is the NDpanel if you want it, and you can do:

In [26]: import pandas as pd
    ...: f=lambda : np.random.random((3,3))
    ...: s=pd.Series([f() for i in range(10)], dtype='O')

In [27]: s.iloc[0]
Out[27]: 
array([[ 0.90986552,  0.30234529,  0.98927833],
       [ 0.40467537,  0.17912555,  0.06101674],
       [ 0.6623446 ,  0.69192764,  0.39398118]])

[jreback]

@alexbw but keep in mind, this is really not efficient (in numpy or pandas), as this is now an object array, which is not able to vectorize things. Your are much better off keeping your scalar data from your images. (I believe we went thru an exercise in saving these to/from HDF a while ago).

[alexbw]

Yep, your suggestions are now in production here, and it's working fine
keeping scalars and higher-D arrays separate.
Just checking in to see if any of the dtype improvements might make my
use-case a little more feasible. Doesn't seem so.

On Wed, Apr 24, 2013 at 10:32 AM, jreback notifications@github.com wrote:

@alexbw https://github.com/alexbw but keep in mind, this is really not
efficient (in numpy or pandas), as this is now an object array, which is
not able to vectorize things. Your are much better off keeping your scalar
data from your images. (I believe we went thru an exercise in saving these
to/from HDF a while ago).

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3443#issuecomment-16934447
.

[jreback]

I don't know if i pointed this out before, this might work for you: http://pandas.pydata.org/pandas-docs/dev/dsintro.html#panel4d-experimental (again the data doesn't have to be homogenous in dtype, but homogeneous in shape, which may or may not help)

[alexbw]

Each data stream (image, velocity, temperature) is homogeneous within
itself, but they're all different sizes. That's the clincher, and it seems
like it's not on the horizon to be supported here. Blaze seems to be going
in this direction, supporting heterogeneous data shapes.

On Wed, Apr 24, 2013 at 10:52 AM, jreback notifications@github.com wrote:

I don't know if i pointed this out before, this might work for you:
http://pandas.pydata.org/pandas-docs/dev/dsintro.html#panel4d-experimental(again the data doesn't have to be homogenous in dtype, but homogeneous in
shape, which may or may not help)

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3443#issuecomment-16935832
.

[jreback]

heteregoenous data shapes are non-trivial. Blaze does seem headed in that direction, but not sure when will happen.

[alexbw]

Ok. And here, by "non-trivial", do you mean that Pandas has no plans to
support a feature like this?

On Wed, Apr 24, 2013 at 11:03 AM, jreback notifications@github.com wrote:

heteregoenous data shapes are non-trivial. Blaze does seem headed in that
direction, but not sure when will happen.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3443#issuecomment-16936562
.

[jreback]

what I mean is to be efficient about ti you would have to have a structure that is essentially a dictionary of 'stuff', where stuff could be heteregenous shaped. This is really best done in a class, and it application specific. You can do it with a frame/panel whatever as @y-p shows above, but it is not 'effeicient', in that numpy holds onto the dtype as 'object'.

When I say efficient I mean that you can move operations down to a lower level (c-level) in order to do things. I am not even sure Blaze will do this, its really quite specific (they are about supporting chunks and operating on those, but those chunks are actually the same shapes, except for 1 dim where the chunking occurs).

There is a tradeoff in code complexity & runtime-efficiency & generality. You basically have to choose where you are on that 3-d surface. Pandas has moderate complexity & generality and high runtime-efficiency. I would say numpy is lower complexity, lower generality with a simiilar runtime-efficiency. I would guess that Blaze is going to be more complex, higher efficiiency (in cases of out-of-core datasets) and about the same generality as numpy (as they are aiming to replace numpy)

So even if someone had the urge to create what you are doing, they are going to have to create a new structure to hold it.

It comes to do what are your bottlenecks, maybe getting more specific will help

[cpcloud]

@jreback Just out of curiosity what is the long term goal of pandas in this vein? If blaze is to replace numpy will pandas diverge from numpy altogether, or will it use blaze as the backend? I see talk about making Series ind. of ndarray in the near future for pkl support and other reasons.

[jreback]

I don't see pandas incompatible via blaze at all. my understanding (and just from reading the blog). is. that blaze is supposed to be the next gen numpy. I think their API will necessarily be very similar to what it is now, and thus be pretty transparent to pandas.

my concerns now are availability and esp compatibility of their product, as it has a fairly complicated build scheme.
In addition they seem to want to incorporate index like features (kind of like labelled arrays). if they do great. I am sure pandas might use some of the infrastructure

I think pandas fills a somewhat higher level view of data right now (and will continue to do so)

as far as your specific comments, I have pushed a PR to decouple series from ndarray (index also needs this addessed). thus pandas will be somewhat easier to modify its backend w/o front end (API) visibility. (so this is a good thing)

supporting arbitrary dshapes within pandas existing objects IMHO is not that useful right now

[jreback]

@wesm chime in?

[alexbw]

Any chance at all of this seeing some love?

[cpcloud]

I think something like a RelationalDataFrame or RelationalSomethingOrOther would be useful here. The idea would be to have a collection of NDFrames that share one or more common axes (Index objects). That way you could keep things separate without the complexity of nd-dtypes. Of course, you now have the complexity of an in-memory relational database.

In this case you could have an object where all objects share the "video frame axis" possibly more if needed.

maybe it could have a query method similar to 0.13 query method that does something similar except the namespace would be expanded to include all the objects on the RelationalThingaMaBob.

[jreback]

@alexbw this is pretty non-trivial, mainly because numpy doesn't support it (ATM), though Blaze is suppopsed to.

@cpcloud has a nice idea, essentially an object to hold DataFrames that has axes that are alignable (think of this as a Panel like), but you could have a mixture too, e.g. for each object only align on certain axes

[cpcloud]

I could maybe see this being implemented by a generalization of BlockManager where the blocks themselves are pandas objects.