Dataset/DataArray to_dataframe() dimensions order mismatch.

[Thomas-Z]

Code Sample

import xarray as xr
import numpy as np

data = xr.DataArray(np.random.randn(2, 3), coords={'x': ['a', 'b']}, dims=('y', 'x'))
ds = xr.Dataset({'foo': data})

# Applied on the Dataset
ds.to_dataframe()

#          foo
#x y          
#a 0  0.348519
#  1 -0.322634
#  2 -0.683181
#b 0  0.197501
#  1  0.504810
#  2 -1.871626

# Applied to the DataArray
ds['foo'].to_dataframe()

#          foo
#y x          
#0 a  0.348519
#  b  0.197501
#1 a -0.322634
#  b  0.504810
#2 a -0.683181
#  b -1.871626

Problem description

The to_dataframe method applied to a DataArray will respect the dimensions order whereas the same method applied to a Dataset will use an alphabetically sorted order.

In both situation to_dataframe calls _to_dataframe() with an argument.
The DataArray uses an OrderedDict but the Dataset uses self.dims (which is a SortedKeyDict) as argument.

Output of xr.show_versions()

INSTALLED VERSIONS ------------------ commit: None python: 3.5.2.final.0 python-bits: 64 OS: Linux OS-release: 4.15.0-23-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: fr_FR.UTF-8 LOCALE: fr_FR.UTF-8 xarray: 0.10.8 pandas: 0.23.4 numpy: 1.14.5 scipy: 1.1.0 netCDF4: 1.4.0 h5netcdf: None h5py: 2.8.0 Nio: None zarr: 2.2.0 bottleneck: None cyordereddict: None dask: 0.18.2 distributed: 1.22.1 matplotlib: 2.2.2 cartopy: None seaborn: None setuptools: 40.0.0 pip: 18.0 conda: None pytest: 3.7.1 IPython: 6.5.0 sphinx: None

[shoyer]

How would you suggest choosing the order of dimensions in Dataset.to_dataframe?

The current choice has the advantage that it matches self.dims and is invariant to the order in which variables were added into the dataset. As for the order of iteration for Dataset.dims, at one point early in xarray, I think we had dims as either a dict or OrderedDict, but for both of those the iteration order is somewhat arbitrary and can differ for two Dataset objects that otherwise look identical. At least SortedKeyDict is ocnsistent.

If we want to change this (which could be OK), I would try to change the iteration order of self.dims instead so it remains consistent. Potentially we could make Dataset.dims iterate over dimensions in order of their appearance on a variable, data variables first followed by coordinates. This would potentially entail the potential need to iterate over all variables to iterate over dimensions, but I think that would probably be acceptable.

[Thomas-Z]

I was kind of expecting to get the order shown when looking at the dims property but i understand your point and it makes sense.

Two things are still bothering me though:

• ds['foo'].to_dataframe() and ds[['foo']].to_dataframe() might have different results
• if we want to have a specific order we have to apply reorder_levels and sort_index which can be quite expensive.

For the first point I don't think anything should be done, it's a special case and even if it could be easily tested it might be ugly.
For the second point I would not change anything to the way the order is defined now, it's consistent and easily predictable.
Instead I would add an additional optional parameter to to_dataframe() (the one from _to_dataframe(ordered_dims)) to allow the user to get the order he wants.

[stale]

In order to maintain a list of currently relevant issues, we mark issues as stale after a period of inactivity

If this issue remains relevant, please comment here or remove the stale label; otherwise it will be marked as closed automatically

[shoyer]

I opened a new issue for explicitly setting dimension order: #4331

This would definitely be a welcome improvement!