Could we define "views" of `Dataset`s?

[heliosdrm]

The only way of taking a fragment of a Dataset is creating another one. This is not optimal, and it might be good to have another subtype of AbstractDataset, whose contents are a view to the data of another Dataset.

The easiest would be to define:

Base.view(d::AbstractDataset, i) = view(d.data, i)

and then some type like DatasetView <: AbstractDataset whose data is a SubArray{<:SArray}, such that one could do:

d = Dataset(x)
dv = Dataset(@view d[xx])

(Notice that this code actually works right now, but it produces a copy of the contents before creating dv.)

Such dv::DatasetView should behave as if I had written dv = d[xx], but that way we would save the superfluous allocation.

I acknowledge that it would be nicer if the interface is directly dv = @view d, but I'm not sure if that would be right. Base.view (and thus the macro @view) is meant to produce a SubArray, which is defined as a container encoding a "view" of a parent AbstractArray. So, although technically it might be feasible to write a method of Base.view that takes and returns an AbstractDataset, for me that would only feel right if we define AbstractDataset <: AbstractArray -- which maybe makes sense too, but might need some deeper thought to ensure that the API of AbstractDataset is fully consistent with what is expected from AbstractArrays (or at least AbstractVectors).

[Datseris]

The best way is I think to change the type parameterization of Dataset from Dataste{D, T} to Dataset{D, T, X} with X the type of the field .data. This will allow us to have views as data while the rest of the interface would operate without a problem. The only thing to keep track of is that the constructor of the Dataset which has to infer properly D, T from the data. This is always simple with the data are AbstractArray{<:SVector{D, T}}, which View is.

[Datseris]

And then we can extend @view dataset = Dataset(@view dataset.data).

[heliosdrm]

And then we can extend @view dataset = Dataset(@view dataset.data).

Let aside whether views are implemented with a new parametrization of Dataset, or with a new subtype of AbstractDataset, there is the other question that I asked:

although technically it might be feasible to write a method of Base.view that takes and returns an AbstractDataset, for me that would only feel right if we define AbstractDataset <: AbstractArray.

What do you think about that?

[Datseris]

although technically it might be feasible to write a method of Base.view that takes and returns an AbstractDataset, for me that would only feel right if we define AbstractDataset <: AbstractArray.

This sounds arbitrary to me. Since when you can extend methods only if such subtyping relations are satisfied? There is no requirement to have an AbstractArray to have an implementation of view. Also, at the moment length for datasets does not return prod(size(dataset)) which is what happens for AbstractArrays, and I'd very much like to keep it that way, since length returns the amount of data points, while size returns the other sensible thing, which is (time length , dimensionality).

So, my counter question is, why change things? What is the benefit of trying to do AbstractDataset <: AbstractArray? It isn't "being allowed to extend view", as this is allowed already.

[heliosdrm]

There is no requirement to have an AbstractArray to have an implementation of view.

That's precisely my question. The Julia docs tell about view returning SubArray objects, and this page tells that "Julia's SubArray type is a container encoding a "view" of a parent AbstractArray". I know that this is no hard restriction: we may define a method of view dispatching on Dataset and returning something different than a SubArray, with no penalty. But my question is not about whether that is allowed, but whether it is good practice.

In other words: creating an interface for Base functions that return different types than those defined in the Base docstring, is it good practice or not? I assume that at least, the docstring of Base.view should be extended too, right?

[Datseris]

The documentation of the Julia docs is about the act of the function view on AbstractArray objects. Here this is not the situation that we are discussing, and thus for me it still falls into the realm of good practice if view(::Dataset) does not return an AbstractArray.

We could extend the docs of Base.view, or we could instead just simply add view as one of the supported functions in the docstring of Dataset. All other functions that Dataset extends like eachcol do not necessarily get their own docstring. Perhaps it is better to just list all of them in the Dataset docstring?

[heliosdrm]

Right. I have been looking into other well-established packages, e.g. DataFrames.jl to see how they deal with views.

The function view dispatches on AbstractDataFrame producing a SubDataFrame, mimicking the SubArray produced by viewing an AbstractArray, although AbstractDataFrame is not a subtype of AbstractArray at all. I think that this situation is a good parallel. So what do you think of view(::Dataset, i) producing a SubDataset, instead of re-parametrizing Dataset?

[Datseris]

Sure that is fine for me. Should be easy to code as well, as many functions are implemented for AbstractDataset.

The only problem is that in the rest of the library I've been defining methods for ::Dataset directly, haven't really consider the possibility of a new type of dataset. But in any case this is a good change, also to reduce my own sloppyness. I have to go through the rest of DynamicalSystems.jl and see what dispatches on Dataset and make it AbstractDataset.

So the only difference of SubDataset is that itsfield is a

data::SubArray{correct parametrizaition}(view(Vector{SVector})

[heliosdrm]

Shall I write a PR for that?

[Datseris]

sure