Virtual dataset chat 2025/09/09

[danielfromearth]

@TomNicholas, @betolink, and I discussed the technical architecture, and where things are heading, with virtual datasets – especially providing references to NASA Earth observation data that are stored in non-cloud-optimized formats. The conversation covered:

• relationship between storage approaches: Icechunk store to Array Lake is sorta like git repo to GitHub
• various lazy loading approaches available in the ecosystem: xarray indexes, dask, virtual datasets
• how VirtualiZarr handles coordinate variables and manifest serialization: loadable_variables become inlined references (serialized data written instead of manifest), and VirtualiZarr defaults to loading 1D coordinate variables
• considered future enhancements: E.g., could expand chunk manifest tuples (path, offset, length) with additional metadata fields (such as CRS, other geo information) to create more database-like functionality
• traced the workflow: from manifest_store (from Parsers) -> virtual dataset (can use xarray for concatenating) -> serialized references