Zarr v3 Support and conda-forge versus pypi dependencies

[scottyhq]

Moving some discussion from #877 (comment) to it's own issue.

Currently the conda-forge feedstock requires Zarr<3 and Virtualizarr>=1.2
https://github.com/conda-forge/earthdata-feedstock/blob/86aaa60fcacaf8a2ae3d8ce608f583b45a7d5fd1/recipe/meta.yaml#L39-L40

    - zarr >=2.12.0, <3.0.0a
    - virtualizarr >=1.2.0

But virtualizarr>=1.3.3 will pin zarr>=3, which I think is going to cause some confusion.
zarr-developers/VirtualiZarr#392
https://github.com/zarr-developers/VirtualiZarr/compare/main..develop#diff-50c86b7ed8ac2cf95bd48334961bf0530cdc77b5a56f852c5c61b89d735fd711

From the pyproject.toml it looks like virtualizarr is actually an optional dependency, so maybe it should just be removed from the conda-forge recipe?

earthaccess/pyproject.toml

Lines 57 to 64 in 18e947e

	[project.optional-dependencies]
	kerchunk = [
	"kerchunk",
	"dask",
	"h5py >=3.8.0",
	"h5netcdf",
	"xarray",
	"zarr >=2.12.0, <3.0.0a",

Alternatively, not sure what work is needed to support Zarr v3 here?

[weiji14]

Thanks @scottyhq for opening this issue to track!

But virtualizarr>=1.3.3 will pin zarr>=3, which I think is going to cause some confusion.

I've checked with Ayush in this Zulip thread, and got his blessing to pin to zarr>=3 (i.e. drop zarr-python=2 compatibility)

From the pyproject.toml it looks like virtualizarr is actually an optional dependency, so maybe it should just be removed from the conda-forge recipe?

Yes, it's usually best to not have opitonal dependencies in a conda recipe (conda/conda#7502), but that should change soon once this CEP is merged (see also https://prefix.dev/blog/ceps_2025#optional-dependencies-conditional-dependencies-and-flags).

Alternatively, not sure what work is needed to support Zarr v3 here?

Working on this at #967. @betolink has kindly given me contributor access during the earthaccess hackathon today, and hopefully we can get this in for the next release.

[mfisher87]

Amazing! I can't believe we're getting optional dependencies soon. 🎉 🎊 🙌 Thanks for sharing this!