Consider not eagerly loading all pages of collections

[hrodmn]

When there are hundreds or thousands of collections, we probably don't want to load them all! One example of a catalog that overwhelm the stac-map site is https://cmr.earthdata.nasa.gov/stac/ALL/

Should we apply some kind of ceiling on the number of pages that get fetched?

[abbottry]

https://developmentseed.org/stac-map/?href=https://cmr.earthdata.nasa.gov/stac/POCLOUD

We've struggled with this from the NASA side, how many should we return per page? Should we cap it and expect the user to page? Feels like there is an opportunity here for an extension that would support returning some sort of vitals to inform pages like this, some minimally useful information for pages to represent this data. How many collections are there, what is the temporal and spatial extent, etc.

[gadomski]

how many should we return per page?

I don't think there's a single answer here, as it depends on how much information is in each collection; e.g. if you have a lot of item-assets, your collections might be larger and therefore have a smaller "optimal" page size.

Should we cap it and expect the user to page?

I don't have a great sense of client support for collection pagination, but for something like your example it seems quite reasonable. It's in the spec, after all.

Feels like there is an opportunity here for an extension that would support returning some sort of vitals to inform pages like this, some minimally useful information for pages to represent this data.

This exists for item search in fields extension. It seems reasonable to extend it to collection pagination (and search).

[github-actions]

🎉 This issue has been resolved in version 0.8.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

[gadomski]

Update, collection-search does call out that fields is possible: https://github.com/stac-api-extensions/collection-search?tab=readme-ov-file#fields. Not sure how many folks are implementing that.