Earthdata access via obstore

[kylebarron]

👋 Hi all,

I've been working on obstore lately. It's a fast, easy-to-use, async-capable interface for S3, GCS, and Azure storage, with a single API to access each.

As of obstore v0.5, it has the concept of "credential providers", which are Python callbacks that customize credentials for a store. Obstore v0.5 ships with a NasaEarthdataCredentialProvider, which allows users to get in-region direct S3 access to Earthdata in a few lines of code:

from obstore.auth.earthdata import NasaEarthdataCredentialProvider
from obstore.store import S3Store

credential_provider = NasaEarthdataCredentialProvider(username="...", password="...")
store = S3Store("landsat-c2", credential_provider=credential_provider)
# use `store` to perform HEAD, LIST, GET requests on the bucket
# store.list()

If you're curious you can look at the implementation.

Aside from being really simple to use, this also provides automatic token refresh. Even if the end user has code that runs for longer than an hour, the end user never needs to think about tokens or when they expire. Obstore will automatically refresh credentials when making requests.

If I'm reading this correctly, get_fsspec_https_session and similar APIs will not automatically refresh tokens.

This discussion is primarily to make the earthaccess community aware of this obstore integration. There's also a couple things we could add on the obstore side to simplify usage; I see this library also supports netrc and env var support, and we could add that behavior to obstore as well.

Any feedback is welcome!

[betolink]

This integration would be great to have! and the refresheable tokens has been on TODO for a long time. I'll take a look at the NasaEarthdataCredentialProvider, a couple questions:\

• I wonder if we could use only bearer tokens instead of user/password for the credentials.
• Will this be a drop-in replacement for fsspec file-like objects? e.g. pass the store to Xarray open()

Thanks for connecting the dots @kylebarron!

[betolink]

Bearer tokens: NASA EDL accounts can use bearer tokens, see: https://urs.earthdata.nasa.gov/documentation/for_users/user_token they are accepted by most APIs including the endpoints to get S3 temporary credentials(almost sure but will confirm tomorrow).
Now, each collection can have 2 types of endpoints to get credentials, an endpoint that belongs to a DAAC or one that belongs to a mission. Most NASA datasets are grouped by DAAC, so HLS for example will use credentials given by LPDAAC: https://data.lpdaac.earthdatacloud.nasa.gov/s3credentials however some datasets will use their own endpoint e.g. the SWOT satellite:
https://archive.swot.podaac.earthdata.nasa.gov/s3credentials

This information is on the metadata returned by CMR. Ideally earthaccess will use the metadata to infer which endpoint to use and configure Obstore, something like:

credential_provider = NasaEarthdataCredentialProvider(token="", auth_endpoint="") # depending on what dataset we are reading.

fsspec-like behavior: most of the data in NASA's CMR is either HDF5 or Netcdf, currently earthaccess uses fsspec to open a them as file-like objects and pass them to Xarray, this has pros and cons, like no refreashable tokens and it can be really slow(the default cache in fsspec doesn't help).

I wonder if Xarray would support memoryview objects, maybe @TomNicholas knows if this could be an option. Another thing it would be interesting to look at, is how Obstore compares to the IceChunk driver when both are used to read Zarr from S3.

Looking forward to implement a workflow with earthaccess+Obstore!

[kylebarron]

Interesting. Are the bearer tokens supported in earthaccess? If so, can you point to the code that handles that for getting s3 temporary credentials? (fwiw the code in obstore here is pure python, so it's easy to read and edit)

Also fwiw I figured I might as well implement the earthdata support in obstore because it looked simple, but if there's a long tail of edge cases like which DAAC to send requests to, maybe that code could live here in earthaccess (I think this credential provider API in obstore is stable, and it can be implemented by third party users or libraries)

Ok, good to know it's mostly hdf5 and netcdf. Obstore has no caching by default, even its fsspec integration.

The fsspec integration aims to be fully compatible with other fsspec implementations (even though it can be tricky sometimes to understand exactly what fsspec behavior is). Obstore's fsspec integration returns pure Python bytes objects (and always incurs the memory copy from Rust into Python). So it should work out of the box.

how Obstore compares to the IceChunk driver when both are used to read Zarr from S3

yes, this is planned. I expect them to be roughly the same performance.

[chuckwondo]

@kylebarron, thanks for all of your work on obstore, and for working on the issue of auto-refreshing credentials! This is much appreciated.

As @betolink pointed out, the s3 credentials endpoint is not fixed, thus, you should not hard-code a specific one, as you have done here: https://github.com/developmentseed/obstore/blob/e905dfc91e3df3d10796edcbdd5eb3ffe8de102d/obstore/python/obstore/auth/earthdata.py#L15

Rather, as Luis's example shows, the endpoint should be an argument to the provider initializer rather than a constant.

I also suggest not using a session object for 2 major reasons:

1. You're using the session in an insecure manner because you're leaking credentials (i.e., supplying the creds on requests that are not only to EarthData Login)
2. The sessions are resource leaks because they are not reliably/consistently closed (because it's not clear when or if the store object using them will ever close the provider, which would close the session).

In other words, it appears that you're using a session to simplify the code, but that simplification not only leaks resources, but also creates a security hole.

Regarding the use of a bearer token, you don't have to worry about how that is obtained, as obtaining a token is not code that would live in obstore. You can think of it like an API token that a user must obtain, and then must supply to the library (perhaps via an env var, but not an env var that obstore should be aware of in this case).

I started to add more details here, but this is perhaps not the right place for obstore implementation details, so feel free to ping me about how to address these issues, as I have worked quite a bit with various ways of obtaining the S3 creds via EDL.

[kylebarron]

As @betolink pointed out, the s3 credentials endpoint is not fixed, thus, you should not hard-code a specific one

Ok that's fine, we can set that as the default and let users override it?

I also suggest not using a session object for 2 major reasons:

That seems reasonable. I don't know the requests API that well and don't actually know how to provide them to a request without going through the session object. Would you be able to provide pointers or even make a PR?

Regarding the use of a bearer token, you don't have to worry about how that is obtained, as obtaining a token is not code that would live in obstore.

Ok, that's fine, but I do have to understand how the requests change when a user passes in this bearer token.

I created developmentseed/obstore#392 if you want to chime in there.

[chuckwondo]

As @betolink pointed out, the s3 credentials endpoint is not fixed, thus, you should not hard-code a specific one

Ok that's fine, we can set that as the default and let users override it?

No, there is no sensible default value here.

I also suggest not using a session object for 2 major reasons:

That seems reasonable. I don't know the requests API that well and don't actually know how to provide them to a request without going through the session object. Would you be able to provide pointers or even make a PR?

Yes, I'd be happy to provide more details, or even a PR. I'll reach out to you if I need a leg up on the PR.

Regarding the use of a bearer token, you don't have to worry about how that is obtained, as obtaining a token is not code that would live in obstore.

Ok, that's fine, but I do have to understand how the requests change when a user passes in this bearer token.

I created developmentseed/obstore#392 if you want to chime in there.

Perfect. I'll chime in over there.