Eliminate flaky "in-region" detection

[chuckwondo]

There are numerous issues surrounding use of S3 vs HTTPS URLs for reading/downloading files via earthaccess, particularly in relation to code that attempts to automatically detect whether or not is is running "in-region" (in us-west-2, specifically). Here are few (and I may have missed some):

• Document why signed S3 URLs might be giving 400s when called from inside us-west-2 #188
• Consolidate access= and in_region= kwargs? #327
• More frictionless S3 direct access #431
• Change us-west-2 check, as only working in EC2 instances #444
• download returns inconsistent types (Paths or strs) depending on in_region #595

Unfortunately, it seems that in-region detection cannot be done reliably, and causes headaches. Specifically, there are cases where the code fails to detect that is it running in-region, and thus erroneously switches to using HTTPS URLs rather than S3 URLs.

I'm starting this discussion to attempt to consolidate input for what I propose we do:

• drop the flaky in-region detection code
• drop the "access" string parameter
• default to S3 and fallback to HTTPS in the face of S3 errors
• provide a keyword-only parameter to allow users to customize this behavior
• provide out-of-the-box mechanisms: HTTPS only, S3 only, S3 w/HTTPS fallback (default)

Of course, we can vary this, but that's the general idea I'm putting forward.

At a high level, I'm thinking that we might do something along the lines of adding a keyword-only parameter to earthaccess.open and earthaccess.download, which would accept a "handler" for reading/downloading files. The "handler" would default to "S3 w/HTTPS fallback".

[Alex-Lewandowski]

I considered dropping access in my attempt to implement most of what you describe (#880). I left it in because I thought that someone may have a use case where they do not want to attempt direct access at all, and skip directly to HTTPS. I wasn't sure, but I erred on the side of leaving it in to provide more control.

[chuckwondo]

Right, that's what I'm describing as a "handler" (for lack of a better term at this point).

Rather than passing and arbitrary string value for access, the idea would be to allow the user to pass a "handler" (likely a function of a certain signature). The default handler would attempt to use S3 and fallback to HTTPS in the face of S3 errors.

However, if the user wants to use HTTPS only, they could specify the HTTPS-only handler. Conversely, if only S3 is desired (without HTTPS fallback), the user could specify the S3-only handler.

All 3 of those handlers we would supply out of the box, using the "S3 w/HTTPS fallback" by default.

Further, since the "handler" parameter would accept any function (of some signature yet to be defined), users can supply their own handler, if none of the 3 provided out of the box suffice.

[itcarroll]

I'm sorry that you got shut down on your first earthaccess PR, @Alex-Lewandowski. To me it looked on the right track, greatly in-need, and just a bit short on preparatory discussion (although perhaps that happened IRL!).

[Alex-Lewandowski]

Oh, that is a cool idea and puts control in the users' hands if none of the out of the box solutions work for them.

To make sure I understand, the handler functions would contain the code that accesses the data? So, for example, if someone wanted to provide a handler that leveraged zarr, kerchunk, etc..., they could write their own and pass it?

[chuckwondo]

That's an interesting thought. I don't think we'll be seeing such data accessed through earthaccess, but at the moment I was thinking mostly along the lines of earthaccess.download, so I'd have to look further into how we might handle earthaccess.open in this context.

[itcarroll]

Discussions around the cloud_hosted parameter (#844, in which @jhkennedy also mentions #526 and #839) are relevant to the big picture here too, probably. So here's additional background to consider:

• Refactor how we determine if granules are cloud-hosted #844
• [proposed enhancement] AWS us-west-2 checking method #231

[itcarroll]

The reason I find the cloud_hosted search parameter relevant: this is when the user indicates whether they anticipate running in_region or not. I think a user would find it intuitive for earthaccess to store that input and make subsequent open or download requests use that intent by default (and then raise or handle as a fallback).