[BUG] search_data may return granules that do not exist anymore

[bastien-k]

Is this issue already tracked somewhere, or is this a new report?

• I've reviewed existing issues and couldn't find a duplicate for this problem.

Current Behavior

I understand that NRT products such as 10.5067/MODIS/MYD11_L2.NRT.061 are deleted on a rolling basis, with only the last 7 days available on the server.

I noticed that the search_data endpoint could still show granules that are no longer on the server.
This will make the download function fail, raising a 404 error.

Expected Behavior

The search function should not return granules that are no longer on the server, and the download function should provide an option to ignore 404 errors.

Steps To Reproduce

This is time dependent, but the following script could demonstrate it:

import earthaccess
import requests

doi = "10.5067/MODIS/MYD11_L2.NRT.061"
temporal = ("2025-11-07T00:00", "2025-11-07T23:59")
bounds = (-122.16838261664851, 38.3103369706418, -121.5725847324255, 38.77823385016004)

granules = earthaccess.search_data(doi=doi, temporal=temporal, bounding_box=bounds)

for granule in granules:
    url = granule.data_links()[0]
    print(url)
    resp = requests.head(url, allow_redirects=True, timeout=5)
    print(resp.status_code)

Result:

https://nrt3.modaps.eosdis.nasa.gov/api/v2/content/archives/allData/61/MYD11_L2/2025/311/MYD11_L2.A2025311.1225.061.2025311142708.NRT.hdf
404
https://nrt3.modaps.eosdis.nasa.gov/api/v2/content/archives/allData/61/MYD11_L2/2025/311/MYD11_L2.A2025311.2150.061.2025311235943.NRT.hdf
200
https://nrt3.modaps.eosdis.nasa.gov/api/v2/content/archives/allData/61/MYD11_L2/2025/311/MYD11_L2.A2025311.2330.061.2025312014959.NRT.hdf
200

The first granule returns a 404.

More importantly, this will fail the download call:

requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://nrt3.modaps.eosdis.nasa.gov/api/v2/content/archives/allData/61/MYD11_L2/2025/311/MYD11_L2.A2025311.1225.061.2025311142708.NRT.hdf

Environment

- OS: Ubuntu 22.04
- Python: 3.13

Additional Context

No response

[chuckwondo]

@bastien-k, thanks for identifying this issue.

While we don't have any control over the search results (we simply query the NASA CMR, so it's up to the data providers to properly clean up their published metadata for things no longer available), we might consider adding some post-query filtering support.

I wouldn't want to eagerly check for 404s on search results, as that is a potentially costly performance hit that may be unnecessary in many cases, but filtering/ignoring during download would make sense.

[bastien-k]

@chuckwondo Yes I would definitely appreciate an argument to make download less sensitive to missing files.
I also had the case on archived products where sometimes a few granules are missing, and it would fail the whole download.

[chuckwondo]

@bastien-k, for now, you can do this (setting local_path to your desired directory in which to download files, and removing show_progress, if you want to see progress bars):

earthaccess.login()
paths = tuple(
    result
    for result in earthaccess.download(
        granules,
        local_path=".",
        show_progress=False,
        pqdm_kwargs={"exception_behaviour": "ignore"},
    )
    if not isinstance(result, Exception)
)
print(paths)

The trick is the pqdm_kwargs setting. The default exception behavior that earthaccess uses is "immediate", meaning that as soon as any single download raises an exception, the whole process is halted.

By overriding it to be "ignore", individual exceptions do not cause the process to fail. Instead, the exceptions are collected, along with local paths for successful downloads, so we simply need to filter out all exceptions from the list returned from the download function.

[bastien-k]

Nice trick, thanks!

[ana-sher]

Hi @chuckwondo!
I was looking at this issue and wondered if it would make sense to add an argument to download() to control this behaviour. The idea:
If the argument is True, set "exception_behaviour": "ignore" by default.
Give warnings for any files that couldn’t be downloaded.
Raise an error only if all files fail.

If the argument is not provided, keep the default immediate behavior.

[chuckwondo]

Hi @chuckwondo! I was looking at this issue and wondered if it would make sense to add an argument to download() to control this behaviour. The idea: If the argument is True, set "exception_behaviour": "ignore" by default. Give warnings for any files that couldn’t be downloaded. Raise an error only if all files fail.

If the argument is not provided, keep the default immediate behavior.

We may want to enhance the download function, but I want to take care not to make it more confusing to use. It is already starting to include too many parameters, and adding yet another parameter that also affects how a separate parameter is (implicitly) set can be rather confusing.

I've started a separate discussion for us to continue this conversation: #1147