move download_link property to an asset

[sbrunato]

move eodag:download_link property to an asset when it is downloadable (do not keep it has a fake-value and when product download was performed from other assets). If the download_link is not required, remove it and do not add it to assets.

• use assets_mapping, see feat(plugins): new search config for assets mapping #1711
• use download_link as key
• use archive, data as roles

When downloading a whole product:

• download only assets having archive in roles
• if no asset has archive in roles, download all assets

Note:

• do not keep assets that do not have href or order_link

Technical notes

Update provider mapping is a long et slow feature to implement, do all updates at once ll makes control faster and steadier. Consider to update: downloadlink, quicklook, thumbnail, order behaviour. Keep care about assets_mapping, feature seems not complete, have to check behaviour.

• quicklook: use roles ['overview']
• thumbnail: use roles ['thumbnail']

Remember to track later in code hard code of fields "eodag:download_link" and "order:status"

Update metadata_mapping by provider

creodias_s3 (creodias_s3.log):

• eodag/resources/provider.yml
  • feat: add assets_mapping downloadlink, quicklook, thumbnail
    • move property eodag:download_link from product to asset downloadlink.href
    • move property eodag:quicklook from product to asset quicklook.href
    • move property eodag:thumbnail from product to asset thumbnail.href
    • move property order:status to asset downloadlink.order:status
  • feat: remove eodag:download_link, eodag:quicklook, eodag:thumbnail properties from product
• eodag/plugin/search/base.py
  in get_assets_from_mapping
  • allow process metadata as json path on all field with string value, not href only
  • allow process metadata with extra filters
  • allow asset without href property if has property order:link
• eodag/api/product/_assets.py
  • fix:centralize/automatize AssetDict sorting (-replication)
  • fix:centralize/automatize Asset, STAC field default
  • fix:centralize/automatize Asset, feed location and remote_location from href on update (used by s3.py)
  • force initialize undefined order:status to OFFLINE_STATUS if order:link is defined
• eodag/plugins/search/qssearch.py
  in QueryStringSearch::normalize_result
  • feat: import additionals assets, force set missing fields title and type
  • fix: when product.properties["assets"] is a list, and not a dict (seen with this provider)
  • feat: check to not add an assets sharing same url than additional one (already declared)
  • feat: remove asset sort (centralize in assetdict)
  • fix: apply only one time "product.assets.update"
• eodag.utils/s3.py
  in update_assets_from_s3
  • content_url is not initialized by product.remote_location anymore, but by product.assets['downloadlink'].remote_location
  • fix: not try s3 gather if no url
  • fix: do not add an assets if url shared with existing one (no duplicates)

cop_ads (cop_ads.log):

• eodag/resources/provider.yml
  • feat: add assets_mapping downloadlink
    • move property eodag:order_link from product to asset downloadlink.order:link
  • feat: remove eodag:order_link from product

cop_cds (cop_cds.log):

• eodag/resources/provider.yml
  • feat: add assets_mapping downloadlink
    • move property eodag:order_link from product to asset downloadlink.order:link
  • feat: remove eodag:order_link from product

cop_dataspace (cop_dataspace.log)

• eodag/resources/provider.yml
  • feat: add assets_mapping downloadlink, quicklook, thumbnail
    • move property eodag:download_link from product to asset downloadlink.href
    • move property eodag:quicklook from product to asset quicklook.href
    • move property eodag:thumbnail from product to asset thumbnail.href
    • move property order:status to asset downloadlink.order:status
    • add file:checksum to downloadlink.file:checksum
    • add file:size to downloadlink.file:size
  • feat: remove eodag:download_link, eodag:quicklook, eodag:thumbnail properties, order:status from product

cop_marine (cop_marine.log)

• eodag/plugins/search/cop_marine.py
  • feat: remove asset "native" (replaced by downloadlink)
  • feat: add assets_mapping downloadlink, quicklook, thumbnail
    • move property eodag:download_link from product to asset downloadlink.href
    • move property eodag:quicklook from product to asset quicklook.href
    • move property eodag:thumbnail from product to asset thumbnail.href
    • makes asset "quicklook" conditionnal, depends "eodag:quicklook" has a value

sara (sara.log)

• eodag/resources/provider.yml
  • feat: add assets_mapping downloadlink, quicklook, thumbnail
    • move property eodag:download_link from product to asset downloadlink.href
    • move property eodag:quicklook from product to asset quicklook.href
    • move property eodag:thumbnail from product to asset thumbnail.href
    • move property order:status from product to asset downloadlink.order:status
    • force remove computed property sara:thumbnail and sara:quicklook
    • complete asset downloadlink with availables values file:size and file:checksum

creodias (creodias.log)

• eodag/resources/provider.yml
  • feat: add assets_mapping downloadlink, quicklook, thumbnail
    • move property eodag:download_link from product to asset downloadlink.href
    • move property eodag:quicklook from product to asset quicklook.href
    • move property eodag:thumbnail from product to asset thumbnail.href
    • move property order:status from product to asset downloadlink.order:status
    • complete asset downloadlink with availables values file:size and file:checksum

dedl (dedl.log)

• eodag/resources/provider.yml
  • feat: add assets_mapping downloadlink, quicklook, thumbnail to stac_provider.yml
    • move property eodag:download_link from product to asset downloadlink.href
    • move property eodag:quicklook from product to asset quicklook.href
    • move property eodag:thumbnail from product to asset thumbnail.href
    • move property order:status from product to asset downloadlink.order:status
      /!\ Warning: metadata_mapping.assets seems not works if not redeclared in providers.yml
  • feat: add assets_mapping downloadlink, quicklook, thumbnail to provider.yml
    • patch asset downloadlink, add alternate
    • patch asset quicklook, map it from assert "ql.jpg", add alternate
    • patch asset thumbnail, map it from assert "ql.jpg", add alternate

dedt_lumi (dedt_lumi.log)

• eodag/resources/provider.yml
  • feat: add assets_mapping downloadlink
    • move property order:status to downloadlink asset
    • add property order:status

dedt_mn5 (dedt_mn5.log)

• eodag/resources/provider.yml
  • feat: add assets_mapping downloadlink
    • move property order:status to downloadlink asset
    • add property order:status

earth_search (earth_search.log)

• eodag/resources/provider.yml
  • fix: This provider has no data to map to quicklook and thumbnail, corresponding asset have to be remove on default definition in stac_provider.yml
  • feat: add assets_mapping downloadlink
    • move property order:status to downloadlink asset
    • add property order:status

earth_search_gcs (earth_search_gcs.log)

• eodag/plugins/search/base.py
  • fix: add method get_asset_mapping, managing overload current config with some config by collection to assets_mapping, update method get_assets_from_mapping, using this new method.
  • fix: does not parse filter with # inside { }
  • fix: allow tree crawl by recursion
• eodag/resources/provider.yml
  • feat: [collection S2_MSI_L1C] add assets_mapping downloadlink, quicklook, thumbnail
    • move property eodag:download_link from product to asset downloadlink.href
    • move property eodag:quicklook from product to asset quicklook.href
    • move property eodag:thumbnail from product to asset thumbnail.href
    • move property order:status from product to asset downloadlink.order:status
    • remove property eodag:download_link
  • feat: [collection L8_OLI_TIRS_C1L1] add assets_mapping downloadlink, quicklook, thumbnail
    • move property eodag:download_link from product to asset downloadlink.href
    • move property eodag:quicklook from product to asset quicklook.href
    • move property eodag:thumbnail from product to asset thumbnail.href
    • move property order:status from product to asset downloadlink.order:status
    • remove property eodag:download_link
• fix quicklook.roles to ['overwiev']
• fix thumbnail.roles to ['thumbnail']

ecmwf (ecmwf.log)

• eodag/plugins/search/base.py
  • fix: add method get_asset_mapping:
    • does not parse filter with # inside { }
    • allow tree crawl by recursion
• eodag/resources/provider.yml
  • feat: add assets_mapping downloadlink
    • move property eodag:download_link from product to asset downloadlink.href
    • move property order:status from product to asset downloadlink.order:status
    • remove property eodag:download_link and order:status from product
  • fix(style): change key asset "downloadlink" to "download_link"
  • fix(style): change existing field order:link to order_link

eumetsat (eumetsat_gs.log)

• eodag/plugins/search/base.py
  • fix: exclude asset if href and order_link not defined (empty string or NOT_AVAILABLE)
• eodag/resources/provider.yml
  • feat: add assets_mapping downloadlink, quicklook, thumbnail
    • move property eodag:download_link from product to asset downloadlink.href
    • move property eodag:quicklook from product to asset quicklook.href
    • move property order:status from product to asset downloadlink.order:status
    • remove property eodag:download_link, eodag:quicklook and order:status from product

geodes (geodes.log)

• eodag/resources/provider.yml
  • feat: add assets_mapping downloadlink
    • move property eodag:download_link from product to asset downloadlink.href
    • move property eodag:quicklook from product to asset quicklook.href
    • move property eodag:thumbnail from product to asset thumbnail.href
    • move property order:status from product to asset downloadlink.order:status
    • remove property eodag:download_link, eodag:quicklook, eodag:thumbnail and order:status from product

hydroweb_next (hydroweb_next.log)

• eodag/resources/provider.yml
  • feat: assets_mapping: nothing to map
  • feat: add unmapped assets

peps (peps.log)

• eodag/plugins/search/base.py
  • fix: exclude asset if href and order_link not defined (empty string or NOT_AVAILABLE or None)
• eodag/resources/provider.yml
  • move property eodag:download_link from product to asset downloadlink.href
  • move property eodag:quicklook from product to asset quicklook.href
  • move property eodag:thumbnail from product to asset thumbnail.href
  • move property order:status from product to asset downloadlink.order:status
  • add mapping file:size and file:checksum to asset downloadlink
  • remove property eodag:download_link, eodag:quicklook, eodag:thumbnail and order:status from product

planetary_computer (planetary_computer.log)

• eodag/resources/provider.yml
  • map asset identified as quicklook to asset quicklook.href
  • map asset identified as thumbnail to asset thumbnail.href

usgs_satapi_aws (usgs_satapi_aws.log)

• eodag/plugins/search/base.py
  • fix: prevent asset url replication will now consider href or order_link as asset "target_url"
• eodag/plugins.search/qssearch.py
  • fix: prevent asset url replication will now consider href or order_link as asset "target_url"
• eodag/utils/s3.py
  • fix: prevent asset url replication will now consider href or order_link as asset "target_url"
• eodag/resources/provider.yml
  • map asset identified as quicklook to asset quicklook.href
  • map asset identified as thumbnail to asset thumbnail.href

cop_ewds (cop_ewds.log)

• eodag/plugins/search/base.py
  in get_assets_from_mapping
  • fix: missing feature in asset mapping: parameters substitution from product properties
  • fix: re-check providers with property substitution in assets (dedt_lumi, cop_cds, cop_ads, ecmwf)
  • fix: complete product.properties with config.product (not always done get when get_assets_from_mapping)
• eodag/resources/provider.yml
  • move order_link from product to asset download_link

wekeo_main (wekeo_main.log)

• eodag/plugins/search/base.py
  • fix: get_metadata_mapping and get_assets_mapping, consider override by collection configuration and "metadata_mapping_from_product" refered collection configuration
    require properties and provider data
  • fix: patch broken wekeo configuation (some queryable not respected)
  • fix: remove anchor reducing visibility, and only use once, or 2 time max (not really avoid replication), makes it simplier.
  • fix: remove useless parameters override by another configuration
    -style: isolate new interpretation of asset_mapping in a stand-alone static class to makes control easier.
• eodag/api/product/metadata_mapping.py
  in format_query_params
  • fix: a case someting happens, function try to force interprete a chunk of configuration without anything to interprete and corrupt string.
• eodag/plugins/search/qsearch.py
  • fix: now, with configuration shared with asset_mapping, it can happens that metadata_mapping is empty, and it's nomal. Remove error when try overload empty metadata_mapping because it's empty (not misconfiguration anymore)
• eodag/resources/provider.yml
  • fix: moved eodag:download:link to asset.downlink.href cannot rebuild it's template because of non importing
  • move eodag:download_link to asset.download_link.href or order_link
  • feat: add a way to insert into string configuration a $.json path (used to build asset.download_link.order_link
  • feat: resolve configuration overload before start interprete templates. it's allow to override a parameters used in another parameter pattern. Use in asset.download_link.order_link to insert _collection value instead of a hardcode.

wekeo_cmens

• eodag/resources/provider.yml
  • feat: add assets_mapping downloadlink
    • move property eodag:download_link from product to asset downloadlink.href
    • move property order:status from product to asset downloadlink.order:status
    • remove property eodag:download_link and order:status from product

/!\ Bug on request even with develop branch

eodag.utils.exceptions.RequestError: ('Skipping error while searching for wekeo_cmems PostJsonSearch instance', ' {"status_code":404,"title":"Not Found","detail":"Param \'dataset_id\' not found"}')

upgrade cannot be tested.

wekeo_ecmwf (wekeo_ecmwf.log)

• eodag/api/product/metadata_mapping.py
  • add filter "url_decode"
• eodag/resources/provider.yml
  • feat: add assets_mapping downloadlink, thumbnail
    • property eodag:download_link is an url encoded json, not an url, property removed.
    • move property order_link to asset download_link, urldecode imported value $.properties.location
    • add property order:status as "orderable" from product to asset downloadlink.order:status
    • add mappable asset thumbnail and quicklook
    • remove property eodag:download_link and order:status from product

meteoblue

• eodag/resources/provider.yml
  • feat: add assets_mapping downloadlink
    • move property eodag:download_link from product to asset downloadlink.href
    • move property order:status from product to asset downloadlink.order:status
    • move property eodag:order_link from product to asset downloadlink.order_link
    • remove property eodag:download_link, order:status and eodag:order_link from product

Provider configuration does not pass it's own validation, new configuration cannot be tested.
#1896

aws_eos (aws_eos.log)

• eodag/resources/provider.yml
  • feat: add assets_mapping downloadlink
    • move property eodag:download_link from product to asset downloadlink.href
    • move property eodag:thumbnail from product to asset downloadlink.thumbnail
    • move property eodag:quicklook from product to asset downloadlink.quicklook
    • move property order:status from product to asset downloadlink.order:status
    • clean unused temporary properties
    • apply to on configuration by product

geodes

• eodag/api/product/metadata_mapping.py
  • add filter "round"
• eodag/resources/provider.yml
  • feat: add assets_mapping downloadlink
    • move property eodag:download_link from product to asset downloadlink.href
    • move property eodag:thumbnail from product to asset downloadlink.thumbnail
    • move property eodag:quicklook from product to asset downloadlink.quicklook
    • move property order:status from product to asset downloadlink.order:status

Test need CNES network access.

usgs

• eodag/resources/provider.yml
  • feat: add assets_mapping downloadlink
    • move property eodag:download_link from product to asset downloadlink.href
    • move property eodag:thumbnail from product to asset downloadlink.thumbnail
    • move property eodag:quicklook from product to asset downloadlink.quicklook
    • move property order:status from product to asset downloadlink.order:status

/!\ Provider troubleshot, auth return http 403 instead of json, tried too with another account (augustin)
Implementation not testable.