[Feature]: Support custom credential providers to be passed to `object_store`

[dlstadther]

Is your feature request related to a problem?

DeltaLake (via object_store) does not support Workload Identity Federation authentication. While I can authenticate with WIF outside of DeltaLake/object_store and get a token, DeltaLake does not support a mechanism for me to provide this token to the object_store library.

While object_store allows user-provided custom credential providers to work around this issue, DeltaLake does not support any storage_option value other than string.

(Polars solved this issue by enabling a UserProvidedGCPToken provider when a plain token is given).

Describe the solution you'd like

Allow for a pattern where the user can provide their own access token which is wrapped by a credential provider for object_store or allow the user to provide their own credential provider.

Describe alternatives you've considered

I have not found any alternatives yet for Workload Identity Federation support or workaround with DeltaLake, only the credential_provider route directly with object_store which isn't supported by DeltaLake's storage_option interface.

Priority

None

Additional context

No response

Contribution

• I'm willing to submit a pull request for this feature
• I can help with testing this feature
• I can help with documentation for this feature

[dlstadther]

@rtyler @ion-elgreco , I've looked through the history of the project and can't identify the motivation behind the dict[str, str] datatype restriction for storage_options. Do either of you know the level of complexity involved to be able to support users to provide a credential_provider value to storage_options which contains a python callable?

I do not have the rust/python-binding experience to gauge this myself.

[ion-elgreco]

@dlstadther this can be easily solved by adding support upstream in object-store

[dlstadther]

The maintainers of that project expressed (a few years ago) openness to accept contributions for alternative (but common) auth flows, but additionally enabled folks to be unblocked by missing native authorization flow support by exposing their CredentialProvider.

Looking more into Polars' solution, it actually looks like the credential_provider kwarg is Polars-specific extension to allow their users entry into object_store's extensible authentication mechanism.

Workload Identity Federation is specifically the auth flow that I require, but that is not guaranteed to the be the only flow that users may require.

Is a solution similar to what Polars did not acceptable in your view for deltalake? If that's so, deltalake users will be limited to only the native authentication flows that object_store supports.