Support for env var substitutions in env.str and env.dict and other composite types

[gnarvaja]

I found some use cases where doing env var substitution inside the VALUE of another envvar it's very helpful.

For example:

DB_URI=postgres://${PGUSER:-gnarvaja}:${PGPASS}@myserver/mydb
PGPASS=awesome

And being able to read the variable like this would be awesome:

>>> env.str("DB_URI", substitute_envs=True)
postgres://gnarvaja:awesome@myserver/mydb

The main use case is when you have the user/pass as individuals secrets (perhaps handled by an operator like https://github.com/zalando/postgres-operator) but you need a URI with parts that don't need to be secret.

Another use case is a workaround when composite variables like dict become too complex. For example:

INIT_KARGS="name=foo,script_params=--http-header Referer=https://radiocut.fm"

If you do env.dict("INIT_KARGS") you get ValueError: too many values to unpack because of the "=" in the second value. So in this case you can do:

INIT_KARGS="name=foo,script_params=--http-header ${HTTP_HEADER}"
HTTP_HEADER="Referer=https://radiocut.fm"

>>> env.dict("INIT_KARGS", substitute_envs=True)
{"name": "foo", "script_params": "--http-header Referer=https://radiocut.fm"}

We can use a syntax like this for the references to the environments variables: ${VARIABLE[:-default][:ttype]}. For example ${YEAR:-2020:tint}. (the :- is how you specify defaults in bash, the third :t parameter is for specifying the type is not str).

Here is the code I'm using in my project (a bit ugly):

envvar_matcher = re.compile(r'\$\{([A-Za-z0-9_]+)(:-[^\}:]*)?(:t[^\}]*)?\}')


def _replace_envs(value):
    ret = ""
    prev_start = 0
    for m in envvar_matcher.finditer(value):
        env_name = m.group(1)
        env_default = m.group(2)
        env_type = m.group(3)
        if env_type is None:
            method = env.str
        else:
            method_name = env_type[2:]
            method = getattr(env, method_name)
        if env_default is None:
            env_value = method(env_name)
        else:
            env_default = env_default[2:]
            env_value = method(env_name, default=env_default)

        if m.start() == 0 and m.end() == len(value):
            return env_value
        ret += value[prev_start:m.start()] + env_value
        prev_start = m.end()
    ret += value[prev_start:]
    return ret

If you agree with this new feature, I can try to do a PR. The new feature will be activated only if explicitly using substitute_envs=True as parameter.

[gnarvaja]

Reviewing the docs and the open PRs, perhaps this feature can be merged or replace Proxied Variables. Proxied variables are a special case of this substitution.

Anyway, I think it shouldn't be enabled by default, as this can be a security issue (For example someone sets: TITLE=${SECRET_ENV_VAR})

[sloria]

Interesting idea. I'm not opposed to this in principle, esp if it doesn't add too much complexity. I agree that it could possibly replace proxied variables.

Would you like to put up a PR sketch?

[gnarvaja]

Done, check #168 .

I wasn't sure about proxied variables, perhaps a path for deprecation might be raising DeprecationWarning for a few versions and later remove

[sloria]

Great, thanks! I won't be able to review the PR in depth for the next few days--work and life commitments all coming fast--but I'll do my best not to lose track of it.

[gnarvaja]

Sure, no problem. Tell if you need I do any fix to the PR. Thank you for this awesome project El mar., 1 sep. 2020 12:57, Steven Loria <notifications@github.com> escribió:

…

Great, thanks! I won't be able to review the PR in depth for the next few days--work and life commitments all coming fast--but I'll do my best not to lose track of it. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#164 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAA6743MQYE6AYIV5BTGA4DSDUKVRANCNFSM4PTVCLWA> .