print(os.environ.keys()) should only print the keys

[asmeurer]

BPO	32300
Nosy	@rhettinger, @vstinner, @bitdancer, @asmeurer, @serhiy-storchaka, @csabella

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2017-12-13.08:37:00.625>
labels = ['type-bug', 'library', '3.9', '3.10']
title = 'print(os.environ.keys()) should only print the keys'
updated_at = <Date 2020-11-06.19:11:50.449>
user = 'https://github.com/asmeurer'

bugs.python.org fields:

activity = <Date 2020-11-06.19:11:50.449>
actor = 'iritkatriel'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2017-12-13.08:37:00.625>
creator = 'Aaron.Meurer'
dependencies = []
files = []
hgrepos = []
issue_num = 32300
keywords = []
message_count = 20.0
messages = ['308190', '308199', '308200', '308219', '308220', '308223', '308241', '308244', '308249', '308251', '308266', '308309', '308314', '308315', '308321', '308334', '308335', '308338', '308357', '308358']
nosy_count = 6.0
nosy_names = ['rhettinger', 'vstinner', 'r.david.murray', 'Aaron.Meurer', 'serhiy.storchaka', 'cheryl.sabella']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue32300'
versions = ['Python 3.9', 'Python 3.10']

[asmeurer]

Take the following scenario which happened to me recently. I am trying to debug an issue on Travis CI involving environment variables. Basically, I am not sure if an environment variable is being set correctly. So in my code, I put

print(os.environ.keys())

The reason I put keys() was 1, I didn't care about the values, and 2, I have secure environment variables set on Travis.

To my surprise, in the Travis logs, I found something like this

KeysView(environ({'TRAVIS_STACK_FEATURES': 'basic cassandra chromium couchdb disabled-ipv6 docker docker-compose elasticsearch firefox go-toolchain google-chrome jdk memcached mongodb mysql neo4j nodejs_interpreter perl_interpreter perlbrew phantomjs postgresql python_interpreter rabbitmq redis riak ruby_interpreter sqlite xserver', 'CI': 'true', ..., 'MANPATH': '/home/travis/.nvm/versions/node/v7.4.0/share/man:/home/travis/.kiex/elixirs/elixir-1.4.5/man:/home/travis/.rvm/rubies/ruby-2.4.1/share/man:/usr/local/man:/usr/local/clang-3.9.0/share/man:/usr/local/share/man:/usr/share/man:/home/travis/.rvm/man'}))

So instead of just printing the keys like I asked for, it printed the whole environment, plus "KeysView(environ(". Included here was my secure environment variable.

Now, fortunately, Travis hides the contents of secure environment variables in the logs, but it didn't used to (https://blog.travis-ci.com/2017-05-08-security-advisory).

Aside from being a potential security issue, it's just annoying that it prints the whole environment. The values are much larger than the keys. With a normal dictionary, print(d.keys()) just prints the keys:

>>> print(dict(a=1, b=2).keys())
dict_keys(['a', 'b'])

[csabella]

For your current situation, list(os.environ) or iter(os.environ) both return keys only.

It looks like the __repr__ on the class for os.environ is printed for os.environ (which is expected). For os.environ.keys(), the same __repr__ is wrapped as a KeysView, for os.environ.values(), it's wrapped as a ValuesView, and os.environ.items(), as an ItemsView.

In 2.7, os.environ.keys() print just keys.

[bitdancer]

This is a consequence of the repr used by KeysView, which it inherits from MappingView. I agree that the result is surprising, but there may not be a generic fix. It's not entirely clear what KeysView should do here, but presumably we could at least add a repr to environ's use of it that would produce something useful.

Note that a workaround when doing something like a travis debug is to print os.environ._data.keys(), which will print the bytes versions of the keys. I wouldn't recommend depending on that continuing to, though ;)

[vstinner]

Usually, I use print(sorted(os.environ)) since I prefer a sorted list and it prevents such issue :-)

David:

I agree that the result is surprising, but there may not be a generic fix.

What about something like:

vstinner@apu$ ./python -c 'import os; print(os.environ.keys())'
<KeysView ['LS_COLORS', 'XDG_MENU_PREFIX', 'LANG', 'GDM_LANG', 'HISTIGNORE', 'HISTCONTROL', 'DISPLAY', 'HOSTNAME', 'EDITOR', 'COLORTERM', 'DESKTOP_AUTOSTART_ID', 'USERNAME', 'XDG_VTNR', 'SSH_AUTH_SOCK', 'XDG_SESSION_ID', 'USER', 'DESKTOP_SESSION', 'GTK2_RC_FILES', 'PWD', 'SSH_ASKPASS', 'HOME', 'JOURNAL_STREAM', 'XDG_SESSION_TYPE', 'XDG_DATA_DIRS', 'XDG_SESSION_DESKTOP', 'LOADEDMODULES', 'MAIL', 'WINDOWPATH', 'VTE_VERSION', 'TERM', 'SHELL', 'QT_IM_MODULE', 'XMODIFIERS', 'XDG_CURRENT_DESKTOP', 'XDG_SEAT', 'SHLVL', 'PYTHONPATH', 'WINDOWID', 'MODULEPATH', 'GDMSESSION', 'GNOME_DESKTOP_SESSION_ID', 'LOGNAME', 'DBUS_SESSION_BUS_ADDRESS', 'XDG_RUNTIME_DIR', 'XAUTHORITY', 'PATH', 'PS1', 'MODULESHOME', 'MAKEFLAGS', 'HISTSIZE', 'SESSION_MANAGER', 'LESSOPEN', 'BASH_FUNC_module%%', 'BASH_FUNC_scl%%', '_', 'OLDPWD']>

vstinner@apu$ git diff
diff --git a/Lib/_collections_abc.py b/Lib/_collections_abc.py
index a5c7bfcda1..5e524dffbf 100644
--- a/Lib/_collections_abc.py
+++ b/Lib/_collections_abc.py
@@ -719,6 +719,9 @@ class KeysView(MappingView, Set):
     def __iter__(self):
         yield from self._mapping
 
+    def __repr__(self):
+        return '<KeysView %r>' % list(self)
+
 KeysView.register(dict_keys)
 
 

list(key_view) is valid. I mimicked dict views implementation:

static PyObject *
dictview_repr(_PyDictViewObject *dv)
{
    PyObject *seq;
    PyObject *result;

    seq = PySequence_List((PyObject *)dv);
    if (seq == NULL)
        return NULL;

    result = PyUnicode_FromFormat("%s(%R)", Py_TYPE(dv)->tp_name, seq);
    Py_DECREF(seq);
    return result;
}

[vstinner]

If we decide to change abc.KeysView.__repr__, IMHO we should also modify abc.ValuesView.__repr__, and maybe also abc.ItemsView.__repr__.

[bitdancer]

Agreed about the other classes if we change this. Your solution looks reasonable to me.

[vstinner]

Cheryl: would you like to work on a PR? If yes, tests are needed.

[serhiy-storchaka]

Don't make KeysView.__repr__ and ValuesView.__repr__ containing the lists of all keys and values. This will make the repr of the view of a mapping which is a proxy of an external DB containing the full content of that DB, which can be gigabytes. See for example rejected bpo-21670.

[asmeurer]

So the best fix is to just override keys() in the _Environ class, so that it returns an EnvironKeysView class that overrides __repr__?

[vstinner]

Don't make KeysView.__repr__ and ValuesView.__repr__ containing the lists of all keys and values. This will make the repr of the view of a mapping which is a proxy of an external DB containing the full content of that DB, which can be gigabytes. See for example rejected bpo-21670.

The current implementation displays repr(self._mapping):

class MappingView(Sized):
    ...
    def __repr__(self):
        return '{0.__class__.__name__}({0._mapping!r})'.format(self)

For your proxy example: what is the proxy? The KeysView subtype, or the mapping?

repr(list), repr(dict) and repr(set) all render their full content in the result, no?

repr() on a list of 1 million of items *will* render all items, even if nobody wants to read such very long string :-)

If you want to get a smarter __repr__() than the default implementation: just override __repr__(), no?

I don't see why KeysView must be special.

[serhiy-storchaka]

shelve.Shelf is the example of such kind.