Series.map and ignore_na depends on mapping type

[behrenhoff]

Hi there,

according to the documentation, the parameter ignore_na="ignore" propagate[s] NaN values, without passing them to the mapping correspondence. (quote from https://pandas.pydata.org/docs/reference/api/pandas.Series.map.html)

It seems this only works if the mapping is a function, not a dictionary. See this test:

import pandas as pd
import numpy as np

mapping = {1: 10, np.nan: 42}

# succeeds as expected (basic simple test)
pd.testing.assert_series_equal(
    pd.Series([1, np.nan, 2]).map(mapping),
    pd.Series([10, 42, np.nan]),
)

# succeeds as expected, note the seemingly unnecessary lambda
pd.testing.assert_series_equal(
    pd.Series([1, np.nan, 2]).map(lambda x: mapping.get(x), na_action="ignore"),
    pd.Series([10, np.nan, np.nan]),
)

# remove the lambda and use the map directly
pd.testing.assert_series_equal(
    pd.Series([1, np.nan, 2]).map(mapping, na_action="ignore"),
    pd.Series([10, np.nan, np.nan]),
)
# Whoops!
# AssertionError: Series are different
# 
# Series values are different (33.33333 %)
# [index]: [0, 1, 2]
# [left]:  [10.0, 42.0, nan]    <-- it did replace the nan even though na_action is ignore
# [right]: [10.0, nan, nan]

So to me it seems the na_action parameter is ignored for dictionaries but works for functions. I don't think it is a good idea to have different behaviour depending on the type of the mapping. So could this either be fixed or corrected in the documentation?

Tested with version 1.4.2, Python 3.10

(granted, np.nan as a dictionary key is actively asking for trouble...)

[rhshadrach]

Thanks for the report! I see the same behavior on main; I agree this is a bug. Would you be interesting in working on a PR to fix?

[behrenhoff]

Hm... Didn't think about it - but why not? :-) I am sort of busy at the moment though.

So the solution would be to update the behavior and not the documentation? The thing is: why would it even be useful to ignore a certain key in the dictionary?

The offending code is in pandas/core/base.py, function _map_values, around line 820 and following. In the is_dict_like(mapper) or the isinstance(mapper, ABCSeries) case, na_action is completely ignored - there is a (quote) "fastpath dict/Series to an efficient map".

Therefore the following also returns a Series of [10, 20] independent of na_action.

pd.Series([1, np.nan]).map(pd.Series([10, 20], index=[1, np.nan]), na_action="ignore")

Why would anyone use nan in a Series index except when asking for trouble? Anyway, it doesn't work as documented.

[rhshadrach]

Why would anyone use nan in a Series index except when asking for trouble?

df = pd.DataFrame({'a': [1, 2, 3, np.nan, 1, 2, np.nan, 1, 2]})
print(df['a'].value_counts(dropna=False))
# 1.0    3
# 2.0    3
# NaN    2
# 3.0    1
# Name: a, dtype: int64

The thing is: why would it even be useful to ignore a certain key in the dictionary?

I can't say I've found a use case. But if you'd like to discuss the possibility of removing the argument, I'd suggest opening a separate issue and to have this one be focused on discussing the buggy behavior.