parser improperly adding invalid kwarg

[staftermath]

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 20.04
• Modin version (modin.__version__): 0.8.2
• Python version: 3.7.9
• Code we can use to reproduce: need to pip install memory_profiler. The csv doesn't matter, for what's it's worth:
  sample.csv

a,b
1,2
3,4

code:

import os
os.environ["MODIN_ENGINE"] = "ray"

import modin.pandas as pd
from memory_profiler import profile


@profile  # code runs fine after removing this
def load_csv():
    return pd.read_csv("./sample.csv")

if __name__ == '__main__':
    load_csv()

Describe the problem

Description: I can't say for sure but I think https://github.com/modin-project/modin/blob/master/modin/pandas/io.py#L106 might be introducing some unexpected parameters. This is causing modin.pandas to have conflict to memory-profiler's decorator @Profile.

I debugged the profile decorator. The unwanted kwarg doesn't seem to be introduced there.
Debug at line https://github.com/modin-project/modin/blob/master/modin/pandas/io.py#L107:

>>> kwargs.keys()
dict_keys(['filepath_or_buffer', 'sep', 'delimiter', 'header', 'names', 'index_col', 'usecols', 'squeeze', 'prefix', 'mangle_dupe_cols',
 'dtype', 'engine', 'converters', 'true_values', 'false_values', 'skipinitialspace', 'skiprows', 'nrows', 'na_values', 'keep_default_na',
 'na_filter', 'verbose', 'skip_blank_lines', 'parse_dates', 'infer_datetime_format', 'keep_date_col', 'date_parser', 'dayfirst', 
'cache_dates', 'iterator', 'chunksize', 'compression', 'thousands', 'decimal', 'lineterminator', 'quotechar', 'quoting', 'escapechar', 
'comment', 'encoding', 'dialect', 'error_bad_lines', 'warn_bad_lines', 'skipfooter', 'doublequote', 'delim_whitespace', 'low_memory', 
'memory_map', 'float_precision', '_', 'kwargs'])  # notice the '_' item

Source code / logs

Traceback (most recent call last):
  File "/home/my_username/Documents/projects/profiling/main.py", line 13, in <module>
    load_csv()
  File "/home/my_username/anaconda3/envs/profiling/lib/python3.7/site-packages/memory_profiler.py", line 1142, in wrapper
    val = prof(func)(*args, **kwargs)
  File "/home/my_username/anaconda3/envs/profiling/lib/python3.7/site-packages/memory_profiler.py", line 717, in f
    return func(*args, **kwds)
  File "/home/my_username/Documents/projects/profiling/main.py", line 10, in load_csv
    return pd.read_csv("./sample.csv")
  File "/home/my_username/anaconda3/envs/profiling/lib/python3.7/site-packages/modin/pandas/io.py", line 109, in parser_func
    return _read(**kwargs)
  File "/home/my_username/anaconda3/envs/profiling/lib/python3.7/site-packages/modin/pandas/io.py", line 127, in _read
    pd_obj = EngineDispatcher.read_csv(**kwargs)
  File "/home/my_username/anaconda3/envs/profiling/lib/python3.7/site-packages/modin/data_management/factories/dispatcher.py", line 104, in read_csv
    return cls.__engine._read_csv(**kwargs)
  File "/home/my_username/anaconda3/envs/profiling/lib/python3.7/site-packages/modin/data_management/factories/factories.py", line 87, in _read_csv
    return cls.io_cls.read_csv(**kwargs)
  File "/home/my_username/anaconda3/envs/profiling/lib/python3.7/site-packages/modin/engines/base/io/file_reader.py", line 29, in read
    query_compiler = cls._read(*args, **kwargs)
  File "/home/my_username/anaconda3/envs/profiling/lib/python3.7/site-packages/modin/engines/base/io/text/csv_reader.py", line 69, in _read
    **dict(kwargs, usecols=None, nrows=0, skipfooter=0, index_col=None),
TypeError: read_csv() got an unexpected keyword argument '_'

Interestingly, this issue seems to dependent on python version, under 3.6.12, the traceback becomes:

Traceback (most recent call last):
  File "main.py", line 13, in <module>
    load_csv()
  File "/home/my_username/anaconda3/envs/profiling3.6/lib/python3.6/site-packages/memory_profiler.py", line 1142, in wrapper
    val = prof(func)(*args, **kwargs)
  File "/home/my_username/anaconda3/envs/profiling3.6/lib/python3.6/site-packages/memory_profiler.py", line 717, in f
    return func(*args, **kwds)
  File "main.py", line 10, in load_csv
    return pd.read_csv("./sample.csv")
  File "/home/my_username/anaconda3/envs/profiling3.6/lib/python3.6/site-packages/modin/pandas/io.py", line 109, in parser_func
    return _read(**kwargs)
  File "/home/my_username/anaconda3/envs/profiling3.6/lib/python3.6/site-packages/modin/pandas/io.py", line 127, in _read
    pd_obj = EngineDispatcher.read_csv(**kwargs)
  File "/home/my_username/anaconda3/envs/profiling3.6/lib/python3.6/site-packages/modin/data_management/factories/dispatcher.py", line 104, in read_csv
    return cls.__engine._read_csv(**kwargs)
  File "/home/my_username/anaconda3/envs/profiling3.6/lib/python3.6/site-packages/modin/data_management/factories/factories.py", line 87, in _read_csv
    return cls.io_cls.read_csv(**kwargs)
  File "/home/my_username/anaconda3/envs/profiling3.6/lib/python3.6/site-packages/modin/engines/base/io/file_reader.py", line 29, in read
    query_compiler = cls._read(*args, **kwargs)
  File "/home/my_username/anaconda3/envs/profiling3.6/lib/python3.6/site-packages/modin/engines/base/io/text/csv_reader.py", line 69, in _read
    **dict(kwargs, usecols=None, nrows=0, skipfooter=0, index_col=None),
TypeError: read_csv() got an unexpected keyword argument 'kwargs'

[devin-petersohn]

Hi @staftermath, thanks for the report!

I was able to reproduce the issue, the decorator is adding local variables that get detected by inspect. I tested with locals as well, and the issue is still there.

The only fix for this is to explicitly build the kwargs with each keyword argument. We originally did this, but changed to this approach in the interest of reducing the code length and improving readability. It looks like we need to move back to the manually creating the dictionary from each keyword argument to avoid these extra local variables popping up internally.

@staftermath You have already been into the code, do you want to add a PR changing this?

[staftermath]

Hi @staftermath, thanks for the report!

I was able to reproduce the issue, the decorator is adding local variables that get detected by inspect. I tested with locals as well, and the issue is still there.

The only fix for this is to explicitly build the kwargs with each keyword argument. We originally did this, but changed to this approach in the interest of reducing the code length and improving readability. It looks like we need to move back to the manually creating the dictionary from each keyword argument to avoid these extra local variables popping up internally.

@staftermath You have already been into the code, do you want to add a PR changing this?

Thanks for looking into this and confirming :) I'd love to work on this when I got the chance. Not sure if you guys have desired timeline for this. Let me know if there is a preferrence.

[devin-petersohn]

The timeline can be flexible for this, I personally have a pretty full plate for several weeks at least. Feel free to work on it as you get the time! Contributing guidelines can be found here: https://modin.readthedocs.io/en/latest/contributing.html

[staftermath]

perfect. Thanks! I'll work on it as soon as I can.