[Question] How to strip blank,new line from data when importing CSV?

[lehoai]

Hi.
I am using vaex.from_csv() to import data but there is no option to strip data.
I saw DataFrameLocal has the

df.str.strip()

to strip data but vaex.from_csv() returns Generator instead of DataFrameLocal so cant use str.strip().
Note: my csv data is a very large file so cant strip all data in memory

Please help.

[JovanVeljanoski]

You have a couple of options:

• wait until feat: Out of core CSV support using Apache Arrow CSV reader (fast 🔥!) #1028 is merged (it is very close). With it you can lazily read a CSV file, no need to put it all into memory!
• convert data to hdf5 as is, then do out-of-core operations without loading stuff to memory
• create a custom loop where you
  • load a chunk of data
  • do the operations you need
  • store to disk as arrow/hdf5/parquet
  • when the above loop is done, concat all the chunks into one big file.

[lehoai]

@JovanVeljanoski thank you for your support. i will wait till #1028 is merged

[JovanVeljanoski]

It is merged and released :)