FEAT-#1067: skiprows support added for read_csv
#2607
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## master #2607 +/- ##
===========================================
- Coverage 85.34% 26.13% -59.22%
===========================================
Files 151 145 -6
Lines 15916 14946 -970
===========================================
- Hits 13584 3906 -9678
- Misses 2332 11040 +8708
Continue to review full report at Codecov.
|
|
|
|
amyskov
force-pushed
the
issue-1067
branch
5 times, most recently
from
8bd2638 to
46aff86
Compare
amyskov
force-pushed
the
issue-1067
branch
3 times, most recently
from
ae42db1 to
6407fe5
Compare
|
Performance numbers obtained by According to the above table, performance of |
|
@anmyachev @dchigarev please review this PR. |
anmyachev
reviewed
Feb 4, 2021
modin/pandas/test/utils.py
Outdated
Comment on lines
733
to
753
| if isinstance(result.index, pandas.MultiIndex): | ||
| levels_number = len(result.index.levels) | ||
| new_index_data = [ | ||
| result.index.get_level_values(lev_id).astype(str) | ||
| for lev_id in range(levels_number) | ||
| ] | ||
| new_index = pandas.MultiIndex.from_arrays(new_index_data) | ||
| new_index.names = result.index.names | ||
| result.index = new_index |
There was a problem hiding this comment.
why not just result.index = result.index.astype(str)?
There was a problem hiding this comment.
If we apply astytpe to the Multiindex object directly we will obtain error
TypeError: Setting <class 'pandas.core.indexes.multi.MultiIndex'> dtype to anything other than object is not supported
But actually in this PR Multiindex conversion is not used, so these changes were removed as unrelated.
There was a problem hiding this comment.
So are you going to delete this?
anmyachev
requested changes
Feb 4, 2021
anmyachev
requested changes
Feb 4, 2021
|
@anmyachev @dchigarev do you have any comments on this PR? |
anmyachev
requested changes
Feb 8, 2021
amyskov
force-pushed
the
issue-1067
branch
2 times, most recently
from
a2f73a4 to
2da042f
Compare
|
@anmyachev PR is ready for review. |
anmyachev
requested changes
Feb 9, 2021
|
@dchigarev please review this |
dchigarev
reviewed
Feb 9, 2021
dchigarev
reviewed
Feb 9, 2021
dchigarev
reviewed
Feb 9, 2021
You are right, we can drop |
There was a problem hiding this comment.
@amyskov it makes sense, maybe we should test the Cython addition first?
There are some test failures here as well. Can you take a look?
amyskov
force-pushed
the
issue-1067
branch
2 times, most recently
from
5dcff16 to
0878d06
Compare
|
@devin-petersohn , this PR was reworked in the way it was discussed offline - now array-like or callable As it can be seen |
|
Hi @devin-petersohn please take a look at this PR. |
|
@amyskov you got some conflicts here |
@devin-petersohn this PR was rebased and now ready for review. |
Unrelated change was moved into separated PR. Yes, it was really long way to get it done) |
Signed-off-by: Dmitry Chigarev <[email protected]> FEAT-modin-project#1067: skiprows support for read_csv added Signed-off-by: Dmitry Chigarev <[email protected]> FEAT-modin-project#1067: make 'skiprows' and 'nrows' work together Signed-off-by: Dmitry Chigarev <[email protected]> FEAT-modin-project#1067: range like 'skiprows' optimizations Signed-off-by: Dmitry Chigarev <[email protected]> Signed-off-by: Alexander Myskov <[email protected]> FEAT-modin-project#1067: rework read_csv part Signed-off-by: Alexander Myskov <[email protected]> FEAT-modin-project#1067: rework json and fwf dispatchers Signed-off-by: Alexander Myskov <[email protected]> FEAT-modin-project#1067: add description Signed-off-by: Alexander Myskov <[email protected]> FEAT-modin-project#1067: add tests Signed-off-by: Alexander Myskov <[email protected]> FEAT-modin-project#1067: mark xfailed tests Signed-off-by: Alexander Myskov <[email protected]> FEAT-modin-project#1067: fix skiprows when encoding is passed Signed-off-by: Alexander Myskov <[email protected]> FEAT-modin-project#1067: handle partitions with empty DataFrames Signed-off-by: Alexander Myskov <[email protected]> FEAT-modin-project#1067: mark xfailed tests Signed-off-by: Alexander Myskov <[email protected]> FEAT-modin-project#1067: remove unused code Signed-off-by: Alexander Myskov <[email protected]> FEAT-modin-project#1067: add asv tests Signed-off-by: Alexander Myskov <[email protected]> FEAT-modin-project#1067: handle narrow dataframes Signed-off-by: Alexander Myskov <[email protected]> FEAT-modin-project#1067: fix asv tests Signed-off-by: Alexander Myskov <[email protected]> FEAT-modin-project#1067: fix Signed-off-by: Alexander Myskov <[email protected]> FEAT-modin-project#1067: remove code duplication Signed-off-by: Alexander Myskov <[email protected]> FEAT-modin-project#1067: consolidate some code into functions Signed-off-by: Alexander Myskov <[email protected]> FEAT-modin-project#1067: correct io test to check pre_reading part Signed-off-by: Alexander Myskov <[email protected]> FEAT-modin-project#1067: remove unrelated changes Signed-off-by: Alexander Myskov <[email protected]> FEAT-modin-project#1067: Apply suggestions from code review Co-authored-by: Anatoly Myachev <[email protected]> Signed-off-by: Alexander Myskov <[email protected]> FEAT-modin-project#1067: correct asv tests Signed-off-by: Alexander Myskov <[email protected]> FEAT-modin-project#1067: addressing review comments Signed-off-by: Alexander Myskov <[email protected]> FEAT-modin-project#1067: Update modin/engines/base/io/text/text_file_dispatcher.py Co-authored-by: Anatoly Myachev <[email protected]> Signed-off-by: Alexander Myskov <[email protected]> FEAT-modin-project#1067: Update modin/engines/base/io/text/csv_dispatcher.py Co-authored-by: Dmitry Chigarev <[email protected]> Signed-off-by: Alexander Myskov <[email protected]> FEAT-modin-project#1067: rework asv tests Signed-off-by: Alexander Myskov <[email protected]> FEAT-modin-project#1067: rollback parser changes Signed-off-by: Alexander Myskov <[email protected]> FEAT-modin-project#1067: Update modin/engines/base/io/text/text_file_dispatcher.py Co-authored-by: Anatoly Myachev <[email protected]> Signed-off-by: Alexander Myskov <[email protected]> FEAT-modin-project#1067: update tests Signed-off-by: Alexander Myskov <[email protected]> FEAT-modin-project#1067: fix for cloud tests Signed-off-by: Alexander Myskov <[email protected]> FEAT-modin-project#1067: Update modin/engines/base/io/text/text_file_dispatcher.py Co-authored-by: Dmitry Chigarev <[email protected]> Signed-off-by: Alexander Myskov <[email protected]> FEAT-modin-project#1067: remove asv tests Signed-off-by: Alexander Myskov <[email protected]> FEAT-modin-project#1067: addressing review comments Signed-off-by: Alexander Myskov <[email protected]> FEAT-modin-project#1067: addressing review comments Signed-off-by: Alexander Myskov <[email protected]> FEAT-modin-project#1067: correct getting dummy_df Signed-off-by: Alexander Myskov <[email protected]> FEAT-modin-project#1067: remove duplicated code Signed-off-by: Alexander Myskov <[email protected]> FEAT-modin-project#1067: addressing review comments Signed-off-by: Alexander Myskov <[email protected]> FEAT-modin-project#1067: addressing review comments Signed-off-by: Alexander Myskov <[email protected]> FEAT-modin-project#1067: handle corner case with callable skiprows Signed-off-by: Alexander Myskov <[email protected]> FEAT-modin-project#1067: localize the work with rows_considered Signed-off-by: Alexander Myskov <[email protected]> FEAT-modin-project#1067: remove unnecessary changes Signed-off-by: Alexander Myskov <[email protected]> FEAT-modin-project#1067: correct docstrings Signed-off-by: Alexander Myskov <[email protected]> FEAT-modin-project#1067: drop complex `skiprows` after data import Signed-off-by: Alexander Myskov <[email protected]> FEAT-modin-project#1067: remove fallback to pandas case Signed-off-by: Alexander Myskov <[email protected]> FEAT-modin-project#1067: optimize rows skipping Signed-off-by: Alexander Myskov <[email protected]> FEAT-modin-project#1067: rollback unrelated changes, formatting Signed-off-by: Alexander Myskov <[email protected]> FEAT-modin-project#1067: fix incorrect rebase Signed-off-by: Alexander Myskov <[email protected]>
Signed-off-by: Alexander Myskov <[email protected]>
Signed-off-by: Alexander Myskov <[email protected]>
Signed-off-by: Alexander Myskov <[email protected]>
|
@devin-petersohn CI is green now. |
devin-petersohn
approved these changes
Aug 31, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces parallel implementation for
read_csvfunction with array-like and callableskiprowsparameter.skiprowsis handled differently depending on the parameter type:skiprowsis integer - rows will be eliminated from partitioned data in thepartitioned_filefunction and wouldn't be actually read.skiprowsis uniformly distributed array (for example [1, 2, 3]) - in this case rows within header and the first row fromskiprowswill be read in the first partition (if there are any) andskiprowswill be handled as integer.skiprowsis array, but not uniformly distributed, or callable - in this caseskiprowswill be dropped only after full dataset is imported (this is done because handling of such kind ofskiprowsis inefficient inpartitioned_filefunction).Note 1: this PR is continuation of #1932.
What do these changes do?
flake8 modinblack --check modingit commit -s