-
Notifications
You must be signed in to change notification settings - Fork 23
Fix excessive regrid memory use #624
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
FYI I just added one minimal commit to resolve the lone trivial merge conflict (new Changelog entries to combine) so I can review without it being in the way. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Clever approach which I agree with as covered in the notes of the opening comment. The implementation is sound and doesn't call for any new testing. The performance improves very significantly in test cases I played around with and did basic profiling on, so passes with flying colours in that respect.
Therefore overall this is all good, though I've raised some minor comments and a question. Please consider those and then merge when ready. Thanks.
Co-authored-by: Sadie L. Bartholomew <[email protected]>
Co-authored-by: Sadie L. Bartholomew <[email protected]>
Co-authored-by: Sadie L. Bartholomew <[email protected]>
Co-authored-by: Sadie L. Bartholomew <[email protected]>
Fixes #623
Notes
numpy
array sincescipy
sparse arrays do not support masks.scipy
sparse array, but creates a separate boolean mask to applied to the regridded data.cf/regrid/regrid.py
pre-creates the sparse matrix representation in a returned regrid operator. This makes sense because the purpose of returning the operator is for efficiency in repeated future uses, so it makes sense not to have to do this every time.coo_array
tocsr_array
is because the latter is much more efficient at row slicing.