Skip to content

Conversation

valeriupredoi
Copy link
Collaborator

@valeriupredoi valeriupredoi commented Feb 26, 2024

Description

This is a sandbox testing a slightly modded Kercunk that kerchunks inly a desired Dataset - the good folk at Kerchunk are already investigating means of implementing such an option in Kerchunk in fsspec/kerchunk#424 - and some timing test results can be seen in fsspec/kerchunk#424 (comment) (after one sets default_cache_type to first as Martin recommended)

Label: needs new Kerchunk

Update as of February 29

With the Kerchunk PR now merged fsspec/kerchunk#424 and the issues related to the newer Kerchunk functionality now ironed out, this is ready for a merge when kerchunk release the new version that contains that PR. Test results are very promising:

Test time / time it spends before going to remote Reductionist
--------------------------------------------------------------
4.87        (0.71)
5.05        (1.06)
7.02        (3.03)
4.84        (0.81)
4.72        (0.74)
4.77        (0.73)
4.75        (0.78)

So all the time spent in PyActiveStorage is sub-1s! This is for the tests/test_compression_remote_reductionist.py::test_compression_and_filters_cmip6_forced_s3_from_local_bigger_file_v1 test.

@valeriupredoi valeriupredoi marked this pull request as draft February 26, 2024 15:08
@valeriupredoi valeriupredoi added the testing testing duh label Feb 27, 2024
@valeriupredoi valeriupredoi marked this pull request as ready for review February 29, 2024 21:09
@valeriupredoi valeriupredoi added enhancement New feature or request and removed needs-new-dependency labels Jan 20, 2025
@valeriupredoi valeriupredoi changed the title [DRAFT] Test optimal kerchunk [OBSOLETE] Test optimal kerchunk Mar 25, 2025
@valeriupredoi valeriupredoi changed the title [OBSOLETE] Test optimal kerchunk [OBSOLETE but still functional for testing] Test optimal kerchunk Mar 25, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request testing testing duh
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant