Skip to content

Fix broken async reference file system _cat_file method #1734

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Oct 23, 2024
Merged

Fix broken async reference file system _cat_file method #1734

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
bd2224f
Fix broken async reference file system _cat_file method
mpiannucci Oct 22, 2024
6bccdd9
Fix start and end constraints for asycn reference cat_file
mpiannucci Oct 23, 2024
b572966
Add test for async reference filesystem
mpiannucci Oct 23, 2024
fad218a
Set session for http filesystem in test
mpiannucci Oct 23, 2024
d2917f7
lint
mpiannucci Oct 23, 2024
640c84c
Use https specifically
mpiannucci Oct 23, 2024
a0f13c2
lint
mpiannucci Oct 23, 2024
a01ffda
more params
martindurant Oct 23, 2024
4ff7afe
lint
martindurant Oct 23, 2024
ec30e32
hammer
martindurant Oct 23, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion fsspec/implementations/reference.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -803,7 +803,9 @@ async def _cat_file(self, path, start=None, end=None, **kwargs):
return part_or_url[start:end]
protocol, _ = split_protocol(part_or_url)
try:
await self.fss[protocol]._cat_file(part_or_url, start=start, end=end)
return await self.fss[protocol]._cat_file(
part_or_url, start=start0, end=end0
)
except Exception as e:
raise ReferenceNotReachable(path, part_or_url) from e

Expand Down
29 changes: 29 additions & 0 deletions fsspec/implementations/tests/test_reference.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -513,6 +513,35 @@ def test_cat_file_ranges(m):
assert fs.cat_file("d", 1, -3) == other[4:10][1:-3]


@pytest.mark.asyncio
async def test_async_cat_file_ranges():
fsspec.get_filesystem_class("http").clear_instance_cache()
fss = fsspec.filesystem("https", asynchronous=True)
session = await fss.set_session()

fs = fsspec.filesystem(
"reference",
fo={
"version": 1,
"refs": {
"reference_time/0": [
"https://noaa-nwm-retro-v2-0-pds.s3.amazonaws.com/full_physics/2017/201704010000.CHRTOUT_DOMAIN1.comp",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to require the network; should this test get a vcr?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suppose we are assuming public archival S3 data should always be available.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

S3 might be, but the network won't (e.g., no network on Fedora builders, and probably many other distributions.)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True, we should have a pytest mark for tests that require network (at all). There will always be some. There are also other possible edges that have come up before, such as the default/only loopback interface being IPv6 rather than IPv4.

39783,
12,
],
},
},
fs={"https": fss},
remote_protocol="https",
asynchronous=True,
)

assert (
await fs._cat_file("reference_time/0") == b"x^K0\xa9d\x04\x00\x03\x13\x01\x0f"
)
await session.close()


@pytest.mark.parametrize(
"fo",
[
Expand Down
Loading