Skip to content

Add implied assertion in WheelInfo - #145

Merged
hoodmane merged 1 commit into
pyodide:mainfrom
Carreau:assrt-http
Oct 24, 2024
Merged

Add implied assertion in WheelInfo#145
hoodmane merged 1 commit into
pyodide:mainfrom
Carreau:assrt-http

Conversation

@Carreau

@Carreau Carreau commented Oct 21, 2024

Copy link
Copy Markdown
Contributor

Some simple indexes (anaconda), only list the full-pathname of download URL (assuming same domain, which make sens).

WheelInfo.download does not work on this case at it tries to make the requests on the current page domain.

Thus we implicitely assume that URL start with http (well https:// would be better, but when you prototype local index could be http.)

The error can be weird if we let it propagate (bad ziplife as you try to decode often a 404 html page as ZIP).

This thus just add an assert at construction time to catch the error early.

It should in the end be pushed earler in the code (likely at parsing time), where we are likely to know the index URL and be able to resolve URLs at that time.

@hoodmane

Copy link
Copy Markdown
Member

Thanks!

@hoodmane

Copy link
Copy Markdown
Member

Looks like this gets hit in many tests though.

@Carreau

Carreau commented Oct 21, 2024

Copy link
Copy Markdown
Contributor Author

Hum, I'll dive into it more.

@Carreau
Carreau force-pushed the assrt-http branch 2 times, most recently from fbb7c55 to ab71634 Compare October 21, 2024 12:47
@Carreau

Carreau commented Oct 21, 2024

Copy link
Copy Markdown
Contributor Author

Ok, some of the conftest.py where mocking the url without https, let's see.

Comment thread tests/test_install.py

msg = "Access-Control-Allow-Origin"
with pytest.raises(ValueError, match=msg):
await micropip.install("htps://x.com/xxx-1.0.0-py3-none-any.whl")

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not entirely clear if the typo here was on purpose.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks pretty accidental to me. If it were on purpose you would expect a comment.

@Carreau

Carreau commented Oct 21, 2024

Copy link
Copy Markdown
Contributor Author

Worse case I'll push a better error in def download(self,...), but I prefer early failure when possible.

Comment thread micropip/wheelinfo.py Outdated
Comment thread tests/test_transaction.py
"path",
[
SNOWBALL_WHEEL,
f"/{SNOWBALL_WHEEL}" f"a/{SNOWBALL_WHEEL}",

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is a mistake, and a , is needed.

Comment thread tests/test_transaction.py Outdated
Comment thread tests/test_transaction.py
@pytest.mark.parametrize("protocol", ["https:", "file:", "emfs:", ""])
@pytest.mark.parametrize(
"protocol",
["http:", "https:", "file:", "emfs:", ""],

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have no clue what emfs is..

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It stands for "Emscripten file system". It's needed because file: points to files in the actual file system. Probably could use some comments and better documentation.

Comment thread tests/test_transaction.py Outdated
from micropip.transaction import WheelInfo

url = protocol + path
url = protocol + path if protocol else "file:///" + path

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not put file:/// in the parametrize decorator and do:

Suggested change
url = protocol + path if protocol else "file:///" + path
url = protocol + path

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's me I'm dumb I need to put that lower; I want to check that WheelInfo.from_url(url) parses things that have no protocol as local files.

@hoodmane hoodmane left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the maintenance work @Carreau! I have one suggestion otherwise lgtm.

Some simple indexes (anaconda), only list the full-pathname of download
URL (assuming same domain, which make sens).

WheelInfo.download does not work on this case at it tries to make the
requests on the current page domain.

Thus we implicitely assume that URL start with http (well https:// would
be better, but when you prototype local index could be http.)

The error can be weird if we let it propagate (bad ziplife as you try to
decode often a 404 html page as ZIP).

This thus just add an assert at construction time to catch the error
early and that we start with actual protocol.

It should in the end be pushed earler in the code (likely at parsing
time), where we are likely to know the index URL and be able to resolve
URLs at that time.

I think there was also a missing comma in test parametrisations, and I
added a !r in a few places as I had some error with empty filenames,
without quotes, so hard to see.

The conftest was also update to explicitely pass a fake URL instead of
just a filename.

I'm not entirely sure we should allow constructing URLs with things that
don't have a protocol though, as this can be confusing.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants