Add add_mock_package by joemarshall · Pull Request #26 · pyodide/micropip

joemarshall · 2022-11-17T15:17:19Z

Added a method to micropip to allow easy mocking of packages (for use in jupyterlite / pyodide)

It takes package name and version, along with a list of implemented modules and optionally their code if needed.

for more information, see https://pre-commit.ci

…ropip into add_package_mock

for more information, see https://pre-commit.ci

rth · 2022-11-17T20:20:35Z

+            path_parts = n.split(".")
+            dir_path = Path(site_packages, *path_parts)
+            os.makedirs(dir_path, exist_ok=True)
+            init_file = dir_path / "__init__.py"


Do we actually need to create these physical files?

I was thinking we could maybe directly initialize sys.modules with the mapping from modules: dict[str, ModuleType | str]? That way it's also easy to use MagicMock as a module or any other custom object defined at runtime to make a given dependency work.

Though yes, the API to create modules dynamically via importlib.util.module_from_spec or by inheriting from ModuleType isn't very straightforward.

Maybe making another custom Finder/Loader would be a good way to go. The API for that isn't too bad.

I actually wrote it initially using mock objects and putting them into sys.modules, but I reverted to writing them out to files. I did it like that because it is really simple and reliable, things like file paths etc. just work. It also works nicely in terms of loading modules, in that init code is only run on actual import. Oh and in the case that you're running somewhere with persistent file system, your mock modules continue to exist on reload of python.

It's a pretty minor change - you just create a module and exec the code into it in the loader, I can put that back in if you reckon the possibility of using mock objects makes sense.

Actually I think rather than take an object it needs to take a callable which takes a module object and does whatever needs to be done on it to make it. I.e. the equivalent of loader.exec_module

That way you can mock module initialisation correctly.

One further thought on this - at some point I want to implement some kind of mega wheel option, to bundle a set of wheels and all deps into a single wheel with no deps. For that use case keeping persistent mock modules would be nice. If it's okay I'd like to keep that as an option in micropip?

It would also be nice for any point that pyodide is run in a persistent file system.

I'm thinking that a flag "persistent" could be added (which only works for string modules) and then the behaviour with finder and loader could be used in the case that the flag isn't set.

ryanking13 · 2022-11-23T01:39:09Z

It would also be nice for any point that pyodide is run in a persistent file system.

I think this is reasonable, as having a physical file would be a most easy way to keep the state of installed packages or to bundle packages (for example, in pyodide-pack) for later use.

We can change the implementation of add_mock_package later if we find a better way to deal with this, but I'm +1 with this approach for now.

ryanking13 · 2022-11-23T01:41:33Z

+            s = dedent(s)
+            path_parts = n.split(".")
+            dir_path = Path(site_packages, *path_parts)
+            os.makedirs(dir_path, exist_ok=True)


Maybe we should raise error if a directory exists, because it would mean that the package is already installed somehow.

ryanking13 · 2022-11-23T01:42:50Z

    return packages
+
+
+def add_mock_package(name, version, *, modules=None):


Could you please add type hints here?

ryanking13 · 2022-11-23T01:43:29Z

+            os.makedirs(dir_path, exist_ok=True)
+            init_file = dir_path / "__init__.py"
+            with open(init_file, "w") as f:
+                f.write(s)


I think we should keep the list of mocked modules, so it can be accessed via e.g. micropip.mock_modules

Added list_mock_modules instead - which uses module installer metadata to list modules it has installed

for more information, see https://pre-commit.ci

…ropip into add_package_mock

for more information, see https://pre-commit.ci

…ropip into add_package_mock

for more information, see https://pre-commit.ci

…ropip into add_package_mock

for more information, see https://pre-commit.ci

joemarshall · 2022-11-23T16:37:42Z

@ryanking13 @rth I've added support for non-persistent mocks, and made the tests work in pyodide. I think I've fixed the things from the reviews above.

There's a bit more code in there to do the in-memory mocking, which is probably worth a look over before merging.

…ropip into add_package_mock

joemarshall · 2022-11-23T22:45:45Z

Oh and it supports listing and removing mocks also btw.

ryanking13

Thanks, @joemarshall. Having a persistent parameter seems good to me. Overall LGTM, if @rth or @hoodmane is also +1 with this, I'll merge.

Could you please also update the changelog? Probably we can follow the format of KeepAChangelog

ryanking13 · 2022-11-24T04:21:32Z

+            with open(init_file, "w") as f:
+                f.write(s)
+        with open(installer_file, "w") as f:
+            f.write("micropip mock package")


Maybe move these hard coded INSTALLER types into a variables?

ryanking13 · 2022-11-24T04:43:02Z

+
+
+_finder = _MockModuleFinder()
+sys.meta_path = [_finder] + sys.meta_path


I'm not sure we want an importing micropip to have a side effect of modifying sys.meta_path, though it does not have any effect when no mock packages are registered.

Perhaps modify sys.meta_path when add_in_memory_distribution is first called?

ryanking13 · 2022-11-24T04:45:02Z

+
+    from micropip._micropip import add_mock_package
+
+    with TemporaryDirectory() as tmpdirname:


There is a tmp_path fixture for this purpose.

ryanking13 · 2022-11-24T04:48:41Z

+    # check package removes okay
+    _micropip.remove_mock_package("micropip_test_bob")
+    del micropip_bob_mod
+    try:


Maybe use with pytest.raises(...) instead?

ryanking13 · 2022-11-24T04:48:59Z

+    # check package removes okay
+    _micropip.remove_mock_package("micropip_test_bob")
+    del micropip_bob_mod
+    try:


Same here (pytest.raises)

for more information, see https://pre-commit.ci

joemarshall · 2022-11-24T11:17:25Z

Okay, done those bits and pieces now and changelog. Good spot about the sys.meta_path thing, that was silly.

…ropip into add_package_mock

rth

Thanks a lot for this work @joemarshall and @ryanking13 for the review!

rth · 2022-11-26T10:02:56Z

+       }
+       ``
+
+    Parameters


I think there is an indentation issue in this docstring, but we can fix it once we start actually building docs for it.

ryanking13 · 2022-11-28T02:07:55Z

I guess we should release 0.1.1 (or 0.2.0) with this. Though probably we need to start building docs for micropip before making a release.

joemarshall and others added 5 commits November 17, 2022 15:11

added support for creating mock packages

525d0d7

Merge branch 'main' of github.com:pyodide/micropip into add_package_mock

111046f

reverted typo change

799560d

removed unneeded imports

f66370c

[pre-commit.ci] auto fixes from pre-commit.com hooks

43fa3f3

for more information, see https://pre-commit.ci

joemarshall mentioned this pull request Nov 17, 2022

Make usable build for pyodide tradingstrategy-ai/trade-executor#56

Open

11 tasks

joemarshall and others added 6 commits November 17, 2022 15:34

remembered to run pre-commit hooks

16739f5

added test

0a6471a

removed debug print

5e5b0d9

merged and linted

0eb9803

Merge branch 'add_package_mock' of https://github.com/joemarshall/mic…

20b74f0

…ropip into add_package_mock

[pre-commit.ci] auto fixes from pre-commit.com hooks

c76a8e6

for more information, see https://pre-commit.ci

rth changed the title ~~add_package_mock~~ add_mock_package Nov 17, 2022

rth reviewed Nov 17, 2022

View reviewed changes

ryanking13 reviewed Nov 23, 2022

View reviewed changes

joemarshall and others added 14 commits November 23, 2022 12:08

dist-info metadata for easier package finding etc.

3418848

[pre-commit.ci] auto fixes from pre-commit.com hooks

ea8c424

for more information, see https://pre-commit.ci

added support for temporary modules

6628396

default to non-persistent mocking

9cb4ac2

in memory sub packages fixes

d6aab95

Merge branch 'add_package_mock' of https://github.com/joemarshall/mic…

005efec

…ropip into add_package_mock

test in pyodide with correct (local) version of micropip

a12cd86

[pre-commit.ci] auto fixes from pre-commit.com hooks

10c7f84

for more information, see https://pre-commit.ci

fixed missing package in test

5d96d84

Merge branch 'add_package_mock' of https://github.com/joemarshall/mic…

786e5e8

…ropip into add_package_mock

dedent in memory modules

35539ef

[pre-commit.ci] auto fixes from pre-commit.com hooks

107004d

for more information, see https://pre-commit.ci

simpler std output capture

ac8bc37

Merge branch 'add_package_mock' of https://github.com/joemarshall/mic…

98b9192

…ropip into add_package_mock

joemarshall and others added 4 commits November 23, 2022 15:46

invalidate importlib cache for pyodide

e83cb59

testing different order to see what is broken

4d719c2

[pre-commit.ci] auto fixes from pre-commit.com hooks

fd037ae

for more information, see https://pre-commit.ci

fixed tests for pyodide

3b77d88

Merge branch 'add_package_mock' of https://github.com/joemarshall/mic…

782b6af

…ropip into add_package_mock

ryanking13 approved these changes Nov 24, 2022

View reviewed changes

joemarshall and others added 3 commits November 24, 2022 10:58

review fixes

d34ff58

[pre-commit.ci] auto fixes from pre-commit.com hooks

5e8f5e6

for more information, see https://pre-commit.ci

added changelog

2c34df0

Merge branch 'add_package_mock' of https://github.com/joemarshall/mic…

9c16b28

…ropip into add_package_mock

rth approved these changes Nov 26, 2022

View reviewed changes

rth changed the title ~~add_mock_package~~ Add add_mock_package Nov 26, 2022

rth merged commit 67527ee into pyodide:main Nov 26, 2022

rth mentioned this pull request Nov 26, 2022

Enhancement ideas from JupyterLite #16

Open

rth mentioned this pull request Nov 29, 2022

Xonsh calls unavailable call getrusage resource.getrusage(resource.RUSAGE_SELF)[:2] pyodide/pyodide#3302

Open

		return packages


		def add_mock_package(name, version, *, modules=None):



		_finder = _MockModuleFinder()
		sys.meta_path = [_finder] + sys.meta_path


		from micropip._micropip import add_mock_package

		with TemporaryDirectory() as tmpdirname:

Uh oh!

Uh oh!

Conversation

joemarshall commented Nov 17, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ryanking13 commented Nov 23, 2022

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ryanking13 Nov 23, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ryanking13 Nov 23, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joemarshall commented Nov 23, 2022

Uh oh!

joemarshall commented Nov 23, 2022

Uh oh!

ryanking13 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joemarshall commented Nov 24, 2022

Uh oh!

rth left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ryanking13 commented Nov 28, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ryanking13 Nov 23, 2022 •

edited

Loading

ryanking13 Nov 23, 2022 •

edited

Loading