Skip to content
This repository was archived by the owner on Nov 1, 2022. It is now read-only.

Lazy load ML dependencies #376

Merged

Conversation

donatasm
Copy link
Contributor

@donatasm donatasm commented Aug 29, 2022

Bringing back lazy loads reverted in #356, because of pip not supporting publishing packages referencing git commit hashes.

Support for Python 3.7 was added in scientific-python/lazy-loader#14, hence package is referenced by version now.

Not bringing back lazy loading pandas or numpy as they're always are loaded anyway, because of the image extensions.

Closes:

@linear
Copy link

linear bot commented Aug 29, 2022

LAY-3689 Lazilly load sdk dependencies that are not always needed

Description

part of https://github.com/layerai/roadmap/issues/19

Currently import layer is quite slow. We should reduce the number of large dependencies we load automatically and add a test to ensure we don't regress this if possible.

An inspiration can be taken in what we did last time we wanted to speed up the SDK - https://github.com/layerai/monorepo/pull/3146/files

Acceptance Criteria

  • [ ]


assert "torch" not in deps
assert "xgboost" not in deps
assert "keras" not in deps
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I'd probably assert something expected to be in deps as well, to make this more sturdy.

@donatasm donatasm force-pushed the donatas/lay-3689-lazilly-load-sdk-dependencies-that-are branch from 4107389 to 7fb1dcf Compare August 30, 2022 06:49
@donatasm donatasm force-pushed the donatas/lay-3689-lazilly-load-sdk-dependencies-that-are branch from 7fb1dcf to 91ec438 Compare August 30, 2022 07:05
@donatasm donatasm merged commit 93fc139 into main Aug 30, 2022
@donatasm donatasm deleted the donatas/lay-3689-lazilly-load-sdk-dependencies-that-are branch August 30, 2022 07:11
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants