Adopt Python 3.12 #2184

dagardner-nv · 2025-02-19T00:18:45Z

Description

Update CI runners
Improve version parsing
Improve catching any type of exception on import (rather than just ImportError-derived)
Several package updates for Python 3.12 compatibility
Code updates to align with Python 3.12
Skip FAISS tests on Arm64 due to issue [BUG]: faiss-cpu on aarch64 target uses deprecated numpy API #2190 & upstream issue faiss fails to import on python==3.12 because of deprecated numpy.distutils facebookresearch/faiss#3936
Fix most documentation build warnings

Closes #2181

By Submitting this PR I confirm:

I am familiar with the Contributing Guidelines.
When the PR is ready for review, new or existing tests cover these changes.
When the PR is ready for review, the documentation is up to date with these changes.

…rpheus into david-25.02-x-25.02

…avid-25.02-x-25.02

…avid-rapids-25.02

… merged

…avid-rapids-25.02

…s#536 is merged" This reverts commit 9bceda0.

…mount for apt, pip and conda

Cleanup DOCA env vars Remove redundant calls to `apt update`, remove call to `apt upgrade`. Fix download location for `doca-host.deb` Consolidate installing of doca packagwes

willkill07 · 2025-03-05T19:40:01Z

/ok to test

Signed-off-by: Will Killian <[email protected]>

willkill07 · 2025-03-05T21:06:01Z

/ok to test

Signed-off-by: Will Killian <[email protected]>

willkill07 · 2025-03-05T22:24:18Z

/ok to test

Signed-off-by: Will Killian <[email protected]>

willkill07 · 2025-03-06T00:59:51Z

/ok to test

Signed-off-by: David Gardner <[email protected]>

dagardner-nv · 2025-03-06T01:21:25Z

/ok to test

Signed-off-by: Will Killian <[email protected]>

willkill07 · 2025-03-06T02:55:44Z

/ok to test

Signed-off-by: Will Killian <[email protected]>

willkill07 · 2025-03-06T04:26:14Z

/ok to test

Signed-off-by: Will Killian <[email protected]>

willkill07 · 2025-03-06T15:07:35Z

/ok to test

willkill07

Here are additional comments to aid in the review of this PR.

ci/conda/recipes/run_conda_build.sh

docs/source/conf.py

python/CMakeLists.txt

.github/workflows/ci_pipe.yml

tests/_utils/__init__.py

Signed-off-by: David Gardner <[email protected]>

dagardner-nv · 2025-03-06T16:41:44Z

/ok to test

efajardo-nv · 2025-03-07T19:35:57Z

Release container build doesn't work for me:

27.89 Adding in variants from internal_defaults
27.89 Adding in variants from /workspace/ci/conda/recipes/morpheus/conda_build_config.yaml
28.59 Error: bad character '=' in package version dependency 'librdkafka'
28.59 Perhaps you meant 'librdkafka =2.6.1'
------
Dockerfile:270
--------------------
 269 |
 270 | >>> RUN --mount=type=cache,id=workspace_cache,target=/workspace/.cache,sharing=locked \
 271 | >>>     --mount=type=cache,id=conda_pkgs,target=/opt/conda/pkgs,sharing=locked \
 272 | >>>     --mount=type=cache,id=pip_cache,target=/root/.cache/pip,sharing=locked \
 273 | >>>     # Install git-lfs before running the build to avoid errors during conda build
 274 | >>>     /opt/conda/bin/mamba install -y -n base -c conda-forge "git-lfs" &&\
 275 | >>>     source activate base &&\
 276 | >>>     git lfs install &&\
 277 | >>>     # Need to get around recent versions of git locking paths until they are deemed safe
 278 | >>>     git config --global --add safe.directory "*" &&\
 279 | >>>     # Change to the morpheus directory and build the conda package
 280 | >>>     cd ${MORPHEUS_ROOT_HOST} &&\
 281 | >>>     MORPHEUS_PYTHON_BUILD_STUBS=OFF CONDA_BLD_PATH=/opt/conda/conda-bld ./ci/conda/recipes/run_conda_build.sh morpheus
 282 |
--------------------
ERROR: failed to solve: process "/bin/bash -c /opt/conda/bin/mamba install -y -n base -c conda-forge \"git-lfs\" &&    source activate base &&    git lfs install &&    git config --global --add safe.directory \"*\" &&    cd ${MORPHEUS_ROOT_HOST} &&    MORPHEUS_PYTHON_BUILD_STUBS=OFF CONDA_BLD_PATH=/opt/conda/conda-bld ./ci/conda/recipes/run_conda_build.sh morpheus" did not complete successfully: exit code: 1

Signed-off-by: Will Killian <[email protected]>

willkill07 · 2025-03-07T20:04:41Z

/ok to test

willkill07 · 2025-03-10T17:37:00Z

/merge

* Update CI runners * Improve version parsing * Improve catching any type of exception on import (rather than just `ImportError`-derived) * Several package updates for Python 3.12 compatibility * Code updates to align with Python 3.12 * Skip FAISS tests on Arm64 due to issue #2190 & upstream issue facebookresearch/faiss#3936 * Fix most documentation build warnings Closes #2181 - I am familiar with the [Contributing Guidelines](https://github.com/nv-morpheus/Morpheus/blob/main/docs/source/developer_guide/contributing.md). - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - David Gardner (https://github.com/dagardner-nv) - Matthew Roeschke (https://github.com/mroeschke) - Will Killian (https://github.com/willkill07) Approvers: - Eli Fajardo (https://github.com/efajardo-nv) - Tad ZeMicheal (https://github.com/tzemicheal) URL: #2184 Signed-off-by: David Gardner <[email protected]>

mroeschke and others added 30 commits January 9, 2025 11:59

Move cudf helper dtype_from_column_view import from type to column

15e4f2d

First pass at Rapids 25.02 deps

cc0bab7

Merge branch 'cudf/dtype_from_column_view' of github.com:mroeschke/Mo…

028fc7d

…rpheus into david-25.02-x-25.02

WIP: Updating to 25.02

5ba2788

Update to match changes in the subword_tokenize function

1fde300

Update to reflect that null_mask was moved to pylibcudf

b4719b8

Update pynvml

290b243

Update stubs

f03448e

Formatting

521aea0

Revert temporary changes

2643d4e

Remove temporary changes and template mrc dep

c9b0afe

Merge branch 'branch-25.02' of github.com:nv-morpheus/Morpheus into d…

89c6206

…avid-25.02-x-25.02

Merge branch 'branch-25.06' of github.com:nv-morpheus/Morpheus into d…

d7cd381

…avid-rapids-25.02

Update child conda yamls based on changes in dependency.yaml

b9782e3

Quick find/replace pass at updating to CUDA 12.8

44e4084

more find/replace

bd7e9e4

Merge branch 'branch-25.06' of github.com:nv-morpheus/Morpheus into d…

edac4c7

…avid-rapids-25.02

Temporarily add the local channel, remove once MRC nv-morpheus#536 is…

9bceda0

… merged

Update stubs

f4897b7

Merge branch 'branch-25.06' of github.com:nv-morpheus/Morpheus into d…

b53a691

…avid-rapids-25.02

Adopt updated branch-25.06 of utilities

5288aeb

Fix cuda version

a2ed3f3

Revert "Temporarily add the local channel, remove once MRC nv-morpheu…

0a86a33

…s#536 is merged" This reverts commit 9bceda0.

Temporarily disable DOCA, currently not building

69de7eb

Avoid bloat in the CI container and speedup builds by using a cached …

164c22d

…mount for apt, pip and conda

Update container versions

8c47244

Work-around DOCA mft version conflicts

5879425

Cleanup DOCA env vars Remove redundant calls to `apt update`, remove call to `apt upgrade`. Fix download location for `doca-host.deb` Consolidate installing of doca packagwes

Update CI container versions

b4592ca

Temp disable doca

9ab5ac2

Pull in DOCA via an apt repo

2ce2d99

Align versions for packaging

2ca4c64

Signed-off-by: Will Killian <[email protected]>

willkill07 force-pushed the david-py-312-2181 branch from bb09517 to 2ca4c64 Compare March 5, 2025 21:05

Remove verbose logging; ensure torch for runtime packaging

5beb420

Signed-off-by: Will Killian <[email protected]>

Remove torch from file

e7f3984

Signed-off-by: Will Killian <[email protected]>

importlib.resources.path now returns a context manager

5ad89c7

Signed-off-by: David Gardner <[email protected]>

Fix outdated runner

8424452

Signed-off-by: Will Killian <[email protected]>

More importlib.resource.path fixes

c884848

Signed-off-by: Will Killian <[email protected]>

willkill07 mentioned this pull request Mar 6, 2025

Remove cudf_helpers.pyx patch for 24.12 bug #2201

Merged

Avoid deprecated importlib API; fix typo

0bb5101

Signed-off-by: Will Killian <[email protected]>

willkill07 reviewed Mar 6, 2025

View reviewed changes

Add comment explaining the PYBIND11_NO_ASSERT_GIL_HELD_INCREF_DECREF

aa950d7

Signed-off-by: David Gardner <[email protected]>

Fix whitespace issue in meta.yaml

7d6ed36

Signed-off-by: Will Killian <[email protected]>

efajardo-nv approved these changes Mar 10, 2025

View reviewed changes

github-project-automation bot moved this from Review - Ready for Review to Review - Approved in Morpheus Boards Mar 10, 2025

tzemicheal approved these changes Mar 10, 2025

View reviewed changes

rapids-bot bot merged commit adf167c into nv-morpheus:branch-25.06 Mar 10, 2025
16 checks passed

github-project-automation bot moved this from Review - Approved to Done in Morpheus Boards Mar 10, 2025

Adopt Python 3.12 #2184

Adopt Python 3.12 #2184

Uh oh!

Conversation

dagardner-nv commented Feb 19, 2025 • edited by willkill07 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

By Submitting this PR I confirm:

Uh oh!

willkill07 commented Mar 5, 2025

Uh oh!

willkill07 commented Mar 5, 2025

Uh oh!

willkill07 commented Mar 5, 2025

Uh oh!

willkill07 commented Mar 6, 2025

Uh oh!

dagardner-nv commented Mar 6, 2025

Uh oh!

willkill07 commented Mar 6, 2025

Uh oh!

willkill07 commented Mar 6, 2025

Uh oh!

willkill07 commented Mar 6, 2025

Uh oh!

willkill07 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dagardner-nv commented Mar 6, 2025

Uh oh!

efajardo-nv commented Mar 7, 2025

Uh oh!

willkill07 commented Mar 7, 2025

Uh oh!

willkill07 commented Mar 10, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

dagardner-nv commented Feb 19, 2025 •

edited by willkill07

Loading