Remove cugraph Python library as a dependency

[rlratzel]

Closes #249

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[alexbarghi-nv]

Looks good, I think we can move this out of draft.

[jameslamb]

Came here looking for this because I was excited to see the note in @alexbarghi-nv's T5T saying the cugraph dependency had been removed from cugraph-pyg.

I can help you with a packaging-codeowners review and finishing this up.

Please also remove the dependency from the cugraph-pyg conda packages:

cugraph-gnn/conda/recipes/cugraph-pyg/recipe.yaml

Line 32 in 0f1f838

- cugraph =${{ minor_version }}

It also looks to me like there are more places where depends_on_cugraph could be removed in dependencies.yaml. If my comment about the use of import_optional and MissingModule in tests is correct, then it looks to me like depends_on_cugraph should just fully be removed from dependencies.yaml.

Otherwise, I see at least 2 specific cases that should be removed.

It looks to me like maybe the entire test_python group is unused and could be removed:

cugraph-gnn/dependencies.yaml

Lines 58 to 67 in 0f1f838

	test_python:
	output: none
	includes:
	- cuda_version
	- depends_on_cugraph
	- depends_on_cudf
	- depends_on_pytorch
	- depends_on_cuml
	- py_version
	- test_python_common

And I don't think pylibwholegraph needs cugraph for its tests:

cugraph-gnn/dependencies.yaml

Line 87 in 0f1f838

- depends_on_cugraph

[alexbarghi-nv]

@jameslamb can you re-review? I removed the unused dependencies as well.

[alexbarghi-nv]

@jameslamb I think I may have removed some of the dependency lists that are actually being used. The problem is it's not clear which are being used when I look at dependencies.yaml.

[jameslamb]

Thanks for the changes! The dependencies.yaml changes break the link between that file and wheels (i.e. remove automation of pyproject.toml updates), so some of them should be reverted.

I also think it's worth taking another pass over this branch looking for other places to remove. I pulled Rick's branch and ran this from the root:

git grep -E 'cugraph\.gnn`

And found more cases that should be updated. For example, there are still some lingering cugraph.gnn.cugraph_comms_init uses in tests and examples that I think you want to move to pylibcugraph.comms.cugraph_comms_init.

[jameslamb]

Generally, these extras like [test] should not contain anything that's already a hard runtime dependency of the project (in [project].dependencies).

Having overlap here means we can miss packaging issues like "hard runtime dependency not listed as such", which might not be caught in CI (where we always install the [test] extra).

But this issue of duplicates looks like it predates this PR (pylibwholegraph and torch-geometric pins are also duplicated across these lists). If you agree with my assessment, I can push a follow-up PR after this to fix that.

[alexbarghi-nv]

Yes, we should definitely fix that.

[jameslamb]

This entry may look unused if you do something like git grep py_build_libwholegraph, but it's not... the rapids-dependency-file-generator pre-commit hook uses this to populate python/libwholegraph/pyproject.toml, and rapids-build-backend uses it at build time to populate dependency metadata for wheels.

Everything in the diff here with a non-empty pyproject_dir: entry should be put back, please.

[alexbarghi-nv]

I think this is fixed.

[jameslamb]

It looks like this group is used right here:

cugraph-gnn/ci/test_python.sh

Line 74 in 9ecbc66

--file-key test_pylibwholegraph \

Removing this is why the tests are failing like this:

./ci/run_pylibwholegraph_pytests.sh: line 9: pytest: command not found

(build link)

I think that test_pylibwholegraph and test_cugraph_pyg should be restored (but depends_on_cugraph should be removed from test_pylibwholegraph because it looks like cugraph is not needed when testing pylibwholegraph).

git grep cugraph python/pylibwholegraph/
# (empty)

[jameslamb]

I'll look into why this didn't cause the build to fail earlier, at this part:

Traceback (most recent call last):
  File "/opt/conda/bin/rapids-dependency-file-generator", line 10, in <module>
    sys.exit(main())
  File "/opt/conda/lib/python3.10/site-packages/rapids_dependency_file_generator/_cli.py", line 136, in main
    make_dependency_files(
  File "/opt/conda/lib/python3.10/site-packages/rapids_dependency_file_generator/_rapids_dependency_file_generator.py", line 410, in make_dependency_files
    file_config = parsed_config.files[file_key]
KeyError: 'test_cugraph_pyg'
[rapids-conda-retry] timeout for conda operations: '45m'
Empty environment created at prefix: /opt/conda/envs/test_cugraph_pyg

https://github.com/rapidsai/cugraph-gnn/actions/runs/16944869038/job/48024049237?pr=271#step:10:863

That might be a bug in rapids-dependency-file-generator.

[jameslamb]

Ah, not a bug, this trapping prevents an early exit:

cugraph-gnn/ci/test_python.sh

Line 28 in 9ecbc66

EXITCODE=0

And looks like that's there intentionally so the job runs all test stuff instead of failing immediately. So nothing needs to be changes about rapids-dependency-file-generator ... but yes these groups need to be restored in dependencies.yaml.

[alexbarghi-nv]

@jameslamb I think we are good to go now.

[jameslamb]

Looks good! Thanks for doing this, really nice simplification to the RAPIDS dependency graph.

I'll put up a follow-up PR with the changes mentioned in https://github.com/rapidsai/cugraph-gnn/pull/271/files#r2274062952

[alexbarghi-nv]

/merge