[CI] Build and test rocm Python packages as part of ci.yml

[ScottTodd]

We've had issues like #1347 and #1552 go undetected since we only install the rocm packages we built and run rocm-sdk test during PyTorch release builds:

• TheRock/.github/workflows/test_pytorch_wheels.yml

  Lines 115 to 117 in bee658b

  	- name: Run rocm-sdk sanity tests
  	run: |
  	rocm-sdk test

• TheRock/external-builds/pytorch/build_prod_wheels.py

  Lines 720 to 728 in bee658b

  	print("+++ Sanity checking installed torch (unavailable is okay on CPU machines):")
  	sanity_check_output = capture(
  	[sys.executable, "-c", "import torch; print(torch.cuda.is_available())"],
  	cwd=tempfile.gettempdir(),
  	)
  	if not sanity_check_output:
  	raise RuntimeError("torch package sanity check failed (see output above)")
  	else:
  	print(f"Sanity check output:\n{sanity_check_output}")

The ci.yml workflow that we run on pull requests and merged commits can do more than just build and test native ROCm packages, it could build and test ROCm Python packages too. Looking at these Windows workflows, we could merge them or have one reuse the other:

• https://github.com/ROCm/TheRock/blob/main/.github/workflows/build_windows_packages.yml (for CI)
• https://github.com/ROCm/TheRock/blob/main/.github/workflows/release_windows_packages.yml (for releases)

The build step that is missing is:

      - name: Build Python Packages
        run: |
          python ./build_tools/build_python_packages.py \
            --artifact-dir=${{ env.BUILD_DIR }}/artifacts \
            --dest-dir=${{ env.BUILD_DIR }}/packages \
            --version=${{ needs.setup_metadata.outputs.version }}

See a recent release workflow run for an example of that: https://github.com/ROCm/TheRock/actions/runs/17904146218/job/50902443944#step:15:66. Note that it takes about 7 minutes, after a 1h40m build (62% cache hits)

Beyond just building, we probably want to upload wheels to some sort of dev release bucket and then install them for testing on machines with GPUs, and then also trigger dev PyTorch builds and tests eventually too.

[ScottTodd]

How about:

• In build_*_packages.yml we build python packages too, and upload dev python packages
• (A) We add a "python_packages" test component:
  • https://github.com/ROCm/TheRock/blob/main/.github/workflows/test_packages.yml
  • https://github.com/ROCm/TheRock/blob/main/build_tools/github_actions/fetch_test_configurations.py
  • https://github.com/ROCm/TheRock/tree/main/build_tools/github_actions/test_executable_scripts
  • https://github.com/ROCm/TheRock/blob/main/.github/actions/setup_test_environment/action.yml install dev python packages using setup_venv.py, skip install_rocm_from-artifacts.py step?
• (B) We add a "python_packages test" that doesn't use that matrix or the setup action

[ScottTodd]

In build_*_packages.yml we build python packages too, and upload dev python packages

Actually, longer term (and maybe now), we want this to be a follow-on job that downloads the artifacts, rather than something inlined into the build workflow. See #1522. That would also be significantly faster to iterate on since we wouldn't need a 40m-2h source build each time for a 6m package build.

[marbre]

• In build_*_packages.yml we build python packages too, and upload dev python packages

I agree in general but this needs a better story for our buckets as it will get quite messy. I am about to file an issue as some of our buckets should re-organized anyway. I will take this into consideration.

[ScottTodd]

• In build_*_packages.yml we build python packages too, and upload dev python packages

I agree in general but this needs a better story for our buckets as it will get quite messy. I am about to file an issue as some of our buckets should re-organized anyway. I will take this into consideration.

How about we upload to therock-artifacts/${WORKFLOW_ID}-${PLATFORM}/packages/python? I'm thinking that every CI run should keep its files all together:

logs/
  build/
  test/
artifacts/
packages/
  python/
  ubuntu/
  windows/

If we wanted, we could create a central pip index page with pointers into the subfolders of that bucket, but each run could also just have its own isolated index for manual download/install (or --extra-index-url and not --index-url).

I think I can sidestep that for now by having a "build and test packages" workflow that runs on test machines instead of a pipeline of build machine --> test machines. The Python packages are pretty fast to build compared to the artifacts.

[ScottTodd]

I'm trying to reuse the "setup_test_environment" composite action for Python package builds and this code still only fetches some artifacts:

TheRock/.github/actions/setup_test_environment/action.yml

Lines 54 to 68 in 8206f4c

	- name: Download and Unpack Artifacts
	shell: bash
	env:
	OUTPUT_ARTIFACTS_DIR: ${{ inputs.OUTPUT_ARTIFACTS_DIR }}
	ARTIFACT_RUN_ID: ${{ inputs.ARTIFACT_RUN_ID }}
	AMDGPU_FAMILIES: ${{ inputs.AMDGPU_FAMILIES }}
	FETCH_ARTIFACT_ARGS: ${{ inputs.FETCH_ARTIFACT_ARGS }}
	GITHUB_TOKEN: ${{ github.token }}
	IS_PR_FROM_FORK: ${{ inputs.IS_PR_FROM_FORK }}
	run: |
	python ./build_tools/install_rocm_from_artifacts.py \
	--run-id=${ARTIFACT_RUN_ID} \
	--amdgpu-family=${AMDGPU_FAMILIES} \
	--output-dir=${OUTPUT_ARTIFACTS_DIR} \
	${FETCH_ARTIFACT_ARGS}

Here are logs from my latest test build: https://github.com/ScottTodd/TheRock/actions/runs/17958277545/job/51075156374#step:4:221

We should really complete refactoring work in https://github.com/ROCm/TheRock/blob/main/build_tools/fetch_artifacts.py and https://github.com/ROCm/TheRock/blob/main/build_tools/install_rocm_from_artifacts.py . That was started in #772 and #926.

[ScottTodd]

I mostly figured out the artifact fetching and workflow setup for a standalone "build python packages" workflow that consumes artifacts from a CI run. Latest test run (just missing some version metadata): https://github.com/ScottTodd/TheRock/actions/runs/17961548042/job/51085710507

I don't think I'll end up reusing setup_test_environment directly, but I did tidy up a few parts of those scripts while trying to use them. I'm aiming to have a good model workflow that other packaging workflows for #1522 can reference.

[ScottTodd]

Here's my latest test workflow run: https://github.com/ScottTodd/TheRock/actions/runs/17992588251/job/51185590614

That ran in my fork on a standard github-hosted runner. The workflow

1. Downloads artifacts from AWS using the run id for a workflow in this repository
2. Builds ROCm Python packages
3. Installs those packages and runs rocm-sdk test (which successfully checks that DLLs are valid, even with no GPU)

I have a similar workflow for Linux that uses the manylinux dockerfile to build, though that runs out of disk space on standard github-hosted runners. I think I can prune the downloaded artifacts to exclude test and dbg artifacts instead of "all".

Here's the main code:

• https://github.com/ScottTodd/TheRock/blob/ci-python-packages-3/.github/workflows/build_windows_python_packages.yml
• https://github.com/ScottTodd/TheRock/blob/ci-python-packages-3/.github/workflows/build_portable_linux_python_packages.yml

The workflows I have so far are probably useful already as part of ci.yml, but a few things are missing:

1. Constructing a version string based on the date, git hash, version file, etc. could be moved into a script or reusable workflow so these workflows can use that same logic via workflow_dispatch
2. I'm not yet uploading the packages anywhere. I'll discuss that with @marbre
3. The workflow could trigger a "test python packages" workflow that installs the uploaded packages and runs tests on GPU machines, then we can also trigger "build pytorch packages" off of the "build python packages" workflows too for Release pipelines should save artifacts and not just monolithic build results #1522
4. I hacked on build_tools/fetch_artifacts.py and build_tools/github_actions/github_actions_utils.py so retrieve_bucket_info() would be compatible with running workflows in a fork but downloading artifacts from this main repo. Need to find a good balance of defaults / configuration there (add more env vars?)