ROCM 5.3 build has been broken for a while

[malfet]

See https://hud.pytorch.org/hud/pytorch/builder/main/1?per_page=50&name_filter=rocm%20(5.3)

IMO it should either be disabled or fixed

[malfet]

Also, why it takes 3+ hours to finish?

[jeffdaily]

We have to build magma from source and miopen from source. That's a bulk of the time spent.

I can't seem to view the failure logs?

[jeffdaily]

If it is timing out can we increase the timeout?

[jeffdaily]

CPack is failing to copy files and create directories during the MIOpen build.

[jeffdaily]

##[warning]You are running out of disk space. The runner will stop working when the machine runs out of disk space. Free space left: 43 MB

[jithunnair-amd]

Also, why it takes 3+ hours to finish?

From https://github.com/pytorch/builder/actions/runs/4029201280/jobs/6926855728 (rocm5.2 manywheel docker build):
RUN bash build_scripts/build.sh && rm -r build_scripts: 18min
RUN bash ./install_rocm_magma.sh && rm install_rocm_magma.sh: 71min
RUN bash ./install_miopen.sh 5.2 && rm install_miopen.sh: 67min

[malfet]

We have to build magma from source and miopen from source. That's a bulk of the time spent.

Why not prebuilt magma for rocm? Same as it is done for magma-cuda?

I can't seem to view the failure logs?
If runner is killed due to a timeout (or running out of disk-space) then logs can not be retrieved. We can probably increase available runner space (or move rocm builds to a different runners)

But 3+ hours build time is bad, imo...

[jithunnair-amd]

But 3+ hours build time is bad, imo...

I totally agree, and we are working in the direction of removing the need to build both Magma and MIOpen from source for ROCm PyTorch wheels