Skip to content

fix rl unit test

fix rl unit test #5025

Triggered via push December 23, 2025 23:03
Status Success
Total duration 46m 44s
Artifacts 23

cicd-main.yml

on: push
is-not-external-contributor
7s
is-not-external-contributor
pre-flight  /  pre-flight
10s
pre-flight / pre-flight
cicd-wait-in-queue
4s
cicd-wait-in-queue
cicd-container-build
46s
cicd-container-build
cicd-parse-unit-tests
5s
cicd-parse-unit-tests
Matrix: cicd-unit-tests-latest
cicd-parse-integration-tests
9s
cicd-parse-integration-tests
Matrix: cicd-integration-tests-latest
Nemo_CICD_Test
8s
Nemo_CICD_Test
Coverage_Fake
0s
Coverage_Fake
Matrix: Coverage
cleanup-taint-node
0s
cleanup-taint-node
Fit to window
Zoom out
Zoom in

Deployment protection rules

Reviewers, timers, and other rules protecting deployments in this run
Event Environments Comment
ko3n1g
approved Dec 23, 2025
test

Automatically approved by queue manager

Annotations

43 errors and 13 warnings
cicd-container-build
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/pipeline_parallel/**/*.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/pipeline_parallel/**/*.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/data/**/*.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/data/**/*.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/models/**/*.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/models/**/*.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/dist_checkpointing/**/*.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/dist_checkpointing/**/*.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/test_model_configs.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/test_model_configs.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/dist_checkpointing/models/test_moe_experts.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/dist_checkpointing/models/test_moe_experts.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/test_fp8_param.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/test_fp8_param.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/dist_checkpointing/models/**/*.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/dist_checkpointing/models/**/*.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/transformer/moe/**/*.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/transformer/moe/**/*.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/distributed/fsdp/**/*.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/distributed/fsdp/**/*.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/transformer/**/*.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/transformer/**/*.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/dist_checkpointing/test_optimizer.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/dist_checkpointing/test_optimizer.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/**/*.py - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
tests/unit_tests/**/*.py - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
gpt/gpt3_mcore_te_tp1_pp4_vp1_dist_optimizer_overlap_grad_reduce_param_gather_overlap_optimizer - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
gpt/gpt3_mcore_te_tp1_pp4_vp1_dist_optimizer_overlap_grad_reduce_param_gather_overlap_optimizer - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_param_gather - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
gpt/gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_param_gather - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
gpt/gpt_static_inference_tp1_pp1_583m_logitsmatch - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
gpt/gpt_static_inference_tp1_pp1_583m_logitsmatch - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
hybrid/hybrid_mr_mcore_te_tp2_pp1_cp4_dgx_a100_1N8G - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
hybrid/hybrid_mr_mcore_te_tp2_pp1_cp4_dgx_a100_1N8G - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
common/moe_perf - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
common/moe_perf - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
hybrid/hybrid_static_inference_tp1_pp1_2B_logitsmatch - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
hybrid/hybrid_static_inference_tp1_pp1_2B_logitsmatch - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_resume_torch_dist_dist_optimizer - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
moe/gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_resume_torch_dist_dist_optimizer - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
gpt/gpt_grpo_tp1_pp1_dp8_583m_throughputtest_github - latest
DEPRECATION: --no-python-version-warning is deprecated. pip 25.1 will enforce this behaviour change. A possible replacement is to remove the flag as it's a no-op. Discussion can be found at https://github.com/pypa/pip/issues/13154
gpt/gpt_grpo_tp1_pp1_dp8_583m_throughputtest_github - latest
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
tests/unit_tests/pipeline_parallel/**/*.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/data/**/*.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/models/**/*.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/dist_checkpointing/**/*.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/test_model_configs.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/dist_checkpointing/models/test_moe_experts.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/test_fp8_param.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/dist_checkpointing/models/**/*.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/transformer/moe/**/*.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/distributed/fsdp/**/*.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/transformer/**/*.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/dist_checkpointing/test_optimizer.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.
tests/unit_tests/**/*.py - latest
No files were found with the provided path: logs. No artifacts will be uploaded.

Artifacts

Produced during runtime
Name Size Digest
NVIDIA~Megatron-LM~D7WBKQ.dockerbuild
101 KB
sha256:71e011c9e5c1a8492d7d64ecf4a706a1bea075fd112770e5dda1e93ec23112bd
coverage-unit-test-20473542078-04606edc-fd54-40bf-abe3-b8d8587e14fd
124 KB
sha256:9a8129b7699108f3f0764bce3d4db0dbc96ac2112c1dcfc02aec5d83247f5a33
coverage-unit-test-20473542078-06b71510-d2c2-478c-86a6-5631e3c4d3e3
131 KB
sha256:90fd6206f57736274a9c21edc80290f40a875263d37ff0220fa2bcdbc1787d53
coverage-unit-test-20473542078-281de9fc-05b6-42df-b28d-485a2657bb53
119 KB
sha256:07640ffed8dde8b608f291342372ae1f06c2aa4b5241386c4083ad4f8b11cb86
coverage-unit-test-20473542078-3085fe95-cf46-4d1b-af98-562537cf52f2
122 KB
sha256:568905f6e0a90a54514af6097f8585be991b52ce5b472e81df729ae672a26c89
coverage-unit-test-20473542078-365c8ded-8da0-4184-8514-2bf67c677568
123 KB
sha256:2d57b483fb67fc1cb3356bf23cefa59eb4d1dab713dc7cc8636bc017ba174e5d
coverage-unit-test-20473542078-51a2bce2-84f6-4831-9f08-18c45cd71951
125 KB
sha256:9c9311f84a23319c205c7d8a9e5d3d83f4c880d65df7f563f7de6b5a75ab6d69
coverage-unit-test-20473542078-5db6eff7-8b4a-4a1d-8306-d402045e6bcc
120 KB
sha256:491c7220c471b26dee5adc8510afcfe0e05130b8e28ac5341492172a26db645d
coverage-unit-test-20473542078-7a74eff8-9504-4cbd-95e4-9ce084119ac3
126 KB
sha256:392471ab1087e34040c86a3fda975ac4076cd117a2be3c1f14702e4cbd192258
coverage-unit-test-20473542078-90b13cb2-a577-45f5-abf1-444e2d55e26d
124 KB
sha256:491b0641a6669b2ad9e1feb80d250304e64a7e4f73096828bd3b21958613fc12
coverage-unit-test-20473542078-91bf6bfe-bdd4-49b7-8583-077b06285b14
123 KB
sha256:f829a2d607a44710e4ed723aef0428a65e0ec8de0789bfe1225f85ff300a9d6d
coverage-unit-test-20473542078-969043d4-1d8e-4e2a-aad4-1d4b1a9e223c
124 KB
sha256:19740b68d2fa8c6a4774cd473dc102fa561a4c307e5945820195b3b74e33bce9
coverage-unit-test-20473542078-a882ca3d-d146-42d5-8a72-fc9d9aa3bd33
122 KB
sha256:8ad2490bbbb253c0b76ff9dd0f190454c7fe1988cbebf5fc15ad6d7937fbeecf
coverage-unit-test-20473542078-f105725b-f1de-42c4-99f1-ce66303c7520
123 KB
sha256:effd8d00ca22a0f096933b20300cb24950223dce6bcfe89ad0566e38821c1a3e
coverage-unit-test-aggregated
28.4 KB
sha256:8d1b454909123cdccbc267e26975ba6668c68275e3f3f3f4727b0ed5cdbd3b3b
logs-gpt3_mcore_te_tp1_pp4_vp1_dist_optimizer_overlap_grad_reduce_param_gather_overlap_optimizer-20473542078-8b52d88b-047c-45e5-8573-c569943fad93
39.7 KB
sha256:eaff8f5508dac87158bc374d3baeebb86eaef455a48670fc67847b2689944854
logs-gpt3_mcore_te_tp4_pp1_resume_torch_dist_dist_optimizer_overlap_grad_reduce_param_gather-20473542078-6a99b3ef-a87b-4de9-8fc3-7cc98e0a8023
78.6 KB
sha256:956ffb1a4979eccecbeb4def23de5134dc906f626d8e8e8951cff94d552dbab4
logs-gpt3_moe_mcore_te_tp4_ep2_etp2_pp2_resume_torch_dist_dist_optimizer-20473542078-0fba53c8-b1c5-4308-841e-d4d099f03c6a
79.4 KB
sha256:f39c74166482226f08fd2a79d7ed63cf6c0735e0d6d512f5add6eb0937d30c44
logs-gpt_grpo_tp1_pp1_dp8_583m_throughputtest_github-20473542078-ecccdf4e-b8d6-48d4-98d9-c677ebe00973
303 KB
sha256:036e676e9ad06d698effad11b6026c3a95ee2b853edc5a1e1b9ff979e95928dd
logs-gpt_static_inference_tp1_pp1_583m_logitsmatch-20473542078-367c294d-2d9a-4ae6-a900-06f19d502e5d
29.8 KB
sha256:6ad5f092a54c93c919d132b84e53d81080206cea44153c546514f3713f46b722
logs-hybrid_mr_mcore_te_tp2_pp1_cp4_dgx_a100_1N8G-20473542078-57747752-48a1-4ac7-b7de-aeb0e46180cb
44.1 KB
sha256:1777dcba7e96a89c510fa49bfb2b5d88aec890a75f23c570255ff9b0323947f5
logs-hybrid_static_inference_tp1_pp1_2B_logitsmatch-20473542078-b8cbd320-48c0-4d99-8124-4d766e789779
27.8 KB
sha256:97070631c2dfe53e6e1633e38c49254b86b282806162fb55c430d82e74fb4ede
logs-moe_perf-20473542078-7194e549-f80a-4340-a982-444e92374c56
25.5 KB
sha256:426589a7729ec4a6b4f251b8ed1a3e65ee546014d978da4c6011f93579c4cf25