Skip to content

Updated HPU workflow file as per torchbench execution on HPU #20

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

arathi-hlab
Copy link

hi , all i tried to run torchbench accuracy test in eval mode and i tried locally with this change here is the result i got
root@arathi-h7ju-g3-l-tfjob-worker-0:~# python benchmark/test.py -v -k eval_hpu
Calling add_step_closure function does not have any effect. It's lazy mode only functionality. (warning logged once)
Calling mark_step function does not have any effect. It's lazy mode only functionality. (warning logged once)
Calling iter_mark_step function does not have any effect. It's lazy mode only functionality. (warning logged once)
test_BERT_pytorch_eval_hpu (main.TestBenchmark) ... ok
test_Background_Matting_eval_hpu (main.TestBenchmark) ... skipped 'Method eval on hpu is not implemented because "", skipping...'
test_LearningToPaint_eval_hpu (main.TestBenchmark) ... ok
test_Super_SloMo_eval_hpu (main.TestBenchmark) ... ok
test_alexnet_eval_hpu (main.TestBenchmark) ... ok
test_basic_gnn_edgecnn_eval_hpu (main.TestBenchmark) ... ok
test_basic_gnn_gcn_eval_hpu (main.TestBenchmark) ... ok
test_basic_gnn_gin_eval_hpu (main.TestBenchmark) ... ok
test_basic_gnn_sage_eval_hpu (main.TestBenchmark) ... ok
test_cm3leon_generate_eval_hpu (main.TestBenchmark) ... ok
test_dcgan_eval_hpu (main.TestBenchmark) ... ok
test_demucs_eval_hpu (main.TestBenchmark) ... ok
test_densenet121_eval_hpu (main.TestBenchmark) ... ok
test_detectron2_fasterrcnn_r_101_c4_eval_hpu (main.TestBenchmark) ... ok
test_detectron2_fasterrcnn_r_101_dc5_eval_hpu (main.TestBenchmark) ... ok
test_detectron2_fasterrcnn_r_101_fpn_eval_hpu (main.TestBenchmark) ... ok
test_detectron2_fasterrcnn_r_50_c4_eval_hpu (main.TestBenchmark) ... ok
test_detectron2_fasterrcnn_r_50_dc5_eval_hpu (main.TestBenchmark) ... ok
test_detectron2_fasterrcnn_r_50_fpn_eval_hpu (main.TestBenchmark) ... ok
test_detectron2_fcos_r_50_fpn_eval_hpu (main.TestBenchmark) ... ERROR
test_detectron2_maskrcnn_eval_hpu (main.TestBenchmark) ... ok
test_detectron2_maskrcnn_r_101_c4_eval_hpu (main.TestBenchmark) ... ok
test_detectron2_maskrcnn_r_101_fpn_eval_hpu (main.TestBenchmark) ... ok
test_detectron2_maskrcnn_r_50_c4_eval_hpu (main.TestBenchmark) ... ok
test_detectron2_maskrcnn_r_50_fpn_eval_hpu (main.TestBenchmark) ... ok
test_dlrm_eval_hpu (main.TestBenchmark) ... ok
test_doctr_det_predictor_eval_hpu (main.TestBenchmark) ... ERROR
test_doctr_reco_predictor_eval_hpu (main.TestBenchmark) ... ERROR
test_drq_eval_hpu (main.TestBenchmark) ... ok
test_fastNLP_Bert_eval_hpu (main.TestBenchmark) ... ERROR
test_functorch_dp_cifar10_eval_hpu (main.TestBenchmark) ... ok
test_functorch_maml_omniglot_eval_hpu (main.TestBenchmark) ... ok
test_hf_Albert_eval_hpu (main.TestBenchmark) ... ok
test_hf_Bart_eval_hpu (main.TestBenchmark) ... ok
test_hf_Bert_eval_hpu (main.TestBenchmark) ... ok
test_hf_Bert_large_eval_hpu (main.TestBenchmark) ... ok
test_hf_BigBird_eval_hpu (main.TestBenchmark) ... ok
test_hf_DistilBert_eval_hpu (main.TestBenchmark) ... ok
test_hf_GPT2_eval_hpu (main.TestBenchmark) ... ok
test_hf_GPT2_large_eval_hpu (main.TestBenchmark) ... ok
test_hf_Longformer_eval_hpu (main.TestBenchmark) ... ok
test_hf_Reformer_eval_hpu (main.TestBenchmark) ... ok
test_hf_Roberta_base_eval_hpu (main.TestBenchmark) ... ok
test_hf_T5_base_eval_hpu (main.TestBenchmark) ... ok
test_hf_T5_eval_hpu (main.TestBenchmark) ... ok
test_hf_T5_generate_eval_hpu (main.TestBenchmark) ... ok
test_hf_T5_large_eval_hpu (main.TestBenchmark) ... ok
test_hf_Whisper_eval_hpu (main.TestBenchmark) ... ok
test_hf_clip_eval_hpu (main.TestBenchmark) ... ok
test_hf_distil_whisper_eval_hpu (main.TestBenchmark) ... ok
test_lennard_jones_eval_hpu (main.TestBenchmark) ... ok
test_llama_eval_hpu (main.TestBenchmark) ... ERROR
test_llama_v2_7b_16h_eval_hpu (main.TestBenchmark) ... ok
test_llava_eval_hpu (main.TestBenchmark) ... ok
test_maml_eval_hpu (main.TestBenchmark) ... ok
test_maml_omniglot_eval_hpu (main.TestBenchmark) ... ok
test_microbench_unbacked_tolist_sum_eval_hpu (main.TestBenchmark) ... ok
test_mnasnet1_0_eval_hpu (main.TestBenchmark) ... ok
test_mobilenet_v2_eval_hpu (main.TestBenchmark) ... ok
test_mobilenet_v3_large_eval_hpu (main.TestBenchmark) ... ok
test_moco_eval_hpu (main.TestBenchmark) ... skipped 'Method eval on hpu is not implemented because "hpu not supported", skipping...'
test_moondream_eval_hpu (main.TestBenchmark) ... ok
test_nanogpt_eval_hpu (main.TestBenchmark) ... ok
test_nvidia_deeprecommender_eval_hpu (main.TestBenchmark) ... ERROR
test_opacus_cifar10_eval_hpu (main.TestBenchmark) ... ok
test_phlippe_densenet_eval_hpu (main.TestBenchmark) ... ok
test_phlippe_resnet_eval_hpu (main.TestBenchmark) ... ok
test_pyhpc_equation_of_state_eval_hpu (main.TestBenchmark) ... ok
test_pyhpc_isoneutral_mixing_eval_hpu (main.TestBenchmark) ... ok
test_pyhpc_turbulent_kinetic_energy_eval_hpu (main.TestBenchmark) ... ok
test_pytorch_CycleGAN_and_pix2pix_eval_hpu (main.TestBenchmark) ... ok
test_pytorch_stargan_eval_hpu (main.TestBenchmark) ... ok
test_pytorch_unet_eval_hpu (main.TestBenchmark) ... ok
test_resnet152_eval_hpu (main.TestBenchmark) ... ok
test_resnet18_eval_hpu (main.TestBenchmark) ... ok
test_resnet50_eval_hpu (main.TestBenchmark) ... ok
test_resnext50_32x4d_eval_hpu (main.TestBenchmark) ... ok
test_sam_eval_hpu (main.TestBenchmark) ... ok
test_sam_fast_eval_hpu (main.TestBenchmark) ... ERROR
test_shufflenet_v2_x1_0_eval_hpu (main.TestBenchmark) ... ok
test_simple_gpt_eval_hpu (main.TestBenchmark) ... skipped 'Method eval on hpu is not implemented because "Model requires CUDA", skipping...'
test_simple_gpt_tp_manual_eval_hpu (main.TestBenchmark) ... skipped 'Method eval on hpu is not implemented because "Model requires CUDA", skipping...'
test_soft_actor_critic_eval_hpu (main.TestBenchmark) ... ok
test_speech_transformer_eval_hpu (main.TestBenchmark) ... ok
test_squeezenet1_1_eval_hpu (main.TestBenchmark) ... ok
test_stable_diffusion_text_encoder_eval_hpu (main.TestBenchmark) ... ok
test_stable_diffusion_unet_eval_hpu (main.TestBenchmark) ... ok
test_tacotron2_eval_hpu (main.TestBenchmark) ... ok
test_timm_efficientdet_eval_hpu (main.TestBenchmark) ... skipped 'Method eval on hpu is not implemented because "The original model code forces the use of CUDA.", skipping...'
test_timm_efficientnet_eval_hpu (main.TestBenchmark) ... ok
test_timm_nfnet_eval_hpu (main.TestBenchmark) ... ok
test_timm_regnet_eval_hpu (main.TestBenchmark) ... ok
test_timm_resnest_eval_hpu (main.TestBenchmark) ... ok
test_timm_vision_transformer_eval_hpu (main.TestBenchmark) ... ok
test_timm_vision_transformer_large_eval_hpu (main.TestBenchmark) ... ok
test_timm_vovnet_eval_hpu (main.TestBenchmark) ... ok
test_torch_multimodal_clip_eval_hpu (main.TestBenchmark) ... ok
test_tts_angular_eval_hpu (main.TestBenchmark) ... ok
test_vgg16_eval_hpu (main.TestBenchmark) ... ok
test_vision_maskrcnn_eval_hpu (main.TestBenchmark) ... ok

Ran 101 tests in 983.330s

FAILED (errors=7, skipped=5)

@arathi-hlab
Copy link
Author

reason of picking torchbench accuracy test is just to aligned with torchbench open source Nightly as they used test.py in their Nightly
https://github.com/pytorch/benchmark/actions/runs/15510623845/job/43671117463

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant