Updated HPU workflow file as per torchbench execution on HPU #20
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
hi , all i tried to run torchbench accuracy test in eval mode and i tried locally with this change here is the result i got
root@arathi-h7ju-g3-l-tfjob-worker-0:~# python benchmark/test.py -v -k eval_hpu
Calling add_step_closure function does not have any effect. It's lazy mode only functionality. (warning logged once)
Calling mark_step function does not have any effect. It's lazy mode only functionality. (warning logged once)
Calling iter_mark_step function does not have any effect. It's lazy mode only functionality. (warning logged once)
test_BERT_pytorch_eval_hpu (main.TestBenchmark) ... ok
test_Background_Matting_eval_hpu (main.TestBenchmark) ... skipped 'Method eval on hpu is not implemented because "", skipping...'
test_LearningToPaint_eval_hpu (main.TestBenchmark) ... ok
test_Super_SloMo_eval_hpu (main.TestBenchmark) ... ok
test_alexnet_eval_hpu (main.TestBenchmark) ... ok
test_basic_gnn_edgecnn_eval_hpu (main.TestBenchmark) ... ok
test_basic_gnn_gcn_eval_hpu (main.TestBenchmark) ... ok
test_basic_gnn_gin_eval_hpu (main.TestBenchmark) ... ok
test_basic_gnn_sage_eval_hpu (main.TestBenchmark) ... ok
test_cm3leon_generate_eval_hpu (main.TestBenchmark) ... ok
test_dcgan_eval_hpu (main.TestBenchmark) ... ok
test_demucs_eval_hpu (main.TestBenchmark) ... ok
test_densenet121_eval_hpu (main.TestBenchmark) ... ok
test_detectron2_fasterrcnn_r_101_c4_eval_hpu (main.TestBenchmark) ... ok
test_detectron2_fasterrcnn_r_101_dc5_eval_hpu (main.TestBenchmark) ... ok
test_detectron2_fasterrcnn_r_101_fpn_eval_hpu (main.TestBenchmark) ... ok
test_detectron2_fasterrcnn_r_50_c4_eval_hpu (main.TestBenchmark) ... ok
test_detectron2_fasterrcnn_r_50_dc5_eval_hpu (main.TestBenchmark) ... ok
test_detectron2_fasterrcnn_r_50_fpn_eval_hpu (main.TestBenchmark) ... ok
test_detectron2_fcos_r_50_fpn_eval_hpu (main.TestBenchmark) ... ERROR
test_detectron2_maskrcnn_eval_hpu (main.TestBenchmark) ... ok
test_detectron2_maskrcnn_r_101_c4_eval_hpu (main.TestBenchmark) ... ok
test_detectron2_maskrcnn_r_101_fpn_eval_hpu (main.TestBenchmark) ... ok
test_detectron2_maskrcnn_r_50_c4_eval_hpu (main.TestBenchmark) ... ok
test_detectron2_maskrcnn_r_50_fpn_eval_hpu (main.TestBenchmark) ... ok
test_dlrm_eval_hpu (main.TestBenchmark) ... ok
test_doctr_det_predictor_eval_hpu (main.TestBenchmark) ... ERROR
test_doctr_reco_predictor_eval_hpu (main.TestBenchmark) ... ERROR
test_drq_eval_hpu (main.TestBenchmark) ... ok
test_fastNLP_Bert_eval_hpu (main.TestBenchmark) ... ERROR
test_functorch_dp_cifar10_eval_hpu (main.TestBenchmark) ... ok
test_functorch_maml_omniglot_eval_hpu (main.TestBenchmark) ... ok
test_hf_Albert_eval_hpu (main.TestBenchmark) ... ok
test_hf_Bart_eval_hpu (main.TestBenchmark) ... ok
test_hf_Bert_eval_hpu (main.TestBenchmark) ... ok
test_hf_Bert_large_eval_hpu (main.TestBenchmark) ... ok
test_hf_BigBird_eval_hpu (main.TestBenchmark) ... ok
test_hf_DistilBert_eval_hpu (main.TestBenchmark) ... ok
test_hf_GPT2_eval_hpu (main.TestBenchmark) ... ok
test_hf_GPT2_large_eval_hpu (main.TestBenchmark) ... ok
test_hf_Longformer_eval_hpu (main.TestBenchmark) ... ok
test_hf_Reformer_eval_hpu (main.TestBenchmark) ... ok
test_hf_Roberta_base_eval_hpu (main.TestBenchmark) ... ok
test_hf_T5_base_eval_hpu (main.TestBenchmark) ... ok
test_hf_T5_eval_hpu (main.TestBenchmark) ... ok
test_hf_T5_generate_eval_hpu (main.TestBenchmark) ... ok
test_hf_T5_large_eval_hpu (main.TestBenchmark) ... ok
test_hf_Whisper_eval_hpu (main.TestBenchmark) ... ok
test_hf_clip_eval_hpu (main.TestBenchmark) ... ok
test_hf_distil_whisper_eval_hpu (main.TestBenchmark) ... ok
test_lennard_jones_eval_hpu (main.TestBenchmark) ... ok
test_llama_eval_hpu (main.TestBenchmark) ... ERROR
test_llama_v2_7b_16h_eval_hpu (main.TestBenchmark) ... ok
test_llava_eval_hpu (main.TestBenchmark) ... ok
test_maml_eval_hpu (main.TestBenchmark) ... ok
test_maml_omniglot_eval_hpu (main.TestBenchmark) ... ok
test_microbench_unbacked_tolist_sum_eval_hpu (main.TestBenchmark) ... ok
test_mnasnet1_0_eval_hpu (main.TestBenchmark) ... ok
test_mobilenet_v2_eval_hpu (main.TestBenchmark) ... ok
test_mobilenet_v3_large_eval_hpu (main.TestBenchmark) ... ok
test_moco_eval_hpu (main.TestBenchmark) ... skipped 'Method eval on hpu is not implemented because "hpu not supported", skipping...'
test_moondream_eval_hpu (main.TestBenchmark) ... ok
test_nanogpt_eval_hpu (main.TestBenchmark) ... ok
test_nvidia_deeprecommender_eval_hpu (main.TestBenchmark) ... ERROR
test_opacus_cifar10_eval_hpu (main.TestBenchmark) ... ok
test_phlippe_densenet_eval_hpu (main.TestBenchmark) ... ok
test_phlippe_resnet_eval_hpu (main.TestBenchmark) ... ok
test_pyhpc_equation_of_state_eval_hpu (main.TestBenchmark) ... ok
test_pyhpc_isoneutral_mixing_eval_hpu (main.TestBenchmark) ... ok
test_pyhpc_turbulent_kinetic_energy_eval_hpu (main.TestBenchmark) ... ok
test_pytorch_CycleGAN_and_pix2pix_eval_hpu (main.TestBenchmark) ... ok
test_pytorch_stargan_eval_hpu (main.TestBenchmark) ... ok
test_pytorch_unet_eval_hpu (main.TestBenchmark) ... ok
test_resnet152_eval_hpu (main.TestBenchmark) ... ok
test_resnet18_eval_hpu (main.TestBenchmark) ... ok
test_resnet50_eval_hpu (main.TestBenchmark) ... ok
test_resnext50_32x4d_eval_hpu (main.TestBenchmark) ... ok
test_sam_eval_hpu (main.TestBenchmark) ... ok
test_sam_fast_eval_hpu (main.TestBenchmark) ... ERROR
test_shufflenet_v2_x1_0_eval_hpu (main.TestBenchmark) ... ok
test_simple_gpt_eval_hpu (main.TestBenchmark) ... skipped 'Method eval on hpu is not implemented because "Model requires CUDA", skipping...'
test_simple_gpt_tp_manual_eval_hpu (main.TestBenchmark) ... skipped 'Method eval on hpu is not implemented because "Model requires CUDA", skipping...'
test_soft_actor_critic_eval_hpu (main.TestBenchmark) ... ok
test_speech_transformer_eval_hpu (main.TestBenchmark) ... ok
test_squeezenet1_1_eval_hpu (main.TestBenchmark) ... ok
test_stable_diffusion_text_encoder_eval_hpu (main.TestBenchmark) ... ok
test_stable_diffusion_unet_eval_hpu (main.TestBenchmark) ... ok
test_tacotron2_eval_hpu (main.TestBenchmark) ... ok
test_timm_efficientdet_eval_hpu (main.TestBenchmark) ... skipped 'Method eval on hpu is not implemented because "The original model code forces the use of CUDA.", skipping...'
test_timm_efficientnet_eval_hpu (main.TestBenchmark) ... ok
test_timm_nfnet_eval_hpu (main.TestBenchmark) ... ok
test_timm_regnet_eval_hpu (main.TestBenchmark) ... ok
test_timm_resnest_eval_hpu (main.TestBenchmark) ... ok
test_timm_vision_transformer_eval_hpu (main.TestBenchmark) ... ok
test_timm_vision_transformer_large_eval_hpu (main.TestBenchmark) ... ok
test_timm_vovnet_eval_hpu (main.TestBenchmark) ... ok
test_torch_multimodal_clip_eval_hpu (main.TestBenchmark) ... ok
test_tts_angular_eval_hpu (main.TestBenchmark) ... ok
test_vgg16_eval_hpu (main.TestBenchmark) ... ok
test_vision_maskrcnn_eval_hpu (main.TestBenchmark) ... ok
Ran 101 tests in 983.330s
FAILED (errors=7, skipped=5)