Updated HPU workflow file as per torchbench execution on HPU #20
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
reason of picking torchbench accuracy test is just to aligned with torchbench open source Nightly as they used test.py in their Nightly 
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
hi , all i tried to run torchbench accuracy test in eval mode and i tried locally with this change here is the result i got
root@arathi-h7ju-g3-l-tfjob-worker-0:~# python benchmark/test.py -v -k eval_hpu
Calling add_step_closure function does not have any effect. It's lazy mode only functionality. (warning logged once)
Calling mark_step function does not have any effect. It's lazy mode only functionality. (warning logged once)
Calling iter_mark_step function does not have any effect. It's lazy mode only functionality. (warning logged once)
test_BERT_pytorch_eval_hpu (main.TestBenchmark) ... ok
test_Background_Matting_eval_hpu (main.TestBenchmark) ... skipped 'Method eval on hpu is not implemented because "", skipping...'
test_LearningToPaint_eval_hpu (main.TestBenchmark) ... ok
test_Super_SloMo_eval_hpu (main.TestBenchmark) ... ok
test_alexnet_eval_hpu (main.TestBenchmark) ... ok
test_basic_gnn_edgecnn_eval_hpu (main.TestBenchmark) ... ok
test_basic_gnn_gcn_eval_hpu (main.TestBenchmark) ... ok
test_basic_gnn_gin_eval_hpu (main.TestBenchmark) ... ok
test_basic_gnn_sage_eval_hpu (main.TestBenchmark) ... ok
test_cm3leon_generate_eval_hpu (main.TestBenchmark) ... ok
test_dcgan_eval_hpu (main.TestBenchmark) ... ok
test_demucs_eval_hpu (main.TestBenchmark) ... ok
test_densenet121_eval_hpu (main.TestBenchmark) ... ok
test_detectron2_fasterrcnn_r_101_c4_eval_hpu (main.TestBenchmark) ... ok
test_detectron2_fasterrcnn_r_101_dc5_eval_hpu (main.TestBenchmark) ... ok
test_detectron2_fasterrcnn_r_101_fpn_eval_hpu (main.TestBenchmark) ... ok
test_detectron2_fasterrcnn_r_50_c4_eval_hpu (main.TestBenchmark) ... ok
test_detectron2_fasterrcnn_r_50_dc5_eval_hpu (main.TestBenchmark) ... ok
test_detectron2_fasterrcnn_r_50_fpn_eval_hpu (main.TestBenchmark) ... ok
test_detectron2_fcos_r_50_fpn_eval_hpu (main.TestBenchmark) ... ERROR
test_detectron2_maskrcnn_eval_hpu (main.TestBenchmark) ... ok
test_detectron2_maskrcnn_r_101_c4_eval_hpu (main.TestBenchmark) ... ok
test_detectron2_maskrcnn_r_101_fpn_eval_hpu (main.TestBenchmark) ... ok
test_detectron2_maskrcnn_r_50_c4_eval_hpu (main.TestBenchmark) ... ok
test_detectron2_maskrcnn_r_50_fpn_eval_hpu (main.TestBenchmark) ... ok
test_dlrm_eval_hpu (main.TestBenchmark) ... ok
test_doctr_det_predictor_eval_hpu (main.TestBenchmark) ... ERROR
test_doctr_reco_predictor_eval_hpu (main.TestBenchmark) ... ERROR
test_drq_eval_hpu (main.TestBenchmark) ... ok
test_fastNLP_Bert_eval_hpu (main.TestBenchmark) ... ERROR
test_functorch_dp_cifar10_eval_hpu (main.TestBenchmark) ... ok
test_functorch_maml_omniglot_eval_hpu (main.TestBenchmark) ... ok
test_hf_Albert_eval_hpu (main.TestBenchmark) ... ok
test_hf_Bart_eval_hpu (main.TestBenchmark) ... ok
test_hf_Bert_eval_hpu (main.TestBenchmark) ... ok
test_hf_Bert_large_eval_hpu (main.TestBenchmark) ... ok
test_hf_BigBird_eval_hpu (main.TestBenchmark) ... ok
test_hf_DistilBert_eval_hpu (main.TestBenchmark) ... ok
test_hf_GPT2_eval_hpu (main.TestBenchmark) ... ok
test_hf_GPT2_large_eval_hpu (main.TestBenchmark) ... ok
test_hf_Longformer_eval_hpu (main.TestBenchmark) ... ok
test_hf_Reformer_eval_hpu (main.TestBenchmark) ... ok
test_hf_Roberta_base_eval_hpu (main.TestBenchmark) ... ok
test_hf_T5_base_eval_hpu (main.TestBenchmark) ... ok
test_hf_T5_eval_hpu (main.TestBenchmark) ... ok
test_hf_T5_generate_eval_hpu (main.TestBenchmark) ... ok
test_hf_T5_large_eval_hpu (main.TestBenchmark) ... ok
test_hf_Whisper_eval_hpu (main.TestBenchmark) ... ok
test_hf_clip_eval_hpu (main.TestBenchmark) ... ok
test_hf_distil_whisper_eval_hpu (main.TestBenchmark) ... ok
test_lennard_jones_eval_hpu (main.TestBenchmark) ... ok
test_llama_eval_hpu (main.TestBenchmark) ... ERROR
test_llama_v2_7b_16h_eval_hpu (main.TestBenchmark) ... ok
test_llava_eval_hpu (main.TestBenchmark) ... ok
test_maml_eval_hpu (main.TestBenchmark) ... ok
test_maml_omniglot_eval_hpu (main.TestBenchmark) ... ok
test_microbench_unbacked_tolist_sum_eval_hpu (main.TestBenchmark) ... ok
test_mnasnet1_0_eval_hpu (main.TestBenchmark) ... ok
test_mobilenet_v2_eval_hpu (main.TestBenchmark) ... ok
test_mobilenet_v3_large_eval_hpu (main.TestBenchmark) ... ok
test_moco_eval_hpu (main.TestBenchmark) ... skipped 'Method eval on hpu is not implemented because "hpu not supported", skipping...'
test_moondream_eval_hpu (main.TestBenchmark) ... ok
test_nanogpt_eval_hpu (main.TestBenchmark) ... ok
test_nvidia_deeprecommender_eval_hpu (main.TestBenchmark) ... ERROR
test_opacus_cifar10_eval_hpu (main.TestBenchmark) ... ok
test_phlippe_densenet_eval_hpu (main.TestBenchmark) ... ok
test_phlippe_resnet_eval_hpu (main.TestBenchmark) ... ok
test_pyhpc_equation_of_state_eval_hpu (main.TestBenchmark) ... ok
test_pyhpc_isoneutral_mixing_eval_hpu (main.TestBenchmark) ... ok
test_pyhpc_turbulent_kinetic_energy_eval_hpu (main.TestBenchmark) ... ok
test_pytorch_CycleGAN_and_pix2pix_eval_hpu (main.TestBenchmark) ... ok
test_pytorch_stargan_eval_hpu (main.TestBenchmark) ... ok
test_pytorch_unet_eval_hpu (main.TestBenchmark) ... ok
test_resnet152_eval_hpu (main.TestBenchmark) ... ok
test_resnet18_eval_hpu (main.TestBenchmark) ... ok
test_resnet50_eval_hpu (main.TestBenchmark) ... ok
test_resnext50_32x4d_eval_hpu (main.TestBenchmark) ... ok
test_sam_eval_hpu (main.TestBenchmark) ... ok
test_sam_fast_eval_hpu (main.TestBenchmark) ... ERROR
test_shufflenet_v2_x1_0_eval_hpu (main.TestBenchmark) ... ok
test_simple_gpt_eval_hpu (main.TestBenchmark) ... skipped 'Method eval on hpu is not implemented because "Model requires CUDA", skipping...'
test_simple_gpt_tp_manual_eval_hpu (main.TestBenchmark) ... skipped 'Method eval on hpu is not implemented because "Model requires CUDA", skipping...'
test_soft_actor_critic_eval_hpu (main.TestBenchmark) ... ok
test_speech_transformer_eval_hpu (main.TestBenchmark) ... ok
test_squeezenet1_1_eval_hpu (main.TestBenchmark) ... ok
test_stable_diffusion_text_encoder_eval_hpu (main.TestBenchmark) ... ok
test_stable_diffusion_unet_eval_hpu (main.TestBenchmark) ... ok
test_tacotron2_eval_hpu (main.TestBenchmark) ... ok
test_timm_efficientdet_eval_hpu (main.TestBenchmark) ... skipped 'Method eval on hpu is not implemented because "The original model code forces the use of CUDA.", skipping...'
test_timm_efficientnet_eval_hpu (main.TestBenchmark) ... ok
test_timm_nfnet_eval_hpu (main.TestBenchmark) ... ok
test_timm_regnet_eval_hpu (main.TestBenchmark) ... ok
test_timm_resnest_eval_hpu (main.TestBenchmark) ... ok
test_timm_vision_transformer_eval_hpu (main.TestBenchmark) ... ok
test_timm_vision_transformer_large_eval_hpu (main.TestBenchmark) ... ok
test_timm_vovnet_eval_hpu (main.TestBenchmark) ... ok
test_torch_multimodal_clip_eval_hpu (main.TestBenchmark) ... ok
test_tts_angular_eval_hpu (main.TestBenchmark) ... ok
test_vgg16_eval_hpu (main.TestBenchmark) ... ok
test_vision_maskrcnn_eval_hpu (main.TestBenchmark) ... ok
Ran 101 tests in 983.330s
FAILED (errors=7, skipped=5)