Some Tests fail for TF Backend with Seg Fault

[sampathweb]

SegFormer Tests are failing with Segmentation Fault on TF Backend. They work fine in Torch and Jax Backends.

Log: https://source.cloud.google.com/results/invocations/84cffec0-1890-479a-8e6f-0a732ad1c891/log

Actually, we have seg fault for other tests too for TensorFlow backend. Here's one on Deep Lab V3 - https://source.cloud.google.com/results/invocations/d6798a84-3033-439d-99ad-a76e6fad5688/log

[tirthasheshpatel]

The CI is now green on master so closing this issue. We can reopen if it comes up again.