Fix flaky tests that are recently popping up

[NicolasHug]

Since #4497 was merged, we're observing a few tests that start randomly failing.

Before #4497, these tests were almost always using the same RNG state, which was set in a test that was run earlier in the test execution suite. Now that all tests are properly independent and that the RNG doesn't leak, these tests run with a new RNG at each execution, and if they're unstable they might fail.

(Note: this is a good thing; it's better to know that they fail now rather than when submiting an unrelated PR, which is what happened in #3032 (comment))

For each of these tests we should find out whether the flakyness is severe or not. A simple solution is to parametrize the test over 100 or 1000 random seeds and check the failure rate. If the failure rate is reasonable we can just set a seed with toch.manual_seed(). If not, we should try to fix the test and make it more robust.

The list of tests so far is:

• test_random_apply - Remove p-value checks in test_transforms.py #4756
• test_stochastic_depth[row-0.2] - test.test_ops.TestStochasticDepth - Fix flakiness on StochasticDepth test #4758
• test_randomperspective_fill[L] - Set seed of test_randomperspective_fill #4759
• test_randomperspective_fill[RGB] - Set seed of test_randomperspective_fill #4759
• test_randomperspective_fill[F] - Set seed of test_randomperspective_fill #4759
• test_random_vertical_flip - Remove p-value checks in test_transforms.py #4756
• test_random_horizontal_flip - Remove p-value checks in test_transforms.py #4756
• test_frozenbatchnorm2d_eps Setting seeds for frozen BN #4761
• test_batched_nms_implementations Setting seeds for test_batched_nms_implementations  #4766
• test_backward[True-cpu] - test.test_ops.TestRoiPool Setting seeds for TestRoiPool backward #4763
• test_backward[False-cpu] - test.test_ops.TestRoiPool Setting seeds for TestRoiPool backward #4763
• test_random_erasing - Set seed of test_random_erasing #4764
• test_color_jitter_hue[hue2-3-cpu] - Fix seed for TestColorJitter #4762
• test_color_jitter_contrast[1.5-3-cuda] - Fix seed for TestColorJitter #4762

cc @pmeier

[NicolasHug]

As we previously discussed offline, I'll start looking into the tests that rely on p-value testing. test_random_apply above is one of them.

I'll change these to just test the case p == 0 and p == 1 -- for all values in-between, we'll just rely on pytorch core tests for torch.rand(1)

[NicolasHug]

looking into test_randomperspective_fill rn

[datumbox]

investigating test_frozenbatchnorm2d_eps

[jdsgomes]

looking at test_batched_nms_implementations

[NicolasHug]

looking at the test_color_jitter_xyz ones

[datumbox]

looking at test.test_ops.TestRoiPool