Enable all tests

[samanklesaria]

Now that deprecated functionality has been removed, we can remove the restrictions given when running the test suite during CI. This PR depends on the functionality in the branch pseudo_main.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/audio/4061

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

• ROCm MI2xx CI/CD workflows failing due to : download from https://api.github.com/repos/pytorch/pytorch timed out.

❌ 8 New Failures

As of commit 88515c1 with merge base 7d4c0bf ():

NEW FAILURES - The following jobs have failed:

• .github/workflows/bandit.yml (gh)
• .github/workflows/integration-test.yml (gh)
• .github/workflows/lint.yml (gh)
• .github/workflows/pr-labels.yml (gh)
• .github/workflows/unittest-macos-cpu.yml (gh)
• .github/workflows/unittest-windows-cpu.yml (gh)
• .github/workflows/unittest-windows-gpu.yml (gh)
• Build documentation / build / Build doc (gh)
  At least one of the pre-conditions you specified did not hold

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[samanklesaria]

The torchscript consistency tests don't seem to work with manual autograd in python (torch.autograd.Function overloads). Pretty much all the torchscript consistency checks seem to have this issue. The only way forward is to either:
A) Investigate more whether it's possible to get torch.autograd.Function working with torchscript
B) Delete the torchscript consistency checks.
As torchscript is essentially deprecated anyway, I propose option (B)

[pearu]

It is possible to fix torchscript consistency tests by avoiding autograd functions under jit-scripting. For instance, rnnt_loss tests pass when applying the following patch:

diff --git a/src/torchaudio/functional/functional.py b/src/torchaudio/functional/functional.py
index 5d4284d5..a72c6091 100644
--- a/src/torchaudio/functional/functional.py
+++ b/src/torchaudio/functional/functional.py
@@ -1821,7 +1821,13 @@ def _rnnt_loss(
     if blank < 0:  # reinterpret blank index if blank < 0.
         blank = logits.shape[-1] + blank
 
-    costs = RnntLoss.apply(
+    if torch.jit.is_scripting():
+        # TorchScript does not support autograd functions
+        func = torch.ops.torchaudio.rnnt_loss_forward
+    else:
+        func = RnntLoss.apply
+
+    costs = func(
         logits,
         targets,
         logit_lengths,
@@ -1831,6 +1837,9 @@ def _rnnt_loss(
         fused_log_softmax
     )
 
+    if torch.jit.is_scripting():
+        costs = costs[0]
+
     if reduction == "mean":
         return costs.mean()
     elif reduction == "sum":

[pearu]

LGTM! Thanks, @samanklesaria!