Caching results of the `filter` will result to inconsistent cache state

[VitalyFedyunin]

Currently it blocks, but we just got lucky:

text/torchtext/datasets/stsb.py

Line 85 in caaa8e3

cache_decompressed_dp = FileOpener(cache_decompressed_dp, mode="b").read_from_tar().filter(_filter_fn)

Please change the order of filter and end_caching

[Nayef211]

@VitalyFedyunin just wondering if this could be the cause of the test failures I'm seeing in my PR? #1732 (comment)

[parmeet]

Yes, @VitalyFedyunin already told me about this. One thing I am wondering is why it is causing failures in STSB and wikitext103 only? We do filtering in between on_disk_cache and end_caching for all the compressed datasets. Should we change to order in other datasets too?

[VitalyFedyunin]

Yes please change all datasets. I think it fails when you try to zip two data pipes in tests and as they are separate graph parts but locking the same files this deadlock happens.

[parmeet]

Yes please change all datasets. I think it fails when you try to zip two data pipes in tests and as they are separate graph parts but locking the same files this deadlock happens.

This PR fixes the issue #1737. Seems like empty/non-existent files/path are creating the problem and it is not really necessary to put filter after end caching. In-fact this issue brought up the bugs in our mock testing for the failing datasets :). The reason we are keeping filter in-between is because we do not want to dump all the files to disk but only the one necessary to build the dataset.

[VitalyFedyunin]

I just afraid that this filter pattern is error prone. And even if I fix deadlock (it is possible), putting filter might lead to cache inconsistency (in case of filter output change between runs).