[FEATURE] eager shuffle deletion for Spark

[zuston]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I have searched in the issues and found no similar issues.

Describe the feature

Eager deletion of Spark shuffle data for completed stages is preferable, because the native Spark cleanup mechanism is too slow to reclaim resources in a timely manner.

celeborn also has similar design. apache/celeborn#3569

Motivation

No response

Describe the solution

No response

Additional context

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

[zuston]

cc @jerqi

[jerqi]

It's better to optimize the vanilla Spark in my view.