Skip to content

Commit 6b47abd

Browse files
itholicHyukjinKwon
authored andcommitted
[SPARK-32812][PYTHON][TESTS] Avoid initiating a process during the main process for run-tests.py
### What changes were proposed in this pull request? In certain environments, seems it fails to run `run-tests.py` script as below: ``` Traceback (most recent call last): File "<string>", line 1, in <module> ... raise RuntimeError(''' RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase. This probably means that you are not using fork to start your child processes and you have forgotten to use the proper idiom in the main module: if __name__ == '__main__': freeze_support() ... The "freeze_support()" line can be omitted if the program is not going to be frozen to produce an executable. Traceback (most recent call last): ... raise EOFError EOFError ``` The reason is that `Manager.dict()` launches another process when the main process is initiated. It works in most environments for an unknown reason but it should be good to avoid such pattern as guided from Python itself. ### Why are the changes needed? To prevent the test failure for Python. ### Does this PR introduce _any_ user-facing change? No, it fixes a test script. ### How was this patch tested? Manually ran the script after fixing. ``` Running PySpark tests. Output is in /.../python/unit-tests.log Will test against the following Python executables: ['/.../python3', 'python3.8'] Will test the following Python tests: ['pyspark.sql.dataframe'] /.../python3 python_implementation is CPython /.../python3 version is: Python 3.8.5 python3.8 python_implementation is CPython python3.8 version is: Python 3.8.5 Starting test(/.../python3): pyspark.sql.dataframe Starting test(python3.8): pyspark.sql.dataframe Finished test(/.../python3): pyspark.sql.dataframe (33s) Finished test(python3.8): pyspark.sql.dataframe (34s) Tests passed in 34 seconds ``` Closes #29666 from itholic/SPARK-32812. Authored-by: itholic <[email protected]> Signed-off-by: HyukjinKwon <[email protected]> (cherry picked from commit c8c082c) Signed-off-by: HyukjinKwon <[email protected]>
1 parent a22e1c5 commit 6b47abd

File tree

1 file changed

+3
-1
lines changed

1 file changed

+3
-1
lines changed

python/run-tests.py

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -51,7 +51,7 @@ def print_red(text):
5151
print('\033[31m' + text + '\033[0m')
5252

5353

54-
SKIPPED_TESTS = Manager().dict()
54+
SKIPPED_TESTS = None
5555
LOG_FILE = os.path.join(SPARK_HOME, "python/unit-tests.log")
5656
FAILURE_REPORTING_LOCK = Lock()
5757
LOGGER = logging.getLogger()
@@ -142,6 +142,7 @@ def run_individual_python_test(target_dir, test_name, pyspark_python):
142142
skipped_counts = len(skipped_tests)
143143
if skipped_counts > 0:
144144
key = (pyspark_python, test_name)
145+
assert SKIPPED_TESTS is not None
145146
SKIPPED_TESTS[key] = skipped_tests
146147
per_test_output.close()
147148
except:
@@ -322,4 +323,5 @@ def process_queue(task_queue):
322323

323324

324325
if __name__ == "__main__":
326+
SKIPPED_TESTS = Manager().dict()
325327
main()

0 commit comments

Comments
 (0)