Test failures on Windows ARM64 install

[zooba]

Ran the test suite on an installed ARM64 machine and saw the errors below.

This is mostly a note-to-self to come back to this, but if anyone else wants to jump in and fix a test or two (I think they're mostly obvious what's gone wrong), feel free. I'm including the skipped tests to double check that they ought to be skipped.

The run command was just py -m test -j4 -w and it should be an otherwise clean machine.

== Tests result: FAILURE ==

42 tests skipped:
    test.test_asyncio.test_unix_events test.test_gdb.test_backtrace
    test.test_gdb.test_cfunction test.test_gdb.test_cfunction_full
    test.test_gdb.test_misc test.test_gdb.test_pretty_print
    test.test_multiprocessing_fork.test_manager
    test.test_multiprocessing_fork.test_misc
    test.test_multiprocessing_fork.test_processes
    test.test_multiprocessing_fork.test_threads
    test.test_multiprocessing_forkserver.test_manager
    test.test_multiprocessing_forkserver.test_misc
    test.test_multiprocessing_forkserver.test_processes
    test.test_multiprocessing_forkserver.test_threads test_asdl_parser
    test_clinic test_dbm_gnu test_dbm_ndbm test_devpoll test_epoll
    test_fcntl test_fork1 test_generated_cases test_grp test_ioctl
    test_kqueue test_openpty test_perf_profiler test_perfmaps
    test_poll test_pty test_pwd test_readline test_resource
    test_syslog test_termios test_threadsignals test_tty test_wait3
    test_wait4 test_xxlimited test_xxtestfuzz

10 tests skipped (resource denied):
    test_curses test_peg_generator test_smtpnet test_socketserver
    test_tkinter test_ttk test_urllib2net test_urllibnet test_winsound
    test_zipfile64

6 tests failed:
    test.test_asyncio.test_subprocess test_audit test_ctypes
    test_launcher test_os test_webbrowser

411 tests OK.

0:07:06 load avg: 19.35 Re-running 6 failed tests in verbose mode in subprocesses
0:07:06 load avg: 19.35 Run 6 tests in parallel using 4 worker processes
0:07:07 load avg: 19.34 [1/6/1] test_ctypes failed (1 failure)
Re-running test_ctypes in verbose mode (matching: test_array_in_struct)
test_array_in_struct (test.test_ctypes.test_structures.StructureTestCase.test_array_in_struct) ... FAIL

======================================================================
FAIL: test_array_in_struct (test.test_ctypes.test_structures.StructureTestCase.test_array_in_struct)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Program Files\Python313-arm64\Lib\test\test_ctypes\test_structures.py", line 669, in test_array_in_struct
    self.assertEqual(result, expected)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
AssertionError: 2.0 != 6.0

----------------------------------------------------------------------
Ran 1 test in 0.005s

FAILED (failures=1)
test test_ctypes failed
0:07:07 load avg: 19.34 [2/6/1] test_audit passed
Re-running test_audit in verbose mode (matching: test_wmi_exec_query)
test_wmi_exec_query (test.test_audit.AuditTest.test_wmi_exec_query) ... ('_wmi.exec_query', ' ', 'SELECT * FROM Win32_OperatingSystem')
ok

----------------------------------------------------------------------
Ran 1 test in 0.268s

OK
0:07:07 load avg: 19.34 [3/6/2] test_webbrowser failed (1 error)
Re-running test_webbrowser in verbose mode (matching: test_synthesize)
test_synthesize (test.test_webbrowser.ImportTest.test_synthesize) ... ERROR

======================================================================
ERROR: test_synthesize (test.test_webbrowser.ImportTest.test_synthesize)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Program Files\Python313-arm64\Lib\test\test_webbrowser.py", line 314, in test_synthesize
    webbrowser.get(sys.executable)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python313-arm64\Lib\webbrowser.py", line 65, in get
    raise Error("could not locate runnable browser")
webbrowser.Error: could not locate runnable browser

----------------------------------------------------------------------
Ran 1 test in 0.017s

FAILED (errors=1)
test test_webbrowser failed
0:07:07 load avg: 19.34 [4/6/3] test_os failed (1 failure)
Re-running test_os in verbose mode (matching: test_pipe_spawnl)
test_pipe_spawnl (test.test_os.FDInheritanceTests.test_pipe_spawnl) ... C:\Program: can't open file 'C:\\Users\\USERNAME\\AppData\\Local\\Temp\\test_python_y8xoiflm\\test_python_4120æ\\Files\\Python313-arm64\\python.exe': [Errno 2] No such file or directory
FAIL

======================================================================
FAIL: test_pipe_spawnl (test.test_os.FDInheritanceTests.test_pipe_spawnl)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Program Files\Python313-arm64\Lib\test\test_os.py", line 4601, in test_pipe_spawnl
    self.assertEqual(exitcode, 0)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
AssertionError: 2 != 0

----------------------------------------------------------------------
Ran 1 test in 0.056s

FAILED (failures=1)
test test_os failed
0:07:08 load avg: 19.27 [5/6/4] test_launcher failed (3 failures)
Re-running test_launcher in verbose mode (matching: test_search_path, test_search_path_exe, test_shebang_command_in_venv)
test_search_path (test.test_launcher.TestLauncher.test_search_path) ... FAIL
test_search_path_exe (test.test_launcher.TestLauncher.test_search_path_exe) ... FAIL
test_shebang_command_in_venv (test.test_launcher.TestLauncher.test_shebang_command_in_venv) ... FAIL

======================================================================
FAIL: test_search_path (test.test_launcher.TestLauncher.test_search_path)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Program Files\Python313-arm64\Lib\test\test_launcher.py", line 626, in test_search_path
    self.assertEqual(f"{sys.executable} -prearg {script} -postarg", data["stdout"].strip())
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'C:\\Program Files\\Python313-arm64\\pytho[115 chars]targ' != '"C:\\Program Files\\Python313-arm64\\pyth[117 chars]targ'
- C:\Program Files\Python313-arm64\python.exe -prearg C:\Users\USERNAME\AppData\Local\Temp\test_python_hde_mn6u\test_python_8920æ\tmpkbsz_fju.py -postarg
+ "C:\Program Files\Python313-arm64\python.exe" -prearg C:\Users\USERNAME\AppData\Local\Temp\test_python_hde_mn6u\test_python_8920æ\tmpkbsz_fju.py -postarg
? +                                           +


======================================================================
FAIL: test_search_path_exe (test.test_launcher.TestLauncher.test_search_path_exe)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Program Files\Python313-arm64\Lib\test\test_launcher.py", line 637, in test_search_path_exe
    self.assertEqual(f"{sys.executable} -prearg {script} -postarg", data["stdout"].strip())
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'C:\\Program Files\\Python313-arm64\\pytho[115 chars]targ' != '"C:\\Program Files\\Python313-arm64\\pyth[117 chars]targ'
- C:\Program Files\Python313-arm64\python.exe -prearg C:\Users\USERNAME\AppData\Local\Temp\test_python_hde_mn6u\test_python_8920æ\tmp1t02n1ni.py -postarg
+ "C:\Program Files\Python313-arm64\python.exe" -prearg C:\Users\USERNAME\AppData\Local\Temp\test_python_hde_mn6u\test_python_8920æ\tmp1t02n1ni.py -postarg
? +                                           +


======================================================================
FAIL: test_shebang_command_in_venv (test.test_launcher.TestLauncher.test_shebang_command_in_venv)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Program Files\Python313-arm64\Lib\test\test_launcher.py", line 741, in test_shebang_command_in_venv
    self.assertEqual(data["stdout"].strip(), f"{sys.executable} arg1 {script}")
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: '"C:\\Program Files\\Python313-arm64\\pyth[105 chars]f.py' != 'C:\\Program Files\\Python313-arm64\\pytho[103 chars]f.py'
- "C:\Program Files\Python313-arm64\python.exe" arg1 C:\Users\USERNAME\AppData\Local\Temp\test_python_hde_mn6u\test_python_8920æ\tmpk4vqht5f.py
? -                                           -
+ C:\Program Files\Python313-arm64\python.exe arg1 C:\Users\USERNAME\AppData\Local\Temp\test_python_hde_mn6u\test_python_8920æ\tmpk4vqht5f.py


----------------------------------------------------------------------
Ran 3 tests in 1.318s

FAILED (failures=3)
test test_launcher failed
0:07:08 load avg: 19.27 [6/6/5] test.test_asyncio.test_subprocess failed (1 error, 1 failure)
Re-running test.test_asyncio.test_subprocess in verbose mode (matching: test_kill_issue43884, test_create_subprocess_env_shell)
test_create_subprocess_env_shell (test.test_asyncio.test_subprocess.SubprocessProactorTests.test_create_subprocess_env_shell) ... 'C:\Program' is not recognized as an internal or external command,
operable program or batch file.
FAIL
test_kill_issue43884 (test.test_asyncio.test_subprocess.SubprocessProactorTests.test_kill_issue43884) ... 'C:\Program' is not recognized as an internal or external command,
operable program or batch file.
ERROR

======================================================================
ERROR: test_kill_issue43884 (test.test_asyncio.test_subprocess.SubprocessProactorTests.test_kill_issue43884)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Program Files\Python313-arm64\Lib\test\test_asyncio\test_subprocess.py", line 225, in test_kill_issue43884
    proc.send_signal(signal.CTRL_BREAK_EVENT)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python313-arm64\Lib\asyncio\subprocess.py", line 140, in send_signal
    self._transport.send_signal(signal)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "C:\Program Files\Python313-arm64\Lib\asyncio\base_subprocess.py", line 146, in send_signal
    self._check_proc()
    ~~~~~~~~~~~~~~~~^^
  File "C:\Program Files\Python313-arm64\Lib\asyncio\base_subprocess.py", line 143, in _check_proc
    raise ProcessLookupError()
ProcessLookupError

======================================================================
FAIL: test_create_subprocess_env_shell (test.test_asyncio.test_subprocess.SubprocessProactorTests.test_create_subprocess_env_shell)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Program Files\Python313-arm64\Lib\test\test_asyncio\test_subprocess.py", line 756, in test_create_subprocess_env_shell
    self.loop.run_until_complete(self.check_stdout_output(main(), b'bar'))
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python313-arm64\Lib\asyncio\base_events.py", line 709, in run_until_complete
    return future.result()
           ~~~~~~~~~~~~~^^
  File "C:\Program Files\Python313-arm64\Lib\test\test_asyncio\test_subprocess.py", line 740, in check_stdout_output
    self.assertEqual(stdout, output)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
AssertionError: b'' != b'bar'

----------------------------------------------------------------------
Ran 2 tests in 1.204s

FAILED (failures=1, errors=1)
test test.test_asyncio.test_subprocess failed
5 tests failed again:
    test.test_asyncio.test_subprocess test_ctypes test_launcher
    test_os test_webbrowser

== Tests result: FAILURE then FAILURE ==

42 tests skipped:
    test.test_asyncio.test_unix_events test.test_gdb.test_backtrace
    test.test_gdb.test_cfunction test.test_gdb.test_cfunction_full
    test.test_gdb.test_misc test.test_gdb.test_pretty_print
    test.test_multiprocessing_fork.test_manager
    test.test_multiprocessing_fork.test_misc
    test.test_multiprocessing_fork.test_processes
    test.test_multiprocessing_fork.test_threads
    test.test_multiprocessing_forkserver.test_manager
    test.test_multiprocessing_forkserver.test_misc
    test.test_multiprocessing_forkserver.test_processes
    test.test_multiprocessing_forkserver.test_threads test_asdl_parser
    test_clinic test_dbm_gnu test_dbm_ndbm test_devpoll test_epoll
    test_fcntl test_fork1 test_generated_cases test_grp test_ioctl
    test_kqueue test_openpty test_perf_profiler test_perfmaps
    test_poll test_pty test_pwd test_readline test_resource
    test_syslog test_termios test_threadsignals test_tty test_wait3
    test_wait4 test_xxlimited test_xxtestfuzz

10 tests skipped (resource denied):
    test_curses test_peg_generator test_smtpnet test_socketserver
    test_tkinter test_ttk test_urllib2net test_urllibnet test_winsound
    test_zipfile64

6 re-run tests:
    test.test_asyncio.test_subprocess test_audit test_ctypes
    test_launcher test_os test_webbrowser

5 tests failed:
    test.test_asyncio.test_subprocess test_ctypes test_launcher
    test_os test_webbrowser

412 tests OK.

Total duration: 7 min 8 sec
Total tests: run=38,410 failures=13 skipped=2,693
Total test files: run=465/469 failed=5 skipped=42 resource_denied=10 rerun=6
Result: FAILURE then FAILURE

Linked PRs

• gh-114272: Fix or skip tests that fail due to spaces in paths #114451
• gh-114272: Allow _wmi audit test to succeed even if it times out #114602
• gh-114272: fix Windows test_asyncio/test_subprocess when sys.executable contains unescaped spaces #128160

[zooba]

The original test_audit failure, since it passed on rerun:

0:01:53 load avg: 14.72 [ 87/469/2] test_audit failed (1 failure) -- running (1): test.test_multiprocessing_spawn.test_processes (1 min 4 sec)
Traceback (most recent call last):
  File "C:\Program Files\Python313-arm64\Lib\test\audit-tests.py", line 559, in <module>
    globals()[test](*sys.argv[2:])
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
  File "C:\Program Files\Python313-arm64\Lib\test\audit-tests.py", line 490, in test_wmi_exec_query
    _wmi.exec_query("SELECT * FROM Win32_OperatingSystem")
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: [WinError 258] The wait operation timed out
test test_audit failed -- Traceback (most recent call last):
  File "C:\Program Files\Python313-arm64\Lib\test\test_audit.py", line 223, in test_wmi_exec_query
    self.fail(stderr)
    ~~~~~~~~~^^^^^^^^
AssertionError: Traceback (most recent call last):
  File "C:\Program Files\Python313-arm64\Lib\test\audit-tests.py", line 559, in <module>
    globals()[test](*sys.argv[2:])
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
  File "C:\Program Files\Python313-arm64\Lib\test\audit-tests.py", line 490, in test_wmi_exec_query
    _wmi.exec_query("SELECT * FROM Win32_OperatingSystem")
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: [WinError 258] The wait operation timed out

[serhiy-storchaka]

======================================================================
FAIL: test_search_path (test.test_launcher.TestLauncher.test_search_path)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Program Files\Python313-arm64\Lib\test\test_launcher.py", line 626, in test_search_path
    self.assertEqual(f"{sys.executable} -prearg {script} -postarg", data["stdout"].strip())

#113129 changed this code. It is likely fixed now.

[zooba]

#113129 changed this code. It is likely fixed now.

That was my change... I don't recall fixing this?

[zooba]

Looks like I did fix it though, okay cool. Nope, just made it more likely that it'd be in a directory without spaces (or possibly more likely...)

[zooba]

#114451 fixes most of the tests, just test_ctypes and test_audit remain.

I suspect/fear that ctypes will need a libffi fix or update, as it's likely not our code at fault.

test_audit probably just needs a retry with delay for the timeout. I think we added similar for the other tests that use WMI.