-
Notifications
You must be signed in to change notification settings - Fork 3k
Closed
Labels
Description
The host test timing_drift_auto returns success based on the individual test cases run, but not on the test as a whole. Because of this, if a crash occurs during teardown after all tests have run and passed then the host test will return pass, even though a crash occurred.
An example of this can be seen on the nightly results for PR #5242:
[1507071477.06][HTST][INF] DUT end time : 16964000.0
[1507071477.06][HTST][INF] min_pass : 16.2592500091 , max_pass : 17.6001000881 for 5.0%%
[1507071477.06][HTST][INF] min_inconclusive : 15.9239000797 , max_inconclusive : 17.97075001
[1507071477.06][HTST][INF] Time reported by device: 16.898
[1507071477.06][HTST][INF] Test passed !!!
[1507071477.07][SERI][TXD] {{pass;0}}
[1507071477.19][CONN][RXD] Thread 0x0 error -6: Not allowed in ISR context
[1507071690.48][HTST][INF] test suite run finished after 240.53 sec...
[1507071690.48][CONN][INF] received special even '__host_test_finished' value='True', finishing
[1507071690.51][HTST][INF] CONN exited with code: 0
[1507071690.51][HTST][INF] No events in queue
[1507071690.51][HTST][INF] stopped consuming events
[1507071690.51][HTST][INF] host test result(): True
[1507071690.51][HTST][WRN] missing __exit event from DUT
[1507071690.51][HTST][WRN] missing __exit_event_queue event from host test
[1507071690.51][HTST][INF] calling blocking teardown()
[1507071690.51][HTST][INF] teardown() finished
[1507071690.51][HTST][INF] {{result;success}}