Issue with greentea results seem to be skewed.

[miteshdedhia7]

Description

Attached are two test reports from few greentea runs.

odd_gt_1.txt
In this file in the test case report section notice that the tests-mbed_drivers-c_strings suite test shows results for the "deep-sleep - high-speed clocks are turned off", "deep-sleep - source of wake-up - lp ticker" tests which are actually part of the deep sleep tests and not the c_string tests.

odd_gt_2.txt
In this file in the test case report section notice that the features-storage-tests-kvstore-static_tests suite shows results for the "Testing callbacks with 0 uchars" test which are part of the call back test suite.

I am running the following commands
mbed test --compile -t IAR -m CYW9P62S1_43012EVB_01
mbedgt -vV

Note that I see this issue intermittently. Unfortunately, I only have the result logs and do not have the full logs for two runs above to tell if the test cases actually ran correctly.

Version info:
mbed --version: 1.10.1
mbedgt --version: 1.7.2

Issue request type

[ ] Question
[ ] Enhancement
[x] Bug

[0xc0170]

regarding 1st:

Attaching the logs:

| CYW9P62S1_43012EVB_01-GCC_ARM | CYW9P62S1_43012EVB_01 | tests-mbed_drivers-c_strings                                                 | deep-sleep - high-speed clocks are turned off                                                     | 0      | 1      | FAIL    | 0.02               |
| CYW9P62S1_43012EVB_01-GCC_ARM | CYW9P62S1_43012EVB_01 | tests-mbed_drivers-c_strings                                                 | deep-sleep - source of wake-up - lp ticker                                                        | 0      | 9      | FAIL    | 0.12               |
| CYW9P62S1_43012EVB_01-GCC_ARM | CYW9P62S1_43012EVB_01 | tests-mbed_drivers-c_strings                                                 | sleep - source of wake-up - us ticker                                                             | 1      | 0      | OK      | 0.02   

Not only test case is incorrect but also test results? This looks like a bug in our tooling. Is this target specific?

cc @ARMmbed/team-cypress Have you experienced this?

[ifyall]

Per @jamesbeyond:
"I think there is some issues with greentea where the test suite name misaligned with test case name, where DUT got reset unexpectedly.
So in the test log, before print the result summary table, a warning line of “mbedgt: utest test case summary mismatch: utest reported passes and failures miscount!” must have been printed out.
Overall, if the target work just fine passed all test without unexpected reset, the test result should be just aligned."

@miteshdedhia7 , please check for the "utest case mismatch" warning.

Ian

[miteshdedhia7]

Yes I have noticed the utest case mismatch warning in the full logs. Unfortunately I do not have the full logs to verify for the specific results attached to this ticket as I mentioned earlier in the description. We will look out for that for checking the validity of the test reports.

[ciarmcom]

@miteshdedhia7 thank you for raising this issue.Please take a look at the following comments:

Could you add some more detail to the description? A good description should be at least 25 words.
What target(s) are you using?
What toolchain(s) are you using?
What version of Mbed OS are you using (tag or sha)?
It would help if you could also specify the versions of any tools you are using?
How can we reproduce your issue?

NOTE: If there are fields which are not applicable then please just add 'n/a' or 'None'.This indicates to us that at least all the fields have been considered.
Please update the issue header with the missing information, the issue will not be mirroredto our internal defect tracking system or investigated until this has been fully resolved.

[adbridge]

@miteshdedhia7 is this issue still valid ?

[miteshdedhia7]

@adbridge I dont have a consistent way of reproducing this so I will close this ticket for now and reopen if I run into the problem again.