[OOB 5.10] IP connectivity test failed

[toyowata]

Description

Mbed OS 5.10 OOB - IP connectivity test failed (Target: K64F, Host: Windows and Mac).

Expected result

All TEST_APPS should pass.

Actual result

test_cmdline is skipped and TCPSERVER_ACCEPT was inconclusive.

Versions

PS C:\test\oob\mbed-os> mbed ls
mbed-os (#3fb5781af180, tag: mbed-os-5.10.0-rc1)

python 2.7.15
mbed-cli 1.8.0
icetea 1.0.1

Test result

PS C:\test\oob\mbed-os> mbed test -m k64f -t arm --icetea -v
[mbed] Working path "C:\test\oob\mbed-os" (program)
[mbed] Auto-installing missing Python modules...

(snip)

+--------------------------------+--------------+------------------------------------+---------------------------------------------+-----------+----------+
| Testcase                       |   Verdict    |            Fail Reason             |                 Skip Reason                 | platforms | duration |
+--------------------------------+--------------+------------------------------------+---------------------------------------------+-----------+----------+
| test_cmdline                   |     skip     |                                    | Test requiring application binary not build |           |    0     |
| UDPSOCKET_BIND_PORT            |     pass     |                                    |                                             |    K64F   |  18.121  |
| TCPSOCKET_BIND_PORT            |     pass     |                                    |                                             |    K64F   |  2.034   |
| TCPSERVER_ACCEPT               | inconclusive | No suitable local device available |                                             |           |  0.196   |
| TCPSOCKET_ECHOTEST_BURST_SHORT |     pass     |                                    |                                             |    K64F   |  6.691   |
+--------------------------------+--------------+------------------------------------+---------------------------------------------+-----------+----------+
+---------------+----------------+
|    Summary    |                |
+---------------+----------------+
| Final Verdict |  INCONCLUSIVE  |
|     count     |       5        |
|    passrate   |    100.00 %    |
|      pass     |       3        |
|      skip     |       1        |
|  inconclusive |       1        |
|    Duration   | 0:00:27.042000 |
+---------------+----------------+
14:09:43.543 | RSP     MainThread: Cleaning up ResourceProvider.
14:09:43.553 | RSP     MainThread: Cleaning up allocator.
14:09:43.559 Cleanup done.

Complete log file :
mbed-os-oob-icetea.txt

Issue request type

[x] Question
[ ] Enhancement
[x] Bug

[0xc0170]

@ARMmbed/mbed-os-test

[juhaylinen]

TCPSERVER_ACCEPT test case requires 2 devices. https://github.com/ARMmbed/mbed-os/blob/master/TEST_APPS/testcases/netsocket/TCPSERVER_ACCEPT.py#L38

The test result is inconclusive if only one device is found.

[OPpuolitaival]

@jonikula can you check this? I think that it could be nice to tell for example (1/2 suitable devices found) or some another way to make clear that need more devices

[jonikula]

That's something we could easily add to the logging and perhaps to the failure reason as well.

[OPpuolitaival]

@0xc0170 this is not a bug. Works as expected. I will create usability improvement ticket related to this.

[MarceloSalazar]

@OPpuolitaival can you please add a link to the enhancement as result of this effort?

[MarceloSalazar]

Hi guys, I've hit the same problem. I didn't understand why the test was inconclusive and whether I needed extra HW platforms.
I believe the solution is to document what are the requirements for each of the tests.

Here is a PR (#8073) with a good start, but IMO should be extended to explain those cases in more detail. Thanks!