Network problems with 7 or more clients connected to the mbed Device Connector

[rspelta]

Description

• Type: Bug?

With seven or more clients connected to the mbed Device Connector some of them are not listed temporally or disappear until I reset the client.

We are using mbed-os-example-client for 10 boards, 3 of them are NUCLEO_F429ZI+X-NUCLEO-IDS01A4 boards and the others are sensor-node boards (it's a new kind of board, we are developing on it, @MarceloSalazar knows it).
The border router uses the nanostack-border-route repository and it is a NUCLEO_F429ZI+X-NUCLEO-IDS01A4, it uses a ethernet cable connected to the internet router (with enabled 6to4 tunneling). The mesh is via Spirit, we have worked with Wolfgang (@betzw) in order to have the spirit driver stable.

The NUCLEO_F429ZI boards use the lastest commit of the repositories, the sensor-node board come from the versions:

mbed-os-sensor-node (5f27acc)
|- easy-connect (6fb5842becae)
| |- atmel-rf-driver (57f22763f4d3)
| |- esp8266-driver (4ed87bf7fe37)
| | - ESP8266\ATParser (269f14532b98) | |- mcr20a-rf-driver (d8810e105d7d) | - stm-spirit1-rf-driver (ac7a4f477222)
|- mbed-client (f8f0fc8b9321)
| |- mbed-client-c (c739b8cbcc57)
| |- mbed-client-classic (f673b8b60779)
| - mbed-client-mbed-tls (7e1b6d815038) |- mbed-os (ed4febefdede) - pal (4e46c0ea8706)

Every client has its security.h file and a different MAC address.
We have done this test:

1. turn on the border router and wait until it is ready
2. turn on one client
3. wait to see it is connected to the mbed connector website and try to request information via API console
4. go to point 2

here you can read the trace of the border router:
https://pastebin.com/ti8U3HgS

From the sniffer seems that the lost clients (I mean who disappear from the mbed connector list) continue to communicate with the border router. We don't have information about how the nanostack works so we have problem to get a better idea what is going on.

To try to understand what happen we have:

• a "sniffer" node in order to see the "frames on the air"
• the console of the border router (attacched the log of the border, the trace is minimal we didn't turn it off completely)
• the console of the nodes (just to know if it is registered to the mbed connector the first time)

Our goal is to have more than 10 boards connected to the mbed Device Connector.
What can we do?

[ciarmcom]

ARM Internal Ref: IOTCLT-1854

[betzw]

Cc @SeppoTakalo, @JanneKiiskila, @screamerbg

[MarceloSalazar]

@rspelta are you using the same mbed Device Connector (mDC) account for all devices?
If so, would you please to generate the credentials for multiple accounts? (lets say 2 or 3). I'd like to understand whether the problem is caused by a limitation on mDC or the Border Router not properly forwarding packets.
It would be useful to have a capture on the packets (tcpdump) sent between the BR and the backhaul network (tunnel).

[kjbracey]

I'm seeing enough "MAC TX fail" and "Source route error" messages in that log to make me think you have some sort of radio driver or hardware problem. Communication between the nodes does not seem to be as reliable as I'd expect.

"MAC TX fail" means "I tried to transmit to a neighbour, but didn't get an Ack after multiple attempts".

"Source route error A->B" means "LoWPAN router A is telling me (the border router) that they had a MAC TX error (see above) when forwarding to B".

I'd run a simple ping test from outside the mesh to the nodes to see what sort of average packet loss you're getting - both with minimum size packets and larger ones (eg 500 or 1000 bytes). The aim should be to have only a couple of percent loss at minimum size, and try to get under 10% at 1000 bytes.

[kjbracey]

I've just spent a little while reviewing the Spirit driver code, just to see if I can see any obvious flaws. (Hard to say much without knowing the hardware, but I can look for general issues.)

I'm a bit wary about the software ack handling - can be tricky.

There is one specific problem that could be affecting performance now - it seems to me the acks are sent with a common send() routine that enables hardware CSMA-CD. An ack should be being sent 192us after transmission completion, without CSMA. Backing off the ack will greatly reduce the chance of packets being successfully acknowledged.

Other notes on ack reception - you're calling TX_DONE whenever tx_sequence == seq_number, whether you were expecting an ack or not. This could cause stack confusion in various ways (eg if you were backing off while someone else used the same sequence number). You should only process an ACK when you actually expect one (TX completed, and AR bit was set in it)

Also, while expecting an ack, it can be beneficial to report TX_FAIL and stop expecting when you receive anything other then an ack with the expected sequence number. The stack will eventually time out if it doesn't get TX_DONE, but receipt of anything else (including a wrongly-numbered ack) indicates a lack of acknowledgment, which can be reported immediately.

[betzw]

Thanks a lot @kjbracey-arm!

I have immediately tried to implement all your indications. Before pushing them, could you pls. be so kind and review these. Above all, pls. let me know if I got your points correctly, am missing something, and if you think the patch fixes the issues you have listed.

[rspelta]

Hi, I used the last patch from @betzw. I done the tests with 9 sensor-node boards. One mbed Device Connector (mDC) account for all devices. Now it seems more stable, all the 9 clients are listed on the website and I didn't see any client disappear from the list.

However I still read these messages in the log:
[ERR ][mMCp]: MAC tx fail
[ERR ][6lAd]: MCPS Data fail by status 233

You can see the log of the border router (trace setted on "info level"):
https://pastebin.com/d01djmcZ

And the log of one client (trace setted on "info level"):
https://pastebin.com/z3yfHAgn

@MarceloSalazar next test will be with more clients connected so I will try your advices in order to have more info.
Thanks you all!

[JanneKiiskila]

Hi all,

I would conclude that this issue is NOT an mbed-client issue, it's just the fact it can't tolerate (especially the TLS handshakes are a very, very touch area) flaky networks. @kjbracey-arm knows the gory details and somehow I feel there would be some options for making it more fault-tolerant (issue towards mbedtls perhaps - https://github.com/ARMmbed/mbedtls) to make the situation better, client can't do much about that TLS handshake thing - we just try to do it and if it fails -> we can't connect to the server.

[kjbracey]

You will always see some "MCPS Data fail" messages - quite a few if the network is busy.

Once a 6LoWPAN network accessing the mbed server has stabilised, I wouldn't expect to see many fails, as it shouldn't be that busy. Looking at the logs, my feeling is that performance is still a bit below par, but I'd need to see some proper stats (eg ping tests, as suggested above).

Looking at those logs, one thing occurred to me, so I just checked the driver. It doesn't actually report "CCA_FAIL", so we can't tell the difference between "channel busy" (225) and "no ack" (233). The driver should be enhanced to distinguish the cases - handling IRQ_MAX_BO_CCA_REACH. Will help diagnostics, and may help performance (Nanostack will respond differently).

I also wonder if your NumBackoffs parameter is set too low. The default for IEEE 802.15.4 is 4, and we tend to use 5. You've got it set to 3. Might be worth checking the other numbers.

The driver should tell Nanostack the number of CSMA retries it did. Maybe you can't get that number out of the driver on success, so saying 0 would be okay, but you should say "4" on a CCA_FAIL if that's your NumBackoffs setting. (ie 4 retries, 5 attempts total).

[betzw]

The number of backoffs can easily be changed here.

[betzw]

I have reflected a bit about CCA & backoff and would like to ask @kjbracey-arm how the Nanostack interfaces to persistent CCA (i.e how to implement integration of a RF supporting persistent CCA into the Nanostack)?