mbed cellular stack does not establish PPP session

[NeilMacMullen]

mbed 5.9 (tag mbed-os-5.9.0), and also present on master 799ba08.

The mbed-os-example-cellular application connects to the network but fails to establish a PPP data channel.

The target modem is a UBLOX SARA U201 (3G) which successfully connects and transfers data using the old C027Interface code and on-modem stack. Screenshot and attached log show some PPP activity but no connection.

pppfail.log

[NeilMacMullen]

Adding @MarceloSalazar

[AriParkkila]

This looks like a problem fixed in PR #6962. Can you cherry-pick that to see if it helps?

[ciarmcom]

ARM Internal Ref: MBOTRIAGE-702

[NeilMacMullen]

@AriParkkila I've checked out #6962 but unfortunately I still see the same behaviour. You probably spotted it in the log but one possible clue might be that on this SIM there are some unsolicited message-waiting indications (+UMWI) Could this be affecting the state machine?

[AriParkkila]

@NeilMacMullen that +UMWI should be handled gracefully with PR #6962. The problem is that after calling connect() the FileHandle has been given to the PPP driver, and no other must read it anymore due to that would confuse PPP link negotiation.

According to the log there are lots of debug prints after CONNECT. I wonder where those might be coming... Those do not look exactly like coming from cellular.debug-at..?

Can you enable TRACE_LEVEL_DEBUG to see more traces, especially prints from ATHandler::process_oob() should show if this is the same problem as in #6962.

[NeilMacMullen]

@AriParkkila I disabled the message waiting and that doesn't make a difference. TRACE_LEVEL_DEBUG is already set in main using

mbed_trace_init(); // initialize trace library
mbed_trace_config_set(TRACE_ACTIVE_LEVEL_DEBUG);)

were you expecting to see more debug?

According to the log there are lots of debug prints after CONNECT

Do you mean the 'line noise'? If so this is raw output from UARTSerial read/write which is tracing I added to confirm that bytes are being sent/received between mcu and modem.

[NeilMacMullen]

Ah - I hadn't realised you need to compile in the debug traces.. Here's the relevant portion of trace around the PPP channel connection...

0> [00017264ms][DBG ][CELL]: lac 0381 8[00017275ms][INFO][CELL]: Cellular state from Attaching network to Activating PDP Context
0> [00017275ms][INFO][CELL]: cellular_status: Attaching network ==> Activating PDP Context
0> [00017276ms][INFO][CELL]: Activate PDP Context (timeout 60000 ms)
0> [00017277ms][DBG ][CELL]: APN in use: mobiledata
0> [00017294ms][DBG ][CELL]: Context id 2
0> [00017307ms][INFO][CELL]: Activate PDP context 2
0> [00020798ms][INFO][CELL]: Cellular state from Activating PDP Context to Connecting network
0> [00020798ms][INFO][CELL]: cellular_status: Activating PDP Context ==> Connecting network
0> [00020799ms][INFO][CELL]: Connect to cellular network (timeout 60000 ms)
0> [00020800ms][INFO][CELL]: FSM: network_callback called with event: 0, intptr: 3, _state: Connecting network
0> [00020801ms][INFO][CELL]: Open data channel in PPP mode
0> [00047823ms][INFO][LPPP]: status_cb: User interrupt
0> [00047823ms][INFO][CELL]: FSM: network_callback called with event: 0, intptr: 2, _state: Connecting network
0> [00047824ms][ERR ][CELL]: Failed to open data channel!
0> [00047825ms][INFO][CELL]: FSM: network_callback called with event: 0, intptr: 2, _state: Connecting network
0> [00047825ms][DBG ][CELL]: Retry State Connecting network, retry 1/10

[kjbracey]

I'm assuming this is a PPP negotation failure - I tried dissecting your log, but the PPP data appears to be somewhat garbled - some sort of Latin-1->UTF-8 conversion I can't reverse, and some other loss.

It appeared to be the higher level giving up, not the PPP - "user interrupt" after a 30-second timeout. It's vaguely possible it was just being slow and extending the 30 second timeout would clear it, but I doubt it.

I think you'll need to turn on enough lwIP debug to show the PPP negotiation. It looks like there's enough happening that we're having a conversation, but don't know what.

I think you'll need to turn on lwip.debug-enabled and lwip.enable-ppp-trace. You will also need to increase lwip.tcpip-thread-stacksize and lwip.ppp-thread-stacksize if they don't auto-increase because of the debug.

[NeilMacMullen]

@kjbracey-arm Thanks Kevin - that sounds plausible, I'm busy this morning but can dump out the raw PPP bytes later today. Yesterday I got as far as establishing that this is failing when lwip::bringup checks to see if iface_is_up (apologies if these aren't exact names - I'm away from the source) , causing ppp_close to be called with USERERR. That seems consistent with the idea that PPP negotiation just isn't getting anywhere. I'm open the idea this may just be a cellular network/sim configuration issue though as I say, the ublox on-chip stack happily connects and sends data using the same sim/modem Anway, more later....

[NeilMacMullen]

I've attached a log of the raw uart trace (captured from uartserial write/read functions.

rawuart.log

[NeilMacMullen]

@kjbracey-arm

You will also need to increase lwip.tcpip-thread-stacksize and lwip.ppp-thread-stacksize

Can you suggest sensible numbers for these? The default appears to be 1600 (words? bytes?). I've tried a random selection of values between 2000 and 8192 but seem to be alternating between stack overflow and heap exhaustion :( (I'm on an RF52 so available ram is ~55K) Here's what I currently have in my target_overrides section...

"lwip.debug-enabled":true,
"lwip.enable-ppp-trace":true,
"lwip.tcpip-thread-stacksize":4096,
"lwip.ppp-thread-stacksize":4096

[kjbracey]

It may be that you just don't have enough RAM in the system to enable lwIP's PPP trace without messing with it - it uses quite a big line buffer on the stack. Maybe reduce buffer sizes in lwip_utils.c?