DNS error (-3009) when using ESP8266 with NUCLEO-091RC on some Wi-Fi router

[zhiyong-ft]

Description of defect

The problem seems about compatibility of mbed+ESP8266 with some WiFi router setup. When tested on one particular WiFi router, mbed+ESP8266 could connect to wifi, get IP address, network mask, gateway etc. But then stuck when trying connect to any http URLs. The error code is: -3009.
However, this doesn't happen with other Wi-Fi routers. The only thing particular about the problematic wifi router is, it has a large black list of URL such as youtbe/facebook etc. But test URLs in mbed-http-example are not in that blacklist, and can be accessed from a computer on the same network.

Target(s) affected by this defect ?

NUCLEO-091RC

Toolchain(s) (name and version) displaying this defect ?

mbed-cli(1.8.3)

What version of Mbed-os are you using (tag or sha) ?

5.13.4

What version(s) of tools are you using. List all that apply (E.g. mbed-cli)

mbed-cli, 1.8.3 from windows installer 0.4.10

How is this defect reproduced ?

This happens only with one particular WiFi router so far, cannot be repeated on other routers.

[ciarmcom]

Internal Jira reference: https://jira.arm.com/browse/MBOTRIAGE-2419

[zhiyong-ft]

Some other information that might be relevant: 1) Hardware flow control was not enabled. 2) When esp8266.debug was set to true and stdio.baud-rate was set to 9600 instead of 115200, I got a different error 3012 with a previously working wifi router when debug was not turned on.
I suspect this has something to do with buffer overrun. Unfortunately so far I couldn't get hardware flow control to work.

[zhiyong-ft]

Visited the Wifi router again with hardware flow controll and debug more turned on, still got the same error. This seems to have something to do with google's 8.8.8.8 DNS server:

Connecting to WiFi...
AT> AT
AT? OK
%n
AT< slAT< “obŒl`��r’ŸîAT< ll`�Œâs“lŒl�Œìl`�Œãr“lìl�Œl`�Œãr’pƒ�ŸAT! ready
AT? OK
%n
AT< 
AT(Timeout)
AT> AT
AT? OK
%n
AT< AT
AT= OK

AT> AT+RST
AT? OK
%n
AT< AT+RST
AT= OK

AT? %n
AT? %n
AT<  ets Jan  8 2013,rst cause:1, boot mode:(3,7)
AT< load 0x40100000, len 2592, room 16 
AT< tail 0
AT< chksum 0xf3
AT< load 0x3ffe8000, len 764, room 8 
AT< tail 4
AT< chksum 0x92
AT< load 0x3ffe82fc, len 676, room 4 
AT< tail 0
AT< chksum 0x22
AT< csum 0x22
AT< 2nd boot version : 1.7(üŒÅ2�±0x1fc000Œâ�ì�ƒoì’sƒòn|ìlŽìŽ�l�c“��žŒ�
AT! ready
AT? %n
AT< 
AT? %n
AT> AT
AT? OK
%n
AT< AT
AT= OK

AT> ATE0
AT? OK
%n
AT< ATE0
AT= OK

AT> AT+UART_CUR=115200,8,1,0,3
AT? OK
%n
AT= OK

AT> AT+GMR
AT? AT version:%*d.%*d.%*d.%*d%n
AT= AT version:1.7.0.0
AT? OK
%n
AT< (Aug 16 2018 00:57:04)
AT< SDK version:3.0.0(d49923c)
AT< compile time:Aug 23 2018 16:58:12
AT< Bin version(Wroom 02):v1.7.0
AT= OK

AT> AT+GMR
AT? SDK version:%*d.%*d.%*d%n
AT< AT version:1.7.0.0(Aug 16 2018 00:57:04)
AT= SDK version:3.0.0
AT? OK
%n
AT< (d49923c)
AT< compile time:Aug 23 2018 16:58:12
AT< Bin version(Wroom 02):v1.7.0
AT= OK

AT> AT+CWMODE_DEF=1
AT? OK
%n
AT= OK

AT> AT+CWCOUNTRY_DEF=0,"CN",1,13
AT? OK
%n
AT= OK

AT> AT+CWCOUNTRY_CUR=0,"CN",1,13
AT? OK
%n
AT= OK

AT> AT+CIPRECVMODE=1
AT? OK
%n
AT= OK

AT> AT+CWMODE_CUR=1
AT? OK
%n
AT= OK

AT> AT+CIPMUX=1
AT? OK
%n
AT= OK

AT> AT+CWDHCP_CUR=1,1
AT? OK
%n
AT= OK

AT> AT+CWJAP_CUR="XXXXXXXXXXXXXXXXXX","YYYYYYYYYYYYYYY"
AT? OK
%n
AT! WIFI 
AT? %*12[^"]
%n
AT= CONNECTED

AT? OK
%n
AT! WIFI 
AT? %*12[^"]
%n
AT= GOT IP

AT? OK
%n
AT= OK

SuccAT? %ess
n

AT> AT+CIFSR
AT? +CIFSR:STAMAC,"%*17[^"]"%n
AT< +CIFSR:STAIP,"192.168.1.34"
AT= +CIFSR:STAMAC,"b4:e6:2d:6b:0e:65"
AT? OK
%n
AT< 
AT= OK

AT? %MAC: n
b4:e6:2d:6b:0e:65
AT> AT+CIFSR
AT? +CIFSR:STAIP,"%*15[^"]"%n
AT= +CIFSR:STAIP,"192.168.1.34"
AT? OK
%n
AT< 
AT< +CIFSR:STAMAC,"b4:e6:2d:6b:0e:65"
AT= OK

AT? IP: 1%n
92.168.1.34
AT> AT+CIPSTA_CUR?
AT? +CIPSTA_CUR:netmask:"%*15[^"]"%n
AT< +CIPSTA_CUR:ip:"192.168.1.34"
AT< +CIPSTA_CUR:gateway:"192.168.1.1"
AT= +CIPSTA_CUR:netmask:"255.255.255.0"
AT? OK
%n
AT< 
AT= OK

AT? %Netman
sk: 255.255.255.0
AT> AT+CIPSTA_CUR?
AT? +CIPSTA_CUR:gateway:"%*15[^"]"%n
AT< +CIPSTA_CUR:ip:"192.168.1.34"
AT= +CIPSTA_CUR:gateway:"192.168.1.1"
AT? OK
%n
AT< 
AT< +CIPSTA_CUR:netmask:"255.255.255.0"
AT= OK

AT? Gatew%n
ay: 192.168.1.1
AT> AT+CWJAP_CUR?
AT? +CWJAP_CUR:"%*[^"]","%*17[^"]"%n
AT= +CWJAP_CUR:"XXXXXXXXXXXXXXXXX","b0:39:56:ec:07:df"
AT? OK
%n
AT< ,3,-63,0
AT= OK

AT? %n
AT> AT+CWLAP="","b0:39:56:ec:07:df",
AT? OK
%n
AT! +CWLAP:
AT? OK
%n
AT< 
AT= OK

RSAT? %SI: -n
55

AT> AT+CIFSR
AT? +CIFSR:STAIP,"%*15[^"]"%n
AT= +CIFSR:STAIP,"192.168.1.34"
AT? OK
%n
AT< 
AT< +CIFSR:STAMAC,"b4:e6:2d:6b:0e:65"
AT= OK

AT? %n
AT? %n
AT> AT+CIPSTART=1,"UDP","8.8.8.8",53
AT? OK
%n
AT< 1,CONNECT
AT= OK

AT? %n
AT> AT+CIPSEND=1,34
AT? >%n
AT< OK
AT= >
AT? SEND OK%n
AT<  
AT< Recv 34 bytes
AT= SEND OK
AT? %n
AT< 
AT? %n
AT? %n
AT? %n
AT? %n
AT? %n
AT> AT+CIPSEND=1,34
AT? >%n
AT< OK
AT= >
AT? SEND OK%n
AT<  
AT< Recv 34 bytes
AT= SEND OK
AT? %n
AT< 
AT? %n
AT? %n
AT? %n
AT? %n
AT? %n
AT> AT+CIPSEND=1,34
AT? >%n
AT< OK
AT= >
AT? SEND OK%n
AT<  
AT< Recv 34 bytes
AT= SEND OK
AT? %n
AT< 
AT? %n
AT? %n
AT? %n
AT? %n
AT? %n
AT> AT+CIPCLOSE=1
AT? OK
%n
AT! 1,CLOSED
AT? OK
%n
AT< 
AT= OK

AT? %
HttpRn
equest failed (error code -3009)

----- HTTP GET response -----


++ MbedOS Error Info ++
Error Status: 0x8001011F Code: 287 Module: 1
Error Message: Operator new out of memory

Location: 0x8010713
Error Value: 0x80003F0
Current Thread: main  Id: 0x20002E3C Entry: 0x8015705 StackSize: 0x1000 StackMem: 0x200017B0 SP: 0x200026A4 
For more info, visit: https://mbed.com/s/error?error=0x8001011F&tgt=NUCLEO_F091RC
-- MbedOS Error Info --

[zhiyong-ft]

The problem seems to be ESP8266 not receiving anything from the DNS server. When URLs are replaced to IP address in HTTP request, the http example code runs just fine.
When a direct request to the same DNS server on the same network and PC, response via UDP, correct response is received.

[0xc0170]

cc @ARMmbed/mbed-os-wan

[AnttiKauppila]

Maybe HTTPExample does not need so much memory?
And when debug is enabled it affects significantly to memory usage
++ MbedOS Error Info ++
Error Status: 0x8001011F Code: 287 Module: 1
Error Message: Operator new out of memory

[zhiyong-ft]

@AnttiKauppila this part actually puzzles me as well. In the example, there are two HTTP requests, 1st for GET, then a POST. If the 1st request fails somehow, http_req would be deleted, then a new request will be created, and mbed shouldn't crash. But oddly, as in log, once that 3009 error shows up, mbed os just crashes.

[michalpasztamobica]

Hi @zhiyong80 , I wonder - when you disable the blacklisting in your router, are you still able to reproduce the issue?
Can you also try the same router with a device that has more memory, like K64F+ESP8266? 32kB of RAM that NUCLEO_F091RC has is not a lot...

EDIT: I came up with more questions...
Would you also proivde us with the application code that you are using and your mbed_app.json? Does the program fail immediatelly or does it only fail after a while? (Perhaps the heap has been indeed depleted)

Antti suggested that you used http example . If this is the case, then please note, that this project explicitly states:

Note: HTTPS requests do not work on targets with less than 128K of RAM due to the size of the TLS handshake. For more background see mbed-http.

The mbed-http may work on smaller targets, but to mitigate memory issued it has some suggestions on the project's page. Have you tried that out?

Furthermore - enabling debug will surely impact the memory usage, especially on a target with 32 kB of RAM. Are you able to attach a debugger instead of using debug prints?

[zhiyong-ft]

@michalpasztamobica I tried STM32F401RE which has 64KB ram, and got the same results, also with mbed-os-5.14.2 and 5.15. The only notable difference with the newer version of mbed-os(5.15?) is, the app will try 3 different DNS servers before giving up. So I don't think this is a ram related problem.

More likely, this might be something related to ESP8266 itself. I dug up some discussion about DNS over UDP failures from years ago. This may still exist on some routers.

I will test the router later today with SparkFun ESP8266 Thing which has nothing to do with Mbed and see what I get.

About app crashing, it is unrelated. I accidentally added an extra line.

        HttpRequest* post_req = new HttpRequest(network, HTTP_POST, "SOME URL");
        post_req->set_header("Content-Type", "application/x-www-form-urlencoded");

        const char body[] = "DUMMY POST FORM DATA";

        HttpResponse* post_res = post_req->send(body, strlen(body));
        if (!post_res) {
            printf("HttpRequest failed (error code %d)\n", post_req->get_error());
            //return 1;
        } else {
           printf("\n----- HTTP POST response -----\n");
           dump_response(post_res);
        }
        delete post_res;   // This line crashes app when (!post_res) or causes memory leak when the code runs in a loop.
        delete post_req;

[zhiyong-ft]

Just tested the same wifi hotspot with Sparkfun ESP8266 Thing Dev, HTTP GET to http://httpbin.org/status/418 worked just fine. So the problem is either in the wifi module (esp-wroom-02 + 1.70 at firmware) or ESP8266 driver on mbed side.

The problem now narrows down to, using UDP socket instead of builtin command (AT+CIPDOMAIN) to resolve domain to IP might be problematic on certain wifi hotspot setups.