ESP8266: Add built-in hostname resolution handling (disabled by default) #12234

michalpasztamobica · 2020-01-09T20:14:30Z

Summary of changes

Fixes #11982 (some routers fail to work with the UDPSocket-approach, but are proved to work when ESP uses its embedded hostname resolution AT command).

The feature is disabled by default and can be enabled in mbed_app.json with "esp8266.built-in-dns": true. When enabled then the synchronous hostname resolution will use the ESP's CIPDOMAIN AT command, but asynchronous resolution will keep on using the UDPSockets as before (no point rewriting the nsapi_dns). Also caching does not work.

I tested with the netsocket-dns testsuite. Obviously caching tests often fail, but if lucky I can get 15/16 passes.

@zhiyong80, please verify if this works with your router.

Impact of changes

None, until user enables the esp8266.built-in-dns option.

Migration actions required

None.

Documentation

"help" section added to mbed_lib.json.

Pull request type

[] Patch update (Bug fix / Target update / Docs update / Test update / Refactor)
[x] Feature update (New feature / Functionality change / New API)
[] Major update (Breaking change E.g. Return code change / API behaviour change)

Test results

[] No Tests required for this change (E.g docs only update)
[x] Covered by existing mbed-os tests (Greentea or Unittest)
[] Tests / results supplied as part of this PR

Reviewers

@AnttiKauppila
@VeijoPesonen
@SeppoTakalo
@zhiyong80

ciarmcom · 2020-01-09T22:00:40Z

@michalpasztamobica, thank you for your changes.
@AnttiKauppila @VeijoPesonen @SeppoTakalo @ARMmbed/mbed-os-ipcore @ARMmbed/mbed-os-maintainers please review.

zhiyong-ft · 2020-01-10T02:20:14Z

AT+CIPDOMAIN works but I still get -3009 error

AT> AT+CIPDOMAIN="httpbin.org"
AT? +CIPDOMAIN:%*15[^
]
%n
AT= +CIPDOMAIN:34.193.212.251

AT? OK
%n
AT= OK

HttpRequest failed (error code -3009)

michalpasztamobica · 2020-01-10T07:19:18Z

I just pushed an astyle error fix. Still waiting for feedback from @zhiyong80 , in the issue report there is hope that his will work.
Please, bear in mind, only the simplest synchronous gethostbyname() will use the CIPDOMAIN AT command.

kjbracey · 2020-01-10T07:59:12Z

Fine as long as it's off by default. We did at some point use the built-in resolver, but stopped doing so because (at least in the firmware at the time) it failed to cope with some responses containing long names from some Amazon cloud servers.

michalpasztamobica · 2020-01-10T08:07:23Z

@kjbracey-arm , yep, I have also found the Espressif forum topic on this 64B response limitation...

This feature is added with minimum effort (no cache'ing, no async operation) to address a very specific usecase reported by @zhiyong80 . It will be kept off by default.

zhiyong-ft · 2020-01-10T17:41:42Z

Just tested the commit a33d397 . When buil-in-dns is turned on, modified mbed-http-example with added ntp works just fine. When built-in-dns is turned off, the same old problem happens again. So as far as the DNS problem is concerned, I believe the referenced commit solves the problem.

Unfortunately, the out of memory problem when -3009 error is thrown creeped back again. I thought I figured out, but no.

Test code:

#include "select-demo.h"

#if DEMO == DEMO_HTTP

#include "mbed.h"
#include "http_request.h"
#include "network-helper.h"
#include "mbed_mem_trace.h"
#include "NTPClient/NTPClient.h"
#include "pins.h"

DigitalOut espEn(ESP_EN);
DigitalOut espRst(ESP_RST);
DigitalIn espBoot(ESP_BOOT, PullNone);

void dump_response(HttpResponse* res) {
    printf("Status: %d - %s\n", res->get_status_code(), res->get_status_message().c_str());

    printf("Headers:\n");
    for (size_t ix = 0; ix < res->get_headers_length(); ix++) {
        printf("\t%s: %s\n", res->get_headers_fields()[ix]->c_str(), res->get_headers_values()[ix]->c_str());
    }
    printf("\nBody (%d bytes):\n\n%s\n", res->get_body_length(), res->get_body_as_string().c_str());
}

Timer t;

int main() {
    time_t curtime = time(NULL);
    printf("System started at: %s\r", ctime(&curtime));
    espEn = 1;
    espRst = 1;
    wait(10.0);
    // Connect to the network with the default networking interface
    // if you use WiFi: see mbed_app.json for the credentials
    NetworkInterface* network = connect_to_default_network_interface();
    if (!network) {
        printf("Cannot connect to the network, see serial output\n");
        return 1;
    }
    NTPClient ntp(network);
    ntp.setTime("pool.ntp.org");
    curtime = time(NULL);
    printf("RTC time set to: %s\r\n", ctime(&curtime));

    for (size_t i = 0; i < 1; i++)
    {
        // By default the body is automatically parsed and stored in a buffer, this is memory heavy.
        // To receive chunked response, pass in a callback as last parameter to the constructor.
        t.reset();
        HttpRequest* get_req = new HttpRequest(network, HTTP_GET, "http://httpbin.org/status/418");

        HttpResponse* get_res = get_req->send();
        if (!get_res) {
            printf("HttpRequest failed (error code %d)\n", get_req->get_error());
        }

        printf("\n----- HTTP GET response -----\n");
        dump_response(get_res);

        delete get_req;
    }

    wait(osWaitForever);
}

#endif

Stdio output with ESP8266 in debug mode, and built-in-dns turned on:

System started at: Thu Jan  1 00:00:00 1970

[NWKH] Connecting to network...
AT> AT
AT? OK
%n
AT< AT
AT= OK

AT> AT+RST
AT? OK
%n
AT< AT+RST
AT= OK

AT? %n
AT? %n
AT<  ets Jan  8 2013,rst cause:1, boot mode:(3,2)
AT< load 0x40100000, len 2592, room 16 
AT< tail 0
AT< chksum 0xf3
AT< load 0x3ffe8000, len 764, room 8 
AT< tail 4
AT< chksum 0x92
AT< load 0x3ffe82fc, len 676, room 4 
AT< tail 0
AT< chksum 0x22
AT< csum 0x22
AT< 2nd boot version : 1.7(5d6f877)
AT< SPI Speed : 40MHz
AT< SPI Mode : QIO
AT< SPI Flash Size & Map: 16Mbit(1024KB+1024KB)
AT< jump to run user1 @ 1000
AT? %n
AT< üŒ…2�±0x1fc000ŒâœAT! ready
AT? %n
AT< 
AT? %n
AT> AT
AT? OK
%n
AT< AT
AT= OK

AT> ATE0
AT? OK
%n
AT< ATE0
AT= OK

AT> AT+UART_CUR=115200,8,1,0,3
AT? OK
%n
AT= OK

AT> AT+GMR
AT? AT version:%*d.%*d.%*d.%*d%n
AT= AT version:1.7.0.0
AT? OK
%n
AT< (Aug 16 2018 00:57:04)
AT< SDK version:3.0.0(d49923c)
AT< compile time:Aug 23 2018 16:58:12
AT< Bin version(Wroom 02):v1.7.0
AT= OK

AT> AT+GMR
AT? SDK version:%*d.%*d.%*d%n
AT< AT version:1.7.0.0(Aug 16 2018 00:57:04)
AT= SDK version:3.0.0
AT? OK
%n
AT< (d49923c)
AT< compile time:Aug 23 2018 16:58:12
AT< Bin version(Wroom 02):v1.7.0
AT= OK

AT> AT+CWMODE_DEF=1
AT? OK
%n
AT= OK

AT> AT+CWCOUNTRY_DEF=0,"US",1,13
AT? OK
%n
AT= OK

AT> AT+CWCOUNTRY_CUR=0,"US",1,13
AT? OK
%n
AT= OK

AT> AT+CIPRECVMODE=1
AT? OK
%n
AT= OK

AT> AT+CWMODE_CUR=1
AT? OK
%n
AT= OK

AT> AT+CIPMUX=1
AT? OK
%n
AT= OK

AT> AT+CWDHCP_CUR=1,1
AT? OK
%n
AT= OK

AT> AT+CWJAP_CUR="SSID","PASSWORD"
AT? OK
%n
AT! WIFI 
AT? %*12[^"]
%n
AT= CONNECTED

AT? OK
%n
AT! WIFI 
AT? %*12[^"]
%n
AT= GOT IP

AT? OK
%n
AT= OK

AT[NWKH] Connected to the network
? %n
AT> AT+CIFSR
AT? +CIFSR:STAIP,"%*15[^"]"%n
AT= +CIFSR:STAIP,"192.168.1.70"
AT? OK
%n
AT< 
AT< +CIFSR:STAMAC,"b4:e6:2d:6b:13:af"
AT= OK

[NWKH] IP address: 192.168.1.70
AT> AT+CIPDOMAIN="pool.ntp.org"
AT? +CIPDOMAIN:%*15[^
]
%n
AT= +CIPDOMAIN:208.76.53.137

AT? OK
%n
AT= OK

AT? %n
AT? %n
AT> AT+CIPSTART=0,"UDP","208.76.53.137",123
AT? OK
%n
AT< 0,CONNECT
AT= OK

AT> AT+CIPSEND=0,48
AT? >%n
AT! OK
AT? >%n
AT< 
AT= >
AT? SEND OK%n
AT<  
AT< Recv 48 bytes
AT= SEND OK
AT? %n
AT< 
AT? %n
AT? %n
AT? %n
AT? %n
AT? %n
AT? %n
AT! +IPD
AT? ,%*d,%n
AT= ,0,
AT? %*d:%n
AT= 48:
AT? %n
AT? %n
AT> AT+CIPCLOSE=0
AT? OK
%n
AT! 0,CLOSED
AT? OK
%n
AT< 
AT= OK

RTC time set to: Fri Jan 10 17:19:31 2020

AT> AT+CIPDOMAIN="httpbin.org"
AT? +CIPDOMAIN:%*15[^
]
%n
AT= +CIPDOMAIN:34.193.212.251

AT? OK
%n
AT= OK

AT? %n
AT> AT+CIPSTART=0,"TCP","34.193.212.251",80
AT? OK
%n
AT< 0,CONNECT
AT= OK

AT? %n
AT> AT+CIPSEND=0,47
AT? >%n
AT! OK
AT? >%n
AT< 
AT= >
AT? SEND OK%n
AT<  
AT< Recv 47 bytes
AT= SEND OK
AT? %n
AT< 
AT? %n
AT? %n
AT? %n
AT? %n
AT! +IPD
AT? ,%*d,%n
AT= ,0,
AT? %*d
%n
AT= 516

AT? %n
AT? %n
AT> AT+CIPRECVDATA=0,2048
AT? OK
%n
AT! +CIPRECVDATA,
AT? %*ld:%n
AT= 516:
AT? OK
%n
AT< 
AT= OK

AT? %n
AT> AT+CIPCLOSE=0
AT? OK
%n
AT! 0,CLOSED
AT? OK
%n
AT< 
AT= OK


----- HTTP GET response -----
Status: 418 - I'M A TEAPOT
Headers:
	Access-Control-Allow-Credentials: true
	Access-Control-Allow-Origin: *
	Date: Fri, 10 Jan 2020 17:19:31 GMT
	Referrer-Policy: no-referrer-when-downgAT? %n
rade
	Server: nginx
	X-Content-Type-Options: nosniff
	X-Frame-Options: DENY
	x-more-info: http://tools.ietf.org/html/rfc2324
	X-XSS-Protection: 1; mode=block
	Content-Length: 135
	Connection: keep-alive

Body (135 bytes):


    -=[ teapot ]=-

       _...._
     .'  _ _ `.
    | ."` ^ `". _,
    \_;`"---"`|//
      |       ;/
      \_     _/
        `"""`

Stdio output when built-in-dns is turned off:


[NWKH] Connecting to network...
AT> AT
AT? OK
%n
AT< AT
AT= OK

AT> AT+RST
AT? OK
%n
AT< AT+RST
AT= OK

AT? %n
AT? %n
AT<  ets Jan  8 2013,rst cause:1, boot mode:(3,7)
AT< load 0x40100000, len 2592, room 16 
AT< tail 0
AT< chksum 0xf3
AT< load 0x3ffe8000, len 764, room 8 
AT< tail 4
AT< chksum 0x92
AT< load 0x3ffe82fc, len 676, room 4 
AT< tail 0
AT< chksum 0x22
AT< csum 0x22
AT< 2nd boot version : 1.7(5d6f877)
AT< SPI Speed : 40MHz
AT< SPI Mode : QIO
AT< SPI Flash Size & Map: 16Mbit(1024KB+1024KB)
AT< jump to run user1 @ 1000
AT? %n
AT< üŒ…2�±0x1fc000ŒâœAT! ready
AT? %n
AT< 
AT? %n
AT> AT
AT? OK
%n
AT< AT
AT= OK

AT> ATE0
AT? OK
%n
AT< ATE0
AT= OK

AT> AT+UART_CUR=115200,8,1,0,3
AT? OK
%n
AT= OK

AT> AT+GMR
AT? AT version:%*d.%*d.%*d.%*d%n
AT= AT version:1.7.0.0
AT? OK
%n
AT< (Aug 16 2018 00:57:04)
AT< SDK version:3.0.0(d49923c)
AT< compile time:Aug 23 2018 16:58:12
AT< Bin version(Wroom 02):v1.7.0
AT= OK

AT> AT+GMR
AT? SDK version:%*d.%*d.%*d%n
AT< AT version:1.7.0.0(Aug 16 2018 00:57:04)
AT= SDK version:3.0.0
AT? OK
%n
AT< (d49923c)
AT< compile time:Aug 23 2018 16:58:12
AT< Bin version(Wroom 02):v1.7.0
AT= OK

AT> AT+CWMODE_DEF=1
AT? OK
%n
AT= OK

AT> AT+CWCOUNTRY_DEF=0,"US",1,13
AT? OK
%n
AT= OK

AT> AT+CWCOUNTRY_CUR=0,"US",1,13
AT? OK
%n
AT= OK

AT> AT+CIPRECVMODE=1
AT? OK
%n
AT= OK

AT> AT+CWMODE_CUR=1
AT? OK
%n
AT= OK

AT> AT+CIPMUX=1
AT? OK
%n
AT= OK

AT> AT+CWDHCP_CUR=1,1
AT? OK
%n
AT= OK

AT> AT+CWJAP_CUR="SSID","PASSWORD"
AT? OK
%n
AT! WIFI 
AT? %*12[^"]
%n
AT= DISCONNECT

AT? OK
%n
AT! WIFI 
AT? %*12[^"]
%n
AT= CONNECTED

AT? OK
%n
AT! WIFI 
AT? %*12[^"]
%n
AT= GOT IP

AT? OK
%n
AT= OK

AT[NWKH] Connected to the network
? %n
AT> AT+CIFSR
AT? +CIFSR:STAIP,"%*15[^"]"%n
AT= +CIFSR:STAIP,"192.168.1.70"
AT? OK
%n
AT< 
AT< +CIFSR:STAMAC,"b4:e6:2d:6b:13:af"
AT= OK

[NWKH] IP address: 192.168.1.70
AT> AT+CIFSR
AT? +CIFSR:STAIP,"%*15[^"]"%n
AT= +CIFSR:STAIP,"192.168.1.70"
AT? OK
%n
AT< 
AT< +CIFSR:STAMAC,"b4:e6:2d:6b:13:af"
AT= OK

AT? %n
AT> AT+CIPSTART=0,"UDP","8.8.8.8",53
AT? OK
%n
AT< 0,CONNECT
AT= OK

AT> AT+CIPSEND=0,30
AT? >%n
AT! OK
AT? >%n
AT< 
AT= >
AT? SEND OK%n
AT<  
AT< Recv 30 bytes
AT= SEND OK
AT? %n
AT< 
AT? %n
AT? %n
AT? %n
AT? %n
AT? %n
AT> AT+CIPCLOSE=0
AT? OK
%n
AT! 0,CLOSED
AT? OK
%n
AT< 
AT= OK

error: 'gethostbyname("pool.ntp.org")' failed with code -3009
AT? %n
AT> AT+CIPSTART=0,"UDP","",123
AT? OK
%n
AT< IP ERROR
AT! ERROR
AT(Aborted)
AT> AT+CIPSTART=0,"UDP","",123
AT? OK
%n
AT< 
AT< IP ERROR
AT! ERROR
AT(Aborted)
RTC time set to: Fri Jan 10 17:28:02 2020

AT> AT+CIFSR
AT? +CIFSR:STAIP,"%*15[^"]"%n
AT< 
AT= +CIFSR:STAIP,"192.168.1.70"
AT? OK
%n
AT< 
AT< +CIFSR:STAMAC,"b4:e6:2d:6b:13:af"
AT= OK

AT? %n
AT> AT+CIPSTART=1,"UDP","8.8.8.8",53
AT? OK
%n
AT< 1,CONNECT
AT= OK

AT? %n
AT> AT+CIPSEND=1,29
AT? >%n
AT! OK
AT? >%n
AT< 
AT= >
AT? SEND OK%n
AT<  
AT< Recv 29 bytes
AT= SEND OK
AT? %n
AT< 
AT? %n
AT? %n
AT? %n
AT? %n
AT? %n
AT> AT+CIPCLOSE=1
AT? OK
%n
AT! 1,CLOSED
AT? OK
%n
AT< 
AT= OK

HttpRequest failed (error code -3009)

----- HTTP GET response -----


++ MbedOS Error Info ++
Error Status: 0x8001011F Code: 287 Module: 1
Error Message: Operator new out of memory

Location: 0x800B449
Error Value: 0x800E166
Current Thread: main Id: 0x20002314 Entry: 0x800BB07 StackSize: 0x2000 StackMem: 0x20002BF0 SP: 0x20004A5C 
For more info, visit: https://mbed.com/s/error?error=0x8001011F&tgt=NUCLEO_F401RE
-- MbedOS Error Info --

michalpasztamobica · 2020-01-10T17:47:42Z

@zhiyong80 , just to make sure I got it right...
"built-in-dns" : false -> -3009 from gethostbyname from one unfortunate router
"built-in-dns" : true -> no errors even from the one unfortunate router
If so, then it sounds like the problem is solved.

@AnttiKauppila , @SeppoTakalo , @VeijoPesonen , may I ask for a code review, please?

zhiyong-ft · 2020-01-10T18:10:34Z

@zhiyong80 , just to make sure I got it right...
"built-in-dns" : false -> -3009 from gethostbyname from one unfortunate router
"built-in-dns" : true -> no errors even from the one unfortunate router
If so, then it sounds like the problem is solved.

@AnttiKauppila , @SeppoTakalo , @VeijoPesonen , may I ask for a code review, please?

Yes, the -3009 error only shows up when built-in-dns equals to false. The would be confirmed by:

AT> AT+CIPSTART=0,"UDP","8.8.8.8",53

0xc0170 · 2020-01-13T09:35:04Z

CI started

mbed-ci · 2020-01-13T10:41:27Z

Test run: SUCCESS

Summary: 11 of 11 test jobs passed
Build number : 1
Build artifacts

ciarmcom requested review from AnttiKauppila, SeppoTakalo, VeijoPesonen and a team January 9, 2020 22:00

ciarmcom added the needs: review label Jan 9, 2020

ESP8266: Add built-in DNS resolution (disabled by default)

f032274

michalpasztamobica force-pushed the esp8266_builtin_dns branch from a33d397 to f032274 Compare January 10, 2020 07:17

VeijoPesonen approved these changes Jan 13, 2020

View reviewed changes

AnttiKauppila approved these changes Jan 13, 2020

View reviewed changes

0xc0170 approved these changes Jan 13, 2020

View reviewed changes

0xc0170 added needs: CI and removed needs: review labels Jan 13, 2020

0xc0170 added ready for merge release-version: 6.0.0-alpha-2 Second pre-release version of 6.0.0 and removed needs: CI ready for merge labels Jan 13, 2020

0xc0170 merged commit 194fc66 into ARMmbed:master Jan 15, 2020

michalpasztamobica mentioned this pull request Mar 25, 2020

ESP8266 set_network static function not working. #12552

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

ESP8266: Add built-in hostname resolution handling (disabled by default) #12234

ESP8266: Add built-in hostname resolution handling (disabled by default) #12234

Uh oh!

michalpasztamobica commented Jan 9, 2020

Uh oh!

ciarmcom commented Jan 9, 2020

Uh oh!

zhiyong-ft commented Jan 10, 2020

Uh oh!

michalpasztamobica commented Jan 10, 2020

Uh oh!

kjbracey commented Jan 10, 2020

Uh oh!

michalpasztamobica commented Jan 10, 2020

Uh oh!

zhiyong-ft commented Jan 10, 2020

Uh oh!

michalpasztamobica commented Jan 10, 2020

Uh oh!

zhiyong-ft commented Jan 10, 2020 •

edited

Loading

Uh oh!

0xc0170 commented Jan 13, 2020

Uh oh!

mbed-ci commented Jan 13, 2020

Uh oh!

Uh oh!

ESP8266: Add built-in hostname resolution handling (disabled by default) #12234

ESP8266: Add built-in hostname resolution handling (disabled by default) #12234

Uh oh!

Conversation

michalpasztamobica commented Jan 9, 2020

Summary of changes

Impact of changes

Migration actions required

Documentation

Pull request type

Test results

Reviewers

Uh oh!

ciarmcom commented Jan 9, 2020

Uh oh!

zhiyong-ft commented Jan 10, 2020

Uh oh!

michalpasztamobica commented Jan 10, 2020

Uh oh!

kjbracey commented Jan 10, 2020

Uh oh!

michalpasztamobica commented Jan 10, 2020

Uh oh!

zhiyong-ft commented Jan 10, 2020

Uh oh!

michalpasztamobica commented Jan 10, 2020

Uh oh!

zhiyong-ft commented Jan 10, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

0xc0170 commented Jan 13, 2020

Uh oh!

mbed-ci commented Jan 13, 2020

Test run: SUCCESS

Uh oh!

Uh oh!

zhiyong-ft commented Jan 10, 2020 •

edited

Loading