Exception(2) thrown after OTA update

[kamran00f]

Basic Infos

• This issue complies with the issue POLICY doc.
• I have read the documentation at readthedocs and the issue is not addressed there.
• I have tested that the issue is present in current master branch (aka latest git).
• I have searched the issue tracker for a similar issue.
• If there is a stack dump, I have decoded it.
• I have filled out all fields below.

Platform

• Hardware: [ESP-01]
• Core Version: [2020-10-11]
• Development Env: [Platformio]
• Operating System: [Ubuntu]

Settings in IDE

• Module: [Generic ESP8266 Module]
• Flash Mode: [DOUT]
• Flash Size: [1MB]
• lwip Variant: [v2 Lower Memory]
• Reset Method: [nodemcu]
• Flash Frequency: [40Mhz]
• CPU Frequency: [80Mhz]
• Upload Using: [SERIAL]
• Upload Speed: [115200] (serial upload only)

Problem Description

Exception thrown after OTA update. Decoded exception message:

Exception 2: InstructionFetchError: Processor internal physical address or data error during instruction fetch
PC: 0x3ffee790
EXCVADDR: 0x3ffee790

The exception is thrown after Progress 100% and before printing "End" in <ArduinoOTA.onEnd()> callback.

MCVE Sketch

#include <Arduino.h>
#include <ESP8266WiFi.h>
#include <errno.h>
#include <ArduinoOTA.h>

int ConnectToWifi(const char *ssid, const char *password, uint32_t timeout_ms)
{
    Serial.printf("Connecting to %s ", ssid);
    WiFi.begin(ssid, password);
    unsigned long ref_millis = millis();
    while(WiFi.status() != WL_CONNECTED)
    {
        Serial.print(".");
        delay(500);

        if(millis() - ref_millis > timeout_ms)
            return -ETIMEDOUT;
    }

    Serial.printf("\r\nConnected! Local IP: ");
    Serial.println(WiFi.localIP());
    return 0;
}

void StartOtaService(char *serverName, char *password, uint16_t port)
{
    ArduinoOTA.setHostname(serverName);
    ArduinoOTA.setPassword(password);
    ArduinoOTA.setPort(port);
    ArduinoOTA.setRebootOnSuccess(true);

    ArduinoOTA.onStart([]() { Serial.println("\r\nStart"); });
    ArduinoOTA.onEnd([]() { Serial.println("\r\nEnd"); });
    ArduinoOTA.onProgress([](unsigned int progress, unsigned int total) {
        Serial.printf("\r\nProgress: %u%%\r", (progress / (total / 100)));
    });
    ArduinoOTA.onError([](ota_error_t error) {
        Serial.printf("\r\nError[%u]: ", error);
        if(error == OTA_AUTH_ERROR)
            Serial.println("\r\nAuth Failed");
        else if(error == OTA_BEGIN_ERROR)
            Serial.println("\r\nBegin Failed");
        else if(error == OTA_CONNECT_ERROR)
            Serial.println("\r\nConnect Failed");
        else if(error == OTA_RECEIVE_ERROR)
            Serial.println("\r\nReceive Failed");
        else if(error == OTA_END_ERROR)
            Serial.println("\r\nEnd Failed");
    });

    ArduinoOTA.begin();
}

void setup() {
    Serial.Begin(115200);
    int status = ConnectToWifi(wifi_ssid, wifi_password, 10000);
    if(status < 0)
    {
        // Unable to connect to wifi in specified time, restart to try again
        ESP.restart();
    }

    StartOtaService("ServerName", "Auth", 10000);
    return 0;
}

void loop() {
    ArduinoOTA.handle();
}

Debug Messages

PC log:

Processing esp01_1m (platform: espressif8266; board: esp01_1m; framework: arduino)
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif8266/esp01_1m.html
PLATFORM: Espressif 8266 (2.6.2) > Espressif Generic ESP8266 ESP-01 1M
HARDWARE: ESP8266 80MHz, 80KB RAM, 1MB Flash
PACKAGES: 
 - framework-arduinoespressif8266 3.20704.0 (2.7.4) 
 - tool-esptool 1.413.0 (4.13) 
 - tool-esptoolpy 1.20800.0 (2.8.0) 
 - tool-mklittlefs 1.203.200522 (2.3) 
 - tool-mkspiffs 1.200.0 (2.0) 
 - toolchain-xtensa 2.40802.200502 (4.8.2)
LDF: Library Dependency Finder -> http://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Found 32 compatible libraries
Scanning dependencies...
Dependency Graph
|-- <ESP8266WiFi> 1.0
|-- <ArduinoOTA> 1.0
|   |-- <ESP8266WiFi> 1.0
|   |-- <ESP8266mDNS> 1.2
|   |   |-- <ESP8266WiFi> 1.0
Building in release mode
Retrieving maximum program size .pio/build/esp01_1m/firmware.elf
Checking size .pio/build/esp01_1m/firmware.elf
Advanced Memory Usage is available via "PlatformIO Home > Project Inspect"
RAM:   [====      ]  36.2% (used 29668 bytes from 81920 bytes)
Flash: [===       ]  30.9% (used 316036 bytes from 1023984 bytes)
Configuring upload protocol...
AVAILABLE: espota, esptool
CURRENT: upload_protocol = espota
Uploading .pio/build/esp01_1m/firmware.bin
02:57:52 [DEBUG]: Options: {'esp_ip': '192.168.1.140', 'host_ip': '0.0.0.0', 'esp_port': 10000, 'host_port': 21559, 'auth': 'Auth', 'image': '.pio/build/esp01_1m/firmware.bin', 'spiffs': False, 'debug': True, 'progress': True}
02:57:52 [INFO]: Starting on 0.0.0.0:21559
02:57:52 [INFO]: Upload size: 320192
02:57:52 [INFO]: Sending invitation to: 192.168.1.140
Authenticating...OK
02:57:52 [INFO]: Waiting for device...

Uploading: [                                                            ] 0% 
Uploading: [=                                                           ] 1% 
..... progresses to 100% .....
Uploading: [============================================================] 99% 
Uploading: [============================================================] 100% Done...

02:58:22 [ERROR]: Error Uploading
*** [upload] Error 1

ESP log:

      Scanning...                                                   
        Stehmann-guest                                                          
        WIFI                                                                    
        Hulsuls_internet                                                        
        CelenoInitialAP290F4B    

Connecting to WIFI .......                                               
Connected! Local IP: 192.168.1.140    
Start

Progress: 0%
..... Progresses to 100% .....                                                             
Progress: 99%                                                                   
Progress: 100%                                                                  
--------------- CUT HERE FOR EXCEPTION DECODER ---------------                  
                                                                                
Exception (2):                                                                  
epc1=0x3ffee790 epc2=0x00000000 epc3=0x00000000 excvaddr=0x3ffee790 depc=0x00000
                                                                                
>>>stack>>>                                                                     
                                                                                
ctx: cont                                                                       
sp: 3ffffbf0 end: 3fffffc0 offset: 0190                                         
3ffffd80:  3ffeed00 40204d28 3ffeecc4 0004e2c0                                  
3ffffd90:  3ffeecf0 00000000 3ffeecc4 40205964                                  
3ffffda0:  eb001000 00000001 000ac000 00000000                                  
3ffffdb0:  0004e2c0 3fff222c 3fffc718 3ffffe60                                  
3ffffdc0:  40203284 3ffffe60 3ffffdfd 40204dbd                                  
3ffffdd0:  00000034 0000000a 3ffffdfc 40205165                                  
3ffffde0:  3ffffe20 3ffffdf0 00000008 3ffeec6c                                  
3ffffdf0:  000002c0 0000000b 00000064 0000000d                                  
3ffffe00:  0000005c 00000000 00000000 00000001                                  
3ffffe10:  3ffe863a 00000000 3ffffdf0 3ffffe20                                  
3ffffe20:  00000000 00000000 3ffeecc4 0004e2c0                                  
3ffffe30:  000001c4 3ffeecc4 0004e2c0 40201084                                  
3ffffe40:  40206ae9 00000001 3ffeebd8 0004e2c0                                  
3ffffe50:  000001c4 3ffeecc4 3ffeebd8 40204651                                  
3ffffe60:  4020d6e0 00000000 00001388 4020922c                                  
3ffffe70:  3ffee75c 3fff1ac4 0000001f 4020d770                                  
3ffffe80:  7e01a8c0 4020d770 7e01a8c0 3ffeedc4                                  
3ffffe90:  3ffeec5c 000001c4 00000000 00000001                                  
3ffffea0:  40239c5b 00000000 4bc6a7f0 3ffeedc4                                  
3ffffeb0:  3fffdad0 00000000 3ffeebd8 40204684                                  
3ffffec0:  00000000 00000000 3ffeed84 4020141c                                  
3ffffed0:  3fffdad0 3ffee780 3ffee75c 4020152c                                  
3ffffee0:  00000000 3fffdad0 3ffeedc4 00000030                                  
3ffffef0:  00000000 00000000 00000001 3ffe84f4                                  
3fffff00:  00000000 3fffdad0 3ffeedc4 00000030                                  
3fffff10:  3ffefb84 3ffef364 3fff03a4 402082ba                                  
3fffff20:  00000000 00000000 4020d770 00000000                                  
3fffff30:  3ffefb84 3ffef364 3ffeebd8 402041fe                                  
3fffff40:  4020d770 00000000 402036f4 4020d158                                  
3fffff50:  4020d770 00000000 3ffeebd8 402011ca                                  
3fffff60:  007a1200 4fb769b0 3ffeeb00 3ffeedc4                                  
3fffff70:  00000000 00000000 3ffeebd8 40201402                                  
3fffff80:  00000000 00000000 00000001 4010022c                                  
3fffff90:  3fffdad0 00000000 3ffeed84 3ffeedc4                                  
3fffffa0:  3fffdad0 00000000 3ffeed84 40206500                                  
3fffffb0:  feefeffe feefeffe 3ffe84f4 40100e91                                  
<<<stack<<<                                                                     
                                                                                
--------------- CUT HERE FOR EXCEPTION DECODER ---------------                  
                                                                                
 ets Jan  8 2013,rst cause:1, boot mode:(3,6)                                   
                                                                                
load 0x4010f000, len 3664, room 16                                              
tail 0                                                                          
chksum 0xcc                                                                     
csum 0xcc                                                                       
v00000000                                                                       
~ld                                                                             
      Scanning...                                                   
        Stehmann-guest                                                          
        WIFI                                                                    
        Hulsuls_internet                                                        
        CelenoInitialAP290F4B                                                   
                                                                                
Connecting to WIFI .......                                               
Connected! Local IP: 192.168.1.140     

platformio.ini:

[env:esp01_1m]
platform = espressif8266
board = esp01_1m
framework = arduino
lib_extra_dirs = ~/Documents/Arduino/libraries
monitor_speed = 115200
lib_deps = ESP8266WiFi
board_build.partitions = mini_spiffs.csv
; board_build.partitions = huge_app.csv
board_build.ldscript = eagle.flash.1m.ld
board_build.filesystem = littlefs

upload_protocol = espota
upload_port = 192.168.1.140
upload_flags =
    --port=10000
    --auth='Auth'

Tried both "board_build.partitions = huge_app.csv" and "board_build.partitions = mini_spiffs.csv". Same result.

[earlephilhower]

Please decode the stack dump as requested in the template, using your own executable. We can't use the raw stack dump or PC/EXECVADDR.

[kamran00f]

Here is the decoded stack dump:

Exception Cause: 2  [InstructionFetchError: Processor internal physical address or data error during instruction fetch]

0x40204d28: MD5Builder::calculate() at ??:?
0x40205964: UpdaterClass::end(bool) at ??:?
0x40203284: WiFiClient::write(unsigned char const*, unsigned int) at ??:?
0x40204dbd: Print::write(char const*) at ??:?
0x40205165: unsigned int Print::printNumber<unsigned long>(unsigned long, unsigned char) at ??:?
0x40201084: std::_Function_handler<void (unsigned int, unsigned int), StartOtaService(char*, char*, unsigned short)::{lambda(unsigned int, unsigned int)#3}>::_M_invoke(std::_Any_data const&, unsigned int, unsigned int) at common.cpp:?
0x40206ae9: __delay at ??:?
0x40204651: ArduinoOTAClass::_runUpdate() at ??:?
0x4020d6e0: esp8266::MDNSImplementation::MDNSResponder::stcMDNSSendParameter::findCachedDomainOffset(void const*, bool) const at ??:?
0x4020922c: esp8266::MDNSImplementation::MDNSResponder::_updateProbeStatus() at ??:?
0x4020d770: esp8266::MDNSImplementation::MDNSResponder::stcMDNSSendParameter::findCachedDomainOffset(void const*, bool) const at ??:?
0x4020d770: esp8266::MDNSImplementation::MDNSResponder::stcMDNSSendParameter::findCachedDomainOffset(void const*, bool) const at ??:?
0x40239c5b: wifi_station_get_connect_status at ??:?
0x40204684: ArduinoOTAClass::handle() at ??:?
0x4020141c: HandleOtaService() at ??:?
0x4020152c: loop at ??:?
0x402082ba: esp8266::MDNSImplementation::MDNSResponder::enableArduino(unsigned short, bool) at ??:?
0x4020d770: esp8266::MDNSImplementation::MDNSResponder::stcMDNSSendParameter::findCachedDomainOffset(void const*, bool) const at ??:?
0x402041fe: ArduinoOTAClass::begin(bool) at ??:?
0x4020d770: esp8266::MDNSImplementation::MDNSResponder::stcMDNSSendParameter::findCachedDomainOffset(void const*, bool) const at ??:?
0x402036f4: std::_Function_base::_Base_manager<std::_Bind<std::_Mem_fn<void (ArduinoOTAClass::*)()> (ArduinoOTAClass*)> >::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) at ??:?
0x4020d158: std::_Function_handler<void (), std::_Bind<std::_Mem_fn<void (ArduinoOTAClass::*)()> (ArduinoOTAClass*)> >::_M_invoke(std::_Any_data const&) at ??:?
0x4020d770: esp8266::MDNSImplementation::MDNSResponder::stcMDNSSendParameter::findCachedDomainOffset(void const*, bool) const at ??:?
0x402011ca: std::_Function_base::_Base_manager<StartOtaService(char*, char*, unsigned short)::{lambda(ota_error_t)#4}>::_M_manager(std::_Any_data&, std::_Function_base::_Base_manager<StartOtaService(char*, char*, unsigned short)::{lambda(ota_error_t)#4}> const&, std::_Manager_operation) at common.cpp:?
0x40201402: StartOtaService(char*, char*, unsigned short) at ??:?
0x4010022c: ets_post at ??:?
0x40206500: loop_wrapper() at core_esp8266_main.cpp:?
0x40100e91: cont_wrapper at ??:?

[kamran00f]

Ok, I could narrow down where this is happening. It happens inside crc32(...) inside eboot_command_calculate_crc32(...) in eboot_command_write(...) in UpdaterClass::end(bool evenIfRemaining) inside if (_command == U_FLASH) {...}.

I checked the parameters that are passed to crc32 functions and they seem fine. This is the code:

static uint32_t eboot_command_calculate_crc32(const struct eboot_command* cmd)
{
    Serial.println("in eboot_command_calculate_crc32");
    Serial.print("cmd=");
    Serial.println((uint32_t)cmd);
    Serial.print("offset=");
    uint32_t offset = offsetof(struct eboot_command, crc32);
    Serial.println(offset);
    Serial.print("crc32 address=");
    Serial.println((uint32_t)crc32);
    
    uint32_t crc = crc32((const uint8_t*) cmd, offsetof(struct eboot_command, crc32), 0xffffffff);

    Serial.println("after crc32");
    return crc;
}

output:

in eboot_command_calculate_crc32
cmd=1073741216
offset=124
crc32 address=1073670160

But for some reason I didn't get inside crc32 function. So I made a copy of the crc32 function as shown in the following code:

static uint32_t local_crc32 (const void* data, size_t length, uint32_t crc)
{
    const uint8_t* ldata = (const uint8_t*)data;
    while (length--)
    {
        uint8_t c = pgm_read_byte(ldata++);
        for (uint32_t i = 0x80; i > 0; i >>= 1)
        {
            bool bit = crc & 0x80000000;
            if (c & i)
                bit = !bit;
            crc <<= 1;
            if (bit)
                crc ^= 0x04c11db7;
        }
    }
    return crc;
}

static uint32_t eboot_command_calculate_crc32(const struct eboot_command* cmd)
{
    Serial.println("in eboot_command_calculate_crc32");
    Serial.print("cmd=");
    Serial.println((uint32_t)cmd);
    Serial.print("offset=");
    uint32_t offset = offsetof(struct eboot_command, crc32);
    Serial.println(offset);
    Serial.print("crc32 address=");
    Serial.println((uint32_t)crc32);
    
    uint32_t crc = local_crc32((const uint8_t*) cmd, offsetof(struct eboot_command, crc32), 0xffffffff);

    Serial.println("after crc32");
    return crc;
}

And after this change, the OTA worked perfectly fine! So something is going wrong while calling crc32 function, but I could not figure out what.

[earlephilhower]

Your MCVE is not failing for me. I had to fix several compile errors, but after doing so I was able to upload 1st via serial port and then via Arduino OTA from the IDE using the "Auth" password. I never got PlatformIO working w/my own setup so can't try it out, but it should use espota.py, same as the IDE to do the upload.

Are you sure it fails with just the MCVE, and doesn't need your full sketch uploaded to crash?

The stack dump doesn't really shed much light, but the faulting PC = **inside RAM ** which is a bad thing.

This could indicate stack overflow (use Tools->Stack Protection->Enabled to turn on the runtime check...it's not perfect, but it can catch some things). It could also be RAM corruption somewhere along the way and a VTABLE/ptr. that was going into flash got overwritten with a 0x3fff(ram) address.

[kamran00f]

I tested the MCVE and OTA works again perfectly fine. I found the problem in my full sketch. I had a global object called crc32! The same name as the function name with problem. Sorry for bothering you and thank you @earlephilhower for your help. Closing this issue...

[lclem]

I had the same issue...
wow this is crazy, such an issue should be get caught at compile time!

[mcspr]

I had the same issue... wow this is crazy, such an issue should be get caught at compile time!

Do you have another example, or it is still this crc32 issue?
Is this a problem with Arduino IDE or PlatformIO (as above)?
Do we have some kind of link-order issue so crc32 from local build overrides crc32 from framework?

[lclem]

This is still the crc32 issue. I had this on Arduino IDE.

…

On Sun, Nov 6, 2022, 14:43 Max Prokhorov ***@***.***> wrote: I had the same issue... wow this is crazy, such an issue should be get caught at compile time! Do you have another example, or it is still this crc32 issue? Is this a problem with Arduino IDE or PlatformIO (as above)? Do we have some kind of link-order issue so crc32 from local build overrides crc32 from framework? — Reply to this email directly, view it on GitHub <#7651 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABCBUPWFWSFMUYW4K3P7B7TWG6YWJANCNFSM4SLNSUZQ> . You are receiving this because you commented.Message ID: ***@***.***>