MQTT not starting after reboot

[regfixit]

Am not sure if this was always the case, but since updating FW I find the MQTT does not connect when the device reboots. I have to go to the MQTT config page (which has all details saved and Enable MQTT tick box marked ) and it says in the password field. I enter the password and click Save and then it connects, although the password field still says .
I modified the handleFileRead() function to allow me to download the mqtt.json file and can see that all the details including the password are there, but it doesn't start unless I re-enter the password and save.
Any ideas on where to look ?

[visualapproach]

Hm, a recent PR changed the default to MQTT enabled = false. It should ofc load the right settings but maybe it's doing things in the wrong order or something. I'll check it out but will not have time until next week.
Thank you for your report

[regfixit]

I did some digging and think may know why it's happening. My day job is a SAP Consultant more used to ABAP programming and still getting to grips with OO stuff, but looking at the file names .cpp I guessed this is written in C++ which I've not used before, but from a bit of playing around with C 30 years ago I was trying to find the main() function to see where it all starts. After a bit of head scratching I realised that with Arduino the main function is in the arduino.h include and calls init(), setup() and loop(). I didn't see an init() but did see setup() and loop() so started with these.

The setup calls startMqtt() function. This calls loadMqtt() which reads the mqtt.json file and passes values from it into mqtt_info, including the field useMqtt. There is a line commented out that would have set a flag enableMqtt:
// enableMqtt = useMqtt; //will be set with start complete timer

after calling loadMqtt(), startMqtt() then calls mqttConnect() - this connects to the MQTT broker, but only if the enableMqtt flag is true.

I looked at where enableMqtt gets set and in the main.h it is first set to false.
It gets set based on mqtt_info->useMqtt in function gotIP() and also in handleSetMqtt().
I think gotIP() is called when the Wifi establishes a connection and gets an IP address.

So what I think happens is that enableMqtt is false to start off with when the system boots.
If the WiFi connects before the loadMqtt() is called then it will still be false when gotIP() is called.
Thereafter it is only set when handleSetMqtt() is called - which I think happens when you press Save on the MQTT config web page.
Or, I guess if the WiFi link drops and is then re-established.

I'm not sure why that line in loadMqtt() is commented out, or why mqttConnect uses a separate flag enableMqtt instead of directly using mqtt_info->useMqtt. I tried uncommenting the line in loadMqtt(), but that made the web gui unresponsive so I reverted the change. I then realised the line
// enableMqtt = useMqtt; //will be set with start complete timer
is maybe based on some old code version that didn't have mqtt_info and is using a now undefined variable useMqtt.
Probably should read
enableMqtt = mqtt_info->useMqtt;

I was going to try that, but was a bit worried I'd bricked the device when the webgui wasn't working after my first change. I'm doing this remotely via VPN as not at the cottage at the moment and was relieved it still let me upload previous firmware. I need to be sure I can turn it on remotely tomorrow so don't want to mess it up by trying anything else yet. So will have another try next week.

[visualapproach]

Good detective work there. It seems likely that is the culprit. You could check out the latest PRs version details to see where the MQTT was changed from earlier versions. Think you can find a mqttenabled=false instead of true somewhere. I'm on the move now so can't check myself yet.

[Gelbwichtel]

Hi regfixit, i had the same problem after upgrading to the last firmware. My last FW was from April2024.
In my case the problem was: First the access point is started and then a one-time attempt is made to establish the MQTT connection.
Of course this doesn't work.
Therefore, I have now reactivated the commented out startMqtt() in the gotIP() method.
Furthermore, I have the impression that the configuration is either no longer being written to the file system, or there are problems reading it. Therefore, I hardcoded the settings in loadMqtt().
Thanks again to visualapproach for the great project. I hope he find a solution soon.

[Gelbwichtel]

I just noticed that when I access the file system in mqtt.json I get the message "Permission denied".
Even if I delete it and rewrite the configuration, I still have no access. This isn't the case with the other JSON files.

[visualapproach]

It's on purpose to not show credentials:)

[Gelbwichtel]

OK. Makes sense. Thanks for the info.

[Gelbwichtel]

@visualapproach I'm still confused by the fact that reading mqtt.json in loadMqtt() isn't working. I noticed that in the old source code, the IP address was four individual integers, but it's now treated as a string. Is it possible that even though I have the new firmware installed, the website is still in its old state and is no longer compatible? Does the ESP8266 perhaps need to be completely (FW and FS) reinstalled?

[visualapproach]

I think it's a matter of the device doing stuff in the wrong order. Need to check it out but no spare time yet.

[Gelbwichtel]

No problem. i have a workaround.

[regfixit]

As visualapproach mentioned, the system stops you downloading the mqtt.json (and wifi.json) files as these contain passwords. In my local copy I commented out the code that prevents this [in handleFileRead() in main.cpp] and saw the file correctly written.
If you upgrade the firmware you also have to update the website files from the data directory - you can do this by uploading them one by one rather than building and uploading filesystem in PlatformIO. This avoids losing the wifi and mqtt details. I think you could also include these two files in your local data directory and then build and upload filesystem.
I was away over the weekend but have some time now to have another look into it. I think as visualapproach says it is just a sequencing things.
I think originally the Mqtt details were originally stored as individual variables (in the config.h there is a commented out section about MQTT server with lines that look like you could uncomment them to set details there, but they no longer work as the actual details have been moved into a structure sMQTT_info instead.

/*
 * MQTT Server
 *
 * You can modify this via Web UI.
 */
/** get or set the state of the MQTT server connection */
// bool useMqtt = false;
// /** get or set the MQTT server IP address */
// IPAddress mqttIpAddress(192,168,0,20);
// /** get or set the MQTT server port */
// int mqttPort = 1883;
// /** get or set the MQTT server username */
// String mqttUsername = "username";
// /** get or set the MQTT server password */
// String mqttPassword = "password";
// /** get or set the unique MQTT client ID */
// String mqttClientId = DEVICE_NAME;
// /** get or set the MQTT topic name */
// String mqttBaseTopic = DEVICE_NAME;
// /** get or set the MQTT telemetry interval */
// int mqttTelemetryInterval = 600;

struct sMQTT_info
{
    String mqttHost;
    int mqttTelemetryInterval;
    int mqttPort;
    String mqttUsername;
    String mqttPassword;
    String mqttClientId;
    String mqttBaseTopic;
    bool useMqtt = false;
};
sMQTT_info* mqtt_info;

but there is also a bit in the setup() module that sets the variables to default values

   LittleFS.begin();
    {
        HeapSelectIram ephemeral;
        bwc = new BWC;
        oneWire = new OneWire(231);
        tempSensors = new DallasTemperature(oneWire);
        // bootlogTimer = new Ticker;
        periodicTimer = new Ticker;
        startComplete_ticker = new Ticker;
        ntpCheck_ticker = new Ticker;
        // checkWifi_ticker = new Ticker;
        updateWSTimer = new Ticker;
        updateMqttTimer = new Ticker;
        mqtt_info = new sMQTT_info;
        mqtt_info->mqttBaseTopic = MQTT_BASE_TOPIC_F;
        mqtt_info->mqttClientId = MQTT_CLIENT_ID_F;
        mqtt_info->mqttHost = F("192.168.0.20");
        mqtt_info->mqttPassword = MQTT_PASSWORD_F;
        mqtt_info->mqttPort = 1883;
        mqtt_info->mqttTelemetryInterval = 600;
        mqtt_info->mqttUsername = MQTT_USER_F;
        mqtt_info->useMqtt = false;
        wifi_info = new sWifi_info{.enableWmApFallback = true};
    }

I think it's just a case of sorting out setting and getting things in a single place and the right order.

Also when I talk about the latest version I was meaning the latest Master release rather than the development branch. It might be that branch does things again a bit differently.

I'll have a more detailed look this week.

I'm enjoying learning new things and thanks once again @visualapproach for such a great project.

[Gelbwichtel]

Hi. Yes, my initial mistake was that I was coming from an older version, when the broker address still consisted of four individual fields. After uploading the mqtt.html file for the appropriate version, it now read the configuration correctly. Regarding the enable, I now also see clearly. As visualapproach commented in the loadMqtt() method, he commented out the assignment "//enableMqtt = mqtt_info->useMqtt" ("will be set with start complete timer") and added it to the end of gotIP(). The only problem is that mqttStart() is called in gotIP(), which actually requires enableMqtt=true. You might want to adjust the code in StartMqtt() so that after the loadMqtt() call, depending on mqtt_info->useMqtt, mqttConnect is executed. If that succeeds, enableMqtt is set to true, or false in the event of an error.

[regfixit]

As I don't have physical access to the device I created (my first) html page to show some of the variable values and added some flags - one to show how many times GotIP has been called and another to say whether enableMqtt was set by GotIP.
After initial boot it shows that GotIP has been called, but MQTT has not been connected:

Then a bit later the MQTT does actually connect by itself without me having to go to the MQTT config page.

Maybe it was always doing this, but I'm pretty sure I had a situation where it just stayed disconnected.

Rather than risk messing my device up remotely I bought a new nodemcu2 unit and uploaded to it. Now I can see from the serial output what is happening.
Setup > Start @ millis: 71
startWiFi() @ millis: 221
WiFi > using WiFi configuration with SSID VM0665067
WiFi > AP info loaded. Waiting for connection ...
Station > disconnected. Starting soft AP
WiFi > soft-AP configuration: OK
WiFi > soft AP mode: OK
WiFi > Soft AP IP: 192.168.4.2
HTTP Server > start/restart.
WS > start. IRam heap: 13336
OTA > start
MQTT > start. Iram heap: 12088
End of setup() @ Millis: 2393 @ line: 110. Heap: 36968
start of gotip millis = 6038
Soft AP > closed
Connected as station with localIP: 192.168.0.51
NTP > start
end of gotip millis = 6248
MQTT > Not connected
MQTT > connecting
MQTT > message not published
MQTT > times not published
MQTT > other published
MQTT > Sending HA discoverysensors
binary_sensors
switch
buttons
climate
MQTT > connect done
MQTT > sending config
MQTT > message published
MQTT > times published
MQTT > other published

So the device is starting and sets up the WiFi and calls MQTT start - at that point it hasn't acquired the IP address so enableMQtt is not set and it doesn't connect. Then the WiFI connects and it calls gotIP (which sets enableMqtt) and closes the soft AP. When the periodic timer runs it sees the enableMqtt is set, but mqttClient->loop is false it so it then starts Mqtt.
If it had acquired the IP address before the setup() had got as far as calling startMqtt and loadMqtt then the mqtt_info would have the default value of false and so gotIP would not have set enableMqtt and it would have stayed disconnected.

I've now uploaded the Development branch onto the test device so I can see what happens with that.
My feeling at the moment is to call the load_mqtt earlier, or to set enableMqtt within load_mqtt or just to use mqtt_info->useMqtt in place of enableMqtt (which I think is a vestige from when mqtt_info did not exist in the code version).

Will update when have more..

[visualapproach]

gotIP() will never run before mqtt_info is loaded from file. The callback function "cb_gotIP()" can be called at any time by the OS but it only sets a flag. gotIP() is called from loop() which will execute after setup(). Only thing I can think of for mqtt_info to be default/invalid is if the file read went wrong.

So it should work...

[regfixit]

I saw the same behaviour in the latest Dev release - i.e. initially stays disconnected and then when periodictimer runs, it connects, so not sure why @Gelbwichtel yours didn't subsequently connect a minute or two after getting its IP.
I've re-written some of the code to rationalise the flag for enabling MqTT - i.e. to only use the mqtt_info; updated the config.h to remove the comment lines that look like they can default the MqTT settings and added some more #define lines so that all the default Mqtt setting is stored in the config.h rather than having the IP address, port and telemetry interval in the main.cpp section.
Have tested following scenarios:

1. No mqtt.json saved - boots OK and does not try to connect to MqTT
2. Then add the the mqtt details in the config page and it connects.
3. Reboot and it connects as soon as it gets the IP address.
4. Remove the wifi credentials and leave the mqtt credentials and reboot - goes into AP mode and does not try to connect to MqTT until wifi details are entered.

Will create a pull request to incorporate changes