IOT ESP8266 Tutorial – OTA Software Updates – Arduino IDE

December 7, 2016 ESP8266, Featured Topic, IOT, OurWeather, SwitchDoc Products, Tutorials, WeatherPlus

IOT ESP8266 Tutorial – OTA Software Updates – Arduino IDEIMG_8160

One of the most important aspects for designing the OurWeather grove connector based no solder weather kit was to provide a method for updating the software on the ESP8266 based WeatherPlus board, the heart of the OurWeather Kit.   We wanted to be able to provide a mechanism for customers to be able to download and install new OurWeather software updates over the ESP8266 WiFi connection allowing the customer to update the OurWeather firmware without connecting Thumbnail-clear cloud dark pantsup to a computer.  Since many people mount this kit remotely in a box, this was an important feature.   The WeatherPlus board can also be programmed directly from the Arduino IDE (here is the Open Source OurWeather WeatherPlus software).

 [section]

Here is a list of our other ESP8266 Tutorials

Solar Power your ESP8266

IOT ESP8266 Tutorial – Displaying the data on the Raspberry Pi with MatPlotLib

IOT ESP8266 Tutorial – Connect to the IBM Bluemix Internet of Things

IOT ESP8266 Tutorial – Using the Arduino IDE

IOT ESP8266 Timer Tutorial – Arduino IDE

OurWeather – ESP8266 Air Quality Sensor Prototype

IOT ESP8266 Tutorial – Using nodeMCU/LUA

OurWeather REST Interface to the Raspberry Pi Released

How to Do It

The update process consists of two sets of software.  First is the software on the ESP8266 and the second section is the web server that serves the update file using a PHP updater file.

The ESP8266 Update Mechanism

The ESP8266 firmware has a set of calls that enable the user program to be updated remotely.    Note:  In order to do this you need to make sure that your program is less than one half of the available program memory.  In the case of the ESP8266 used by WeatherPlus, the total program memory available is 1MB.   This limits your program to less than 500K bytes.    The current OurWeather image is about 400K bytes.

The code is pretty simple.

Note:   By experimenting, here are the instructions we print out on the display for the customer.    Following this sequence gives about 98% updating success on the first IOT ESP8266 Tutorialtry:

  • – Wait 120 Seconds
  • – Unplug Power
  • – Wait 15 Seconds
  • – Plug Power In

Usually the ESP8266 reboots itself using the new software.   If the update fails, it boots up with the old software.

This does have ramifications if you have a very remote station.  In that case, you should add a hardware watchdog time to remove power and reapply power (you can use our Dual WatchDog Timer and our USB PowerControl to accomplish this).

Following is the ESP8266 update code from OurWeather in the Arduino IDE.   Pretty simple, eh?  the updateDisplay routine displays status on the OurWeather OLED display for the customer to see.

    


    updateDisplay(DISPLAY_UPDATING);
    delay(5000);

    updateDisplay(DISPLAY_UPDATE_FINISHED);
    t_httpUpdate_return ret = ESPhttpUpdate.update("www.switchdoc.com", 80, "/OurWeatherUpdater.php", WEATHERPLUSESP8266VERSION);
    switch (ret) {
      case HTTP_UPDATE_FAILED:
        Serial.println("[update] Update failed.");
        updateDisplay(DISPLAY_NO_UPDATE_FAILED);
        delay(5000);

        return 1;

        break;
      case HTTP_UPDATE_NO_UPDATES:
        Serial.println("[update] Update no Updates.");
        updateDisplay(DISPLAY_NO_UPDATE_AVAILABLE);
        delay(5000);
        return 2;
        break;
      case HTTP_UPDATE_OK:

        Serial.println("[update] Update ok."); // may not called we reboot the ESP

        return 3;
        break;

 

 

The Webserver Side – The PHP File

The second part of the software is the PHP file on a web server that serves up the file to the ESP8266.    This is accomplished by using PHP to provide the file in the proper http format to the ESP8266.   Here is the file in it’s entirety.   We have left the debugging statements in the PHP to show you how we read out the HTTP headers in the error_log during debugging.   See the section below on debugging.

<?PHP
error_log("in Updater",0);

function returnCurrentVersion()
{
        error_log("return Current Version=",0);
        return "019";  // latest version of OurWeather Released
}
header('Content-type: text/plain; charset=utf8', true);

function check_header($name, $value = false) {
    if(!isset($_SERVER[$name])) {
        return false;
    }
    if($value && $_SERVER[$name] != $value) {
        return false;
    }
    return true;
}

function sendFile($path) {
    error_log("sendFile=".$path,0);
    header($_SERVER["SERVER_PROTOCOL"].' 200 OK', true, 200);
    header('Content-Length: '.filesize($path), true);
    header("Cache-Control: no-cache");
    header("Pragma: no-cache");
    header('Content-Type: application/octet-stream', true);
    header('Content-Disposition: attachment; filename='.basename($path),true);
    header('Content-Transfer-Encoding: binary', true);
error_log("$path:  ".$path,0);
error_log("md5_file:".md5_file($path),0);
    header('x-MD5: '.md5_file($path), true);
error_log("Content-Length:  ".filesize($path),0);


$headers = headers_list();
$myheaders = "";

foreach ($headers as $header => $value) {
        $myheaders=$myheaders."$header: $value \n";
}
error_log('headers='.$myheaders);

    $returnError = readfile($path);
error_log("returnError5=".$returnError);
}

if(!check_header('HTTP_USER_AGENT', 'ESP8266-http-Update')) {
    header($_SERVER["SERVER_PROTOCOL"].' 403 Forbidden', true, 403);
    echo "Only used for OurWeather Softwdare Updater.\n";
    exit();
}

if(
    !check_header('HTTP_X_ESP8266_STA_MAC') ||
    !check_header('HTTP_X_ESP8266_AP_MAC') ||
    !check_header('HTTP_X_ESP8266_FREE_SPACE') ||
    !check_header('HTTP_X_ESP8266_SKETCH_SIZE') ||
    !check_header('HTTP_X_ESP8266_CHIP_SIZE') ||
    !check_header('HTTP_X_ESP8266_SDK_VERSION') ||
    !check_header('HTTP_X_ESP8266_VERSION')
) {
    header($_SERVER["SERVER_PROTOCOL"].' 403 Forbidden', true, 403);
    echo "Only used for OurWeather Software Updater. (header)\n";
    exit();
}

$db = array(
    "18:FE:AA:AA:AA:AA" => "DOOR-7-g14f53a19",
    "18:FE:AA:AA:AA:BB" => "TEMP-1.0.0"
);



    if(returnCurrentVersion() != $_SERVER['HTTP_X_ESP8266_VERSION']) {

        error_log("sending ./bin/"."OW-".returnCurrentVersion().".bin",0);
        sendFile("./bin/"."OW-".returnCurrentVersion().".bin");
    } else {
        header($_SERVER["SERVER_PROTOCOL"].' 304 Not Modified', true, 304);
        error_log("304 Not modified",0);
    }
    exit();
header($_SERVER["SERVER_PROTOCOL"].' 500 no version for ESP MAC', true, 500);

?>

 

If It Doesn’t Work

We had a pretty easy time of getting this to work on our web server.   However, about 6 months after release, the update mechanism quit working.    We dug into the problem and after investigating both ends of the connection, we found that our Apache web server was not sending the “Content-Length” HTTP header and so the ESP8266 updating software rejected the request because the ESP8266 receiving software did not know how long the software update file was.   After a lot of investigating we found out that the Apache server had been updated on our hosting service and it changed the default behavior of the server when sending files.   The default behavior was now changed to sending the files compressed (gzip) and in that case, Apache no longer sends the “Content-Length” header and that breaks the system.

The fix was to add this directive into your .htaccess file.   This stops the compression of the HTTP pages and Apache then sends the “Content-Length” and the ESP8266 accepts the update from the server.   Note the commented line would just apply that to just PHP files and you could modify it to only apply to the Updater file alone.   We did not test those two configurations but we will in the future.

 

< IfModule mod_env.c>

     SetEnv no-gzip dont-vary

     #SetEnvIfNoCase Request_URI "\.php$" no-gzip dont-vary

</ IfModule>