Part of the Getting Started with ESP8266 + Arduino series.

As part of my exploration of the basics of the ESP8266/Arduino/PlatformIO development experience, I wanted to learn how to make HTTPS requests.

I figured this would be more challenging than using Golang on a Raspberry Pi, for example, because TLS certificate verification requires knowing the current time and knowing which root certificate(s) are trustworthy. Both of these are free if you’re running Linux, like the Raspberry Pi, but on a microcontroller you’ll have to do it yourself. Luckily, the Arduino system and the ESP8266 Arduino core make both of those tasks pretty easy.

Basic Setup

There is a “basic HTTPS client” example, but it doesn’t do a very good job of demonstrating how to use the Arduino/ESP8266 HTTPClient class (which is used to actually make HTTP requests) with the WiFiClientSecure class.

We start by declaring a global, shared WiFiClientSecure:

#include <WiFiClientSecureBearSSL.h>
BearSSL::WiFiClientSecure wifiClient;

Then, pass that wifiClient when calling .begin(…) on our HTTP client. Here’s how that looks (with error-handling and other code removed for clarity):

#include <ESP8266HTTPClient.h>

    // ...
    HTTPClient httpClient;
    httpClient.begin(wifiClient, "https://ip.dzdz.cz");
    int respCode = httpClient.GET();
    if (respCode >= 400) {
        // HTTP error
    } else if (respCode > 0) {
        String result = httpClient.getString();
        // Do something with result...
    } else {
        // Other error occurred
    }
    httpClient.end();
    // ...

First, though, we’ll need to tell the wifiClient how to verify TLS certificates. Let’s discuss our options…

Certificate Verification Methods

Certificate Store

Docs & examples: BearSSL_CertStore example; Core documentation

This was the approach I used for this demo, because will work with the widest array of servers and closely mirrors how I’m used to this process working. (This is how pretty much everything on your computer & phone verifies certificates.) This method involves generating a list of trusted root certificates, writing it to your ESP8266’s flash memory (either in LittleFS or SPIFFS), and then telling the WiFiClientSecure to use it to verify certificates.

This approach does have a downside: the need to figure out which certificates to include a root certificate store. The most common approach here is to download and use the full set of root certificates included in Mozilla’s CA Certificate Program. My friend David Adrian notes that this isn’t a perfect approach from a security standpoint; this list is intended to support web PKI across a range of applications, not just TLS connections, so you’ll end up including some root certificates that shouldn’t be used for HTTPS. He also notes, “there’s not really a better option.” If using this approach, you should be prepare to update your root certificate store. For more on this topic, check out the “How to be a Certificate Authority, feat. Ryan Sleevi” episode of the “Security. Cryptography. Whatever.” podcast.

I’ll walk through using a certificate store in more detail below; first, a discussion and some links on other methods you might use:

X509List

Docs & examples: BearSSL_Validation example; Core documentation

I haven’t played with this, but based on the documentation and examples, this is conceptually similar to the Certificate Store. This class (BearSSL::X509List) lets you tell the WifiClientSecure to trust one or more certificates.

The difference from BearSSL::CertStore is that these certificates are stored in RAM, rather than stored in LittleFS (or SPIFFS) on the device. You can see this in the example code — it uses a certificate defined as a literal in a header, rather than reading it from flash as the CertStore example does.

(Incidentally, the documentation and example linked above don’t cover adding multiple certificates; to do that, look into the X509List’s append methods.

This is clearly simpler and easier than using a CertStore, but it’ll take up more and more of the device’s limited RAM as you add certificates. Use this if you only need/want to trust one or a few certificate authorities for your application.

In a future blog post, I’ll explore using X509List in an application where I know I only need to trust certificates from Let’s Encrypt.

Pinned public key (fragile)

Docs & examples: BearSSL_Validation example; Core documentation

With this approach, you simply embed the server’s public key in your code, and tell BearSSL only to trust certificates which use that key.

Though straightforward, this approach has some downsides:

• A key has no expiration date. If the key is compromised in the future, your application will continue trusting certificates issued with that key. Expiration dates on TLS certificates exist, in part, to mitigate this; even if one is compromised, the damage is contained to the time before the certificate expires. Trusting a public key for eternity does not provide such safety.
• It’s fragile. Public keys can change for any number of reasons, including key rotation after a compromise, simply forgetting the key file when moving to a new server, etc. HTTP Public Key Pinning is deprecated for this reason; too many things can easily go wrong.

Check certificate fingerprint (very fragile)

Docs & examples: BearSSL_Validation example; Core documentation

Using this approach, you hardcode the cryptographic fingerprint of the server’s TLS certificate in your code. I’ve seen a lot of Arduino example code using this approach, and while it is more-or-less secure, this approach has significant downsides:

• It’s even more fragile than public key pinning. If the server’s certificate changes at all — say, because the old one expired — this check will fail. TLS certificates change a lot; they are not designed to be permanent fixtures. In fact, as of mid-2020, Google Chrome, Mozilla Firefox, and Apple Safari no longer accept certificates with validity periods longer than a year. So, if you use this approach for any server you don’t control, your code is nearly guaranteed to break within a year.
• From what I can tell in the documentation, this check also ignores the certificate’s expiration time, so you have the same problem as public key pinning: if the certificate’s key is ever compromised, your application will continue trusting it forever.

If you really can’t use a certificate store or X509List, you should consider using a pinned public key instead of a hardcoded certificate fingerprint.

Skip verification (insecure)

It’s possible to tell WiFiClientSecure to completely skip verification, trusting absolutely any certificate presented for any server. This is completely insecure, and you should not do this.

Allow any self-signed certificate (insecure)

It’s also possible to configure WiFiClientSecure to accept any self-signed certificate. This is also totally insecure; it’s no better security-wise than completely skipping verification, and you should not do this. I don’t know why this option even exists.

Making HTTPS Requests Using a Certificate Store

Now the fun part: how do we use a certificate store for HTTPS connections on ESP8266/Arduino?

Prerequisite: Generating the Certificate Store

First, we’ll need to generate a certificate store to flash to the device. The most common — really, the only practical way to do this as a hobbyist — is to use Mozilla’s list of trusted root certificates.

Download and run this Python script, certs-from-mozilla.py. This script works on macOS, and it ought to work just fine on Linux. It will fetch Mozilla’s current list of trusted root certificates and pack them into an archive which you can flash to your ESP8266’s memory.

It’s simple to use:

$ ./certs-from-mozilla.py
AC Camerfirma, S.A.:AC Camerfirma SA CIF A82743287:http://www.chambersign.org -> data/ca_000.der
# ... output snipped ...
TrustCor Systems:TrustCor Systems S. de R.L.:TrustCor Certificate Authority -> data/ca_146.der
ar: creating archive data/certs.ar

$ ls
certs-from-mozilla.py* data/

$ ls data
certs.ar

The resulting file, certs.ar, is what you need to flash to your device.

Prerequisite: Flashing the Store to ESP8266 Flash memory

I’ll perform this step using the PlatformIO toolchain, since that’s what I’m using for my experiments. If you’re using the standard Arduino IDE, refer to these instructions provided by the ESP8266 Arduino core documentation.

At the root of your project, create a data folder, and place certs.ar inside. You can see this structure in my example project.

For new projects, you should choose to use LittleFS rather than SPIFFS, as SPIFFS is deprecated. To do that, add this line to platformio.ini:

board_build.filesystem = littlefs

Connect your dev board to your computer via USB, and then from your project’s root directory, run platformio run --target uploadfs. This will upload everything from your data folder to LittleFS in the board’s flash memory. If all goes well, your output should look something like this.

Prerequisite: Initializing the Filesystem & Certificate Store

First of all, at the beginning of the program’s setup function, call LittleFS.begin(). This took me a few minutes to figure out — I skipped over it while reading the example code!

We need to declare a global, shared CertStore, tell it to load certs.ar from flash memory, and finally configure our WiFiClientSecure to use that certificate store.

Putting that all together, it looks like this:

#include <Arduino.h>
#include <ESP8266WiFi.h>
#include <CertStoreBearSSL.h>
#include <FS.h>
#include <LittleFS.h>

BearSSL::CertStore certStore;
WiFiClientSecure wifiClient;

void setup() {
    LittleFS.begin();

    int numCerts = certStore.initCertStore(LittleFS, PSTR("/certs.idx"), PSTR("/certs.ar"));
    Serial.printf_P(PSTR("%lu: read %d CA certs into store\r\n"), millis(), numCerts);
    if (numCerts == 0) {
        Serial.println(F("!!! No certs found. Did you run certs-from-mozilla.py and upload the LittleFS directory?"));
        return;
    }

    wifiClient.setCertStore(&certStore);
}

Making a Request Using the Certificate Store

Once the certificate store is set up to read certs.ar from flash, and your WiFiClientSecure is configured to use the certificate store, the hard work is done. Simply make an HTTPS request as described in “Basic Setup,” above:

#include <ESP8266HTTPClient.h>

    // ...
    HTTPClient httpClient;
    httpClient.begin(wifiClient, "https://ip.dzdz.cz");
    int respCode = httpClient.GET();
    if (respCode >= 400) {
        // HTTP error
    } else if (respCode > 0) {
        String result = httpClient.getString();
        // Do something with result...
    } else {
        // Other error occurred
    }
    httpClient.end();
    // ...

I’ll note that you can reuse that httpClient instance, but you may encounter a bug when reusing it to connect to a different server. See my post, “Reusing an ESP8266 HTTPClient”, for details.

Timekeeping

Proper TLS certificate verification (as performed by the certificate store and X509List approaches, described above) requires knowing the current time, since certificates are only valid for a certain time window. Surprisingly, thanks to some timekeeping functionality built into the ESP8266 Arduino core, this turned out to be straightforward.

I started with this implementation, borrowed from some example code buried in the Arduino ESP8266WiFi library:

#include <Arduino.h>
#include <time.h>

// Set time via NTP, as required for x.509 validation
void setClock() {
  configTime(3 * 3600, 0, "pool.ntp.org", "time.nist.gov");
  Serial.print("Waiting for NTP time sync: ");
  time_t now = time(nullptr);
  while (now < 8 * 3600 * 2) {
    delay(500);
    Serial.print(".");
    now = time(nullptr);
  }
  Serial.println("");
  struct tm timeinfo;
  gmtime_r(&now, &timeinfo);
  Serial.print("Current time: ");
  Serial.print(asctime(&timeinfo));
}

This fetches the time from an NTP server asynchronously, waits for that process to complete, and then prints the current time.

I also wanted to figure out how to handle timezones in the Arduino world, so I did a little searching. (This will be useful for another project I have planned.) It turns out that the example code above is hardcoded to a timezone of UTC+3, with no Daylight Saving Time offset. A comment in the configTime function it uses explicitly says, “The other API should be preferred”. This naturally led me to wonder, “what other function?”

That other version of configTime takes a timezone argument, and using that function I came up with this implementation:

#include <Arduino.h>
#include <time.h>
#include <TZ.h>

/**
 * Sets the system time via NTP, as required for x.509 verification
 * see https://github.com/esp8266/Arduino/blob/master/libraries/ESP8266WiFi/examples/BearSSL_CertStore/BearSSL_CertStore.ino
 */
void setClock() {
    #define CFG_TZ TZ_America_Detroit
    configTime(CFG_TZ, "pool.ntp.org", "time.nist.gov");

    Serial.printf_P(PSTR("%lu: Waiting for NTP time sync "), millis());
    time_t now = time(nullptr);
    while (now < 8 * 3600 * 2) {
        delay(250);
        Serial.print(".");
        now = time(nullptr);
    }
    Serial.print(F("\r\n"));
    struct tm timeinfo;
    gmtime_r(&now, &timeinfo);
    Serial.printf_P(PSTR("Current time (UTC):   %s"), asctime(&timeinfo));
    localtime_r(&now, &timeinfo);
    Serial.printf_P(PSTR("Current time (Local): %s"), asctime(&timeinfo));
}

The CFG_TZ definition refers to a zone defined in TZ.h, part of the Arduino core.

So now when this demo starts up, it prints the current time in UTC and in my local timezone. Nice!

Performance Concerns & Solutions

Processor Frequency

The ESP8266 Arduino core documentation notes that the computations involved in a TLS handshake are expensive, particularly on a microcontroller with no hardware acceleration for the operations involved.

The documentation therefore recommends running the ESP8266 at 160MHz, rather than the default 80MHz, for a sketch that uses TLS. According to some forum threads, this can allegedly cause instability or issues with WiFi, but it is officially supported and I haven’t noticed any such issues in my (admittedly brief) testing.

To switch to 160MHz, add the line board_build.f_cpu = 160000000L to your platformio.ini file.

Sessions

The ESP8266 Arduino core supports TLS sessions, which, if supported by the server, can make subsequent HTTPS connections faster. The trade-off, of course, is using a little bit of precious RAM as a cache.

Using this feature is straightforward. Define a BearSSL::Session in the global scope, alongside your WiFiClientSecure:

#include <BearSSLHelpers.h>
BearSSL::Session tlsSession;

Then, tell the WiFiClientSecure to use that session cache:

wifiClient.setSession(&tlsSession);

An example is also provided by the ESP8266 Arduino core.

Limiting Cipher Selection

It is possible to limit cipher selection, which can theoretically be useful if you know ahead of time what cipher(s) your server supports (and will continue supporting). I would not recommend doing this. You’ll likely end up using a less secure configuration, and if the server’s configuration ever has to change, all your ESP8266 clients will break.

And don’t call setCiphersLessSecure(), for obvious reasons — the hint is in the method name.

As the docs say, “there is very rarely reason to use these calls”.

Check Return Codes!

The subset of C++ we’re working with here doesn’t have any advanced error-handling features; checking functions’ return codes for errors is critical.

The codes returned by HTTPClient’s GET() method and friends break down like this:

• Negative numbers: The request did not succeed. This includes connection errors, timeouts, and seemingly (?) running out of RAM (that’d be HTTPC_ERROR_TOO_LESS_RAM).
• Numbers in the range of HTTP status codes (approximately, 1xx - 5xx): The request succeeded and the server returned this response.
• Anything else: These codes are undefined or undocumented.

In my code, I simplify this a bit, by:

1. First, checking if the return code is equal to or greater than 400 (indicates an HTTP error);
2. Then, assuming anything above zero is a success;
3. Finally, passing anything else (eg. library-level errors) to HTTPClient::errorToString(...) and logging the resulting error string.

Limiting TLS Versions

The BearSSL:WiFiClientSecure class allows you to limit the TLS versions your application will use.

TLS 1.0 and 1.1, though supported by BearSSL, contain various security flaws. They are no longer supported by some widely-used client software; for example, Google Chrome removed support for it completely in Chrome 84, which was released in July 2020. Unless you know you need a legacy TLS version, it’s a good idea to use TLS 1.2. Do this with this line:

wifiClient.setSSLVersion(BR_TLS12, BR_TLS12);

Unfortunately, this API doesn’t allow setting only a minimum version requirement. I’d like to see that feature added, since really what I want to do is to say “don’t use anything older than TLS 1.2, but if a server allows using anything newer, do that.” (Go allows doing this; see the MaxVersion field in crypto/tls.Config.) As it stands, this line will preclude using TLS 1.3, once BearSSL supports it.

That seems unlikely to be a practical issue in the near term, though: BearSSL does not currently support TLS 1.3. The project’s TLS 1.3 Status page outlines the plan to support TLS 1.3; it’s ten steps, some of which are quite involved; and I don’t see any indication that anyone has started to work on it.

tl;dr: Limiting your project to TLS 1.2 is the best you can do, and using TLS 1.3 isn’t going to be possible for some time anyway.

Putting it all together

Honestly, reading my sample project’s code is probably easier than reading through this blog post! It walks through setting up the basic WiFi connection, WiFiClientSecure, TLS session, and certificate store; then using an HTTP client to make multiple HTTPS requests. Plus there’s some ArduinoJson usage thrown in for good measure.

And of course, you can always refer to the official documentation on BearSSL::WifiClientSecure.

Appendix: Working with JSON

I had planned to add some notes here on using ArduinoJson, but their website and documentation are stellar. Go read them!