Emad Alashi – a long-time coworker of mine – recently blogged about a soil moisture sensor that he’s built using exactly this combination of M5Stack Atom + ESPHome. He’s working on battery power, and thus needs to put the device into deep sleep most of the time to conserve energy.
The challenge: Combining deep sleep behaviour with over-the-air updates. It’s incredibly hard to push an over-the-air firmware update to the device when it’s only awake for a few seconds at a time!
The solution: Publish a flag that says “stay awake”. When the device next wakes up, it’ll read this flag, and skip a further sleep cycle. It’s essentially an advertised maintenance mode.
Emad hit this challenge in his project and implemented the solution that’s documented with the ESPHome Deep Sleep Component.
That solution relies on an MQTT broker holding the message, and the device checking for this pending message on boot:
I wanted to document an alternate approach, that avoids the need to introduce an MQTT connection, and sticks with a purely Home Assistant-native approach instead.
One of the things I really like about ESPHome is how natively it is integrated with Home Assistant, and the entity model that’s already there. Let’s avoid adding another protocol and set of messaging concepts in the mix.
Introducing a Global Flag
First up, we need a way of storing the “please stay awake” flag.
Home Assistant has the concept of “helpers”, which are just easy places to store an extra little bit of state like this.
They’re available in the web interface under Configuration > Helpers.
Screenshot of Home Assistant Configuration screen
I’ve gone ahead and created one for our flag. This will be a single global flag, that all of my sleep-aware devices can watch and respond to.
Type
Toggle / Boolean
Name
Prevent Deep Sleep
Icon
mdi:sleep-off
Entity ID
input_boolean.prevent_deep_sleep
Helper configuration
Screenshot of helper configuration screen
If you’re more of a fan of managing Home Assistant via configuration.yaml, you can declare the helper there too:
input_boolean:
prevent_deep_sleep:
name: Prevent Deep Sleep
icon: mdi:sleep-off
When our device boots up, and connects to the Home Assistant API, it’ll read the helper value in to the local state. If Home Assistant isn’t connected, or is offline, it’ll default to false.
⚠ Important note: The connection is actually triggered from Home Assistant to the ESP-device, and it’s only initiated if the device is setup under Home Assistant > Integrations. Just because you can see the device in the ESPHome web interface doesn’t mean it’s actually setup as an integration. If your firmware seems to be ignoring the helper value, it’s probably not actually connected.
With the value available, we can now combine that with the deep_sleep component to setup the right balance of power saving logic.
Here’s the complete, working ESPHome config for an M5Stack Atom Lite:
substitutions:
device_name: demo_deepsleep
friendly_name: Demo Deep Sleep
## Board config
esphome:
name: ${device_name}
platform: ESP32
board: m5stack-core-esp32
on_boot:
then:
– light.turn_on: status_led
– script.execute: consider_deep_sleep
## Boilerplate, same for all devices
wifi:
ssid: !secret wifi_ssid
password: !secret wifi_password
power_save_mode: none
ap:
ssid: Fallback ${device_name}
logger:
api:
password: !secret esphome_secret
ota:
password: !secret esphome_secret
## Hardware
light:
– platform: fastled_clockless
chipset: WS2812B
pin: 27
num_leds: 1
rgb_order: GRB
id: status_led
## Deep Sleep
deep_sleep:
id: deep_sleep_control
sleep_duration: 30s
# Will only pick up a value if this device is configured in Home Assistant > Integrations
# If the device isn't configured, or Home Assistant is offline, it'll default to false
binary_sensor:
– platform: homeassistant
id: prevent_deep_sleep
entity_id: input_boolean.prevent_deep_sleep
script:
– id: consider_deep_sleep
mode: queued
then:
– delay: 10s
– if:
condition:
binary_sensor.is_on: prevent_deep_sleep
then:
– logger.log: 'Skipping sleep, per prevent_deep_sleep'
Now I have a simple, global toggle when I want to put devices into development / maintenance mode.
The process becomes:
I want to do some device maintenance
I turn the “Prevent Deep Sleep” toggle on in Home Assistant
Over time, all of the different battery-based devices come out of their regular sleep cycles, and then stay on
I do the maintenance I want
I turn the “Prevent Deep Sleep” toggle back to off
All the devices resume their normal, power-saving sleep cycle
No extra brokers or message constructs had to be deployed. 🤘
Tip: Quick Bar
I don’t really want to add this toggle to the regular UI surface in Home Assistant, but I also don’t want to go digging for it in the dev tools every time I want to turn it on or off.
Home Assistant includes a ‘quick bar‘, modelled off the command palette in VS Code.
Just press e (for ‘entity’) as a hotkey anywhere in the frontend, then type the name of the toggle:
Our goal is to be able to quickly and cheaply integrate new sensors and controls. We want to be able to buy a flow meter / particulate matter sensor / pressure transducer / other crazy thing off AliExpress, and get it integrated into our control plane without it becoming a massive project.
Too many electronics and automation projects just die on the desk. We want the flexibility of our own electronics, but we want to be off the breadboard and into production as fast as we can.
As a software developer, the place I often getting tripped up is, amusingly, at the coding stage. Some fairly simple electronics suddenly need a lot of code to actually integrate well. Writing it yourself is a blackhole that’s easy to fall into.
Last year, I fell in love with ESPHome: the perfect piece of glue between ESP-based devices and Home Assistant.
I live in an inner-city apartment. There’s a concrete slab, brick walls, and no ceiling cavity access. Oh, and we rent, so drilling into things is frowned upon. The home automation potential in a scenario like this consists of some coloured lights, and watering about four plants on a timer. It’s not exactly an inspiring IoT site. There’s also not much wrong with the humble light switch that works instantly, every time.
In complete contrast to this, my parents live on 100 acres / 40 hectares of land, a few hours out of the city. It’s quintessential rural Australia. They’re mostly off the grid: there’s a skerrick of Telstra 4G signal, and some power, except when there isn’t, which is surprisingly often. This is an absolute IoT playground, and we’ve given that a fair crack over the seven years as they’ve established themselves there.
Opportunity
On a property like this, water is critical. There are the basic living requirements, the opportunity of gardens, and the very real risk of bushfires. All up, we have ~300,000 litres of water tanks on the property, and then a small dam and two creeks. We use sensors to track tank volumes. We measure flow rates at key plumbing points (for both instantaneous oversight and tracking cumulative usage). We use power meters to detect when pressure pumps are running.
Energy management is also important. Whilst there is the option of grid connectivity, we try to run as off-grid as possible, or at least export-only. This requires some smarts around load management. For example, instead of just having a hot water system that aggressively chases a specific temperature, we want the hot water systems to head for a temperature range, but only use excess solar production to do it. If there’s a bit less sun for the day, it’s ok if the water it a few degrees cooler: don’t burn down the batteries or import power just to hit a specific temperature.
And then there’s a safety aspect. The property is on the top of an escarpment where storms can roll in fast from a few different directions. By running things like lightning sensors on the property, we can trigger our own localised alerts for approaching storms.
Challenges
The challenge is to convert all these possibilities into something real, that works, and doesn’t cost an absolute mint. Over the years, we found this incredibly hard to solve for. You’ll find a solution for measuring tank volume, but it comes with its own LoRA gateway, a cloud dependency, and a new app. You’ll find a cheap Z-Wave temperature sensor, but it’s only really good for a room, and doesn’t have a probe that you can put into the measurement point in a hot water system. You’ll find a flow meter, but it’s only an industrial solution that wants to talk RS485 serial. Who even does that anymore⁈ You’ll find a garage door opener that works for the extra-high roller doors on the shed, but it has its own 433MHz RF remote.
It’s easy to end up with a horrible mishmash of radio technologies, software platforms, and APIs, not to mention some scary pricing as you drift from the traditional world of home automation into the more dated world of industrial automation.
Goal
We want to be able to dream up crazy new ideas, pick and choose the right sensors, and then integrate them with relative ease and consistency. That means a balance between the ability to build new things ourselves, but not having to go and custom fabricate a circuit board and write new firmware just to add a new sensor.
Considerations
Sense
Most sensors give out some kind of analogue signal (like a pressure sensor where the output voltage varies from 0V to 5V depending on applied pressure), a pulse (like a flow meter that pulses once for every 500mL of water that flows through it), or a digital signal (like a temperate and humidity sensor with a 1-Wire output).
To handle all of these scenarios, we’ll need some GPIO pins (the most basic of digital connections), and something with an analogue-to-digital-converter onboard (so that we can measure that variable pressure scenario).
Affect
Most of the outputs that we’re dealing with are straight up binary: turn a pump on, open a valve, flash a light. This means that they can be again driven by some GPIO pins, paired with an appropriately sized relay.
For more complex outputs, like sound, we can defer back to more consumer-grade hardware, like just broadcasting an announcement on the Google Home speaker in the kitchen.
Plug and Play
As much as we’re building things ourselves, we don’t need to put too many barriers in front of ourselves. We’re after a module that we can connect sensors to easily, without too much soldering, and certainly not having to build up our own circuit boards / Veroboard. We want to be out of the lab and into the environment as fast as possible.
Secure
There’s a persistent joke about how the ‘S’ in IoT stands for security.
Security was absolutely a consideration for us though: both on-property, and when it comes to remote access. When you’re switching things like power, or controlling a precious resource like water, you want to be confident that you’re the only person in control.
Our preference has been to keep connectivity and control local to the property, via authenticated connections, and then apply a separate remote access approach further up the stack. This means the hardware needs to have enough power and smarts to handle secured local connections, and not be dependent on its own path to the internet.
Compute
Some sensors require both precise timing and persistence to count things like pulses and turn them into natural measures. For example, a flow meter might give you a signal pulse for very 500mL of water that flows through it. If you miss a pulse and stop counting for a bit, you’re missing water. Our preference has been to count pulses on the hardware attached to the sensor, and then report back the natural values (L/min, L/day) whenever the network and central compute is available. We want to keep sensor-specific concepts, like pulses, at the sensor, and just send meaningful information over the network.
As much as we want things to be connected, they can still be somewhat smart in their own right. If a hardware module has both a temperature sensor and a relay to turn a heating element on and off, it’s perfectly capable of being a thermostat on its own, regardless of what’s happening to the wider network or any centralised compute. It will be smarter when the central controls are online, because it can be aware of other datapoints like the solar charge status, but it doesn’t have to be totally bound to the availability of those other systems to operate its basic function. Achieving this balance requires us to be able to run basic logic directly on the module.
The world is not static. If we want to run logic on these devices, and keep them secure, they need to be able to receive firmware updates over-the-air. We don’t want to be clambering around under sheds with laptops to reprogram a thermostat.
Connect
Network standards are still the multi-billion-dollar question in IoT.
Early on, we deployed a number of Z-Wave and Zigbee based devices. These are two different mesh protocols, at the mid-point of their VHS vs. Betamax battle for dominance. They’re common in consumer automation solutions like smart switches, and good for extremely low power environments, like where you want to be able to throw a temperature sensor in the corner of a room with a coin-cell battery in it and then forget about it for a year. The sensor ecosystem is very consumer focussed (you’ll find a million temperate sensors, but no tank pressure sensors). The communication protocol is constrained: by design, it’s a very low bandwidth data network, operating up to 40kbps for Z-Wave, or a whopping 250kbps for Zigbee. Range is limited to keep within the power limits, so typically as low as ~15-20m. There’s no common way of building on-device logic, and if you do manage to, then it’s incredibly hard to apply firmware updates over-the-air for either of them.
Our exploration continued into LoRA and NB-IoT, but we’ve opted away from both for this property. They’d each be very compelling options if we wanted to cover the larger property, such as if it was more of a working farm with distributed infrastructure than predominantly bushland and gardens with clustered buildings.
Ultimately, we settled on Wi-Fi as our preferred connectivity on the property. We’re already got great coverage throughout the house, cottage, shed, and key outdoor locations via a UniFi deployment. Whilst this is heavily centred on the built infrastructure, and not the full 100 acres of property, that’s where most of the sense-and-act needs to occur anyway. Investing in the Wi-Fi coverage provides other benefits, like Wi-Fi calling for mobiles where we’re otherwise on the fringe of 4G coverage. The UniFi infrastructure is readily extensible, as Lars has proven on his property with the deployment of Llama Cam, and even extending the coverage to places without power. Finally, it gives us a pretty amazing management plane that’s a lot nicer to work with than trying to diagnose a Z-Wave/Zigbee mesh.
UniFi infrastructure topology
Power
We’re happy to depend on a power source for most devices: they’re usually located near other loads that are already powered, or it’s easy enough to get power to them, such as dropping in a 12V cable out to a water tank at the same time as trenching the plumbing in. If we really want to run on battery, it’ll be ok to have a larger 12v battery and a small solar panel or something: we don’t need to try and run off a coin-cell battery for years on end.
Cost
The approach needs to be reasonably affordable if it’s going to grow over time: tens of dollars to add a new thing, not hundreds.
Whilst there’s no room to optimise on sensor cost, it does give something to calibrate against for the target cost of the compute module. We wanted to target ~$5 per module. It felt like the right cost, relative to the sensors themselves.
Hardware Options
So many options! Let’s run through them against those considerations:
Raspberry Pi
By this point, most software developers are jumping in with “Raspberry Pi! You can even buy hats for them!”. That’s certainly a response I started from. It feels safe: it has ports that you’re used to, you can plug in screens, it runs a whole operating system, and it has an IO header exposed for those plug-and-play scenarios.
Raspberry Pi 4
They’re also completely overpowered for what we need here. These things are small PC replacements: they’re full computers, not basic microcontrollers. Memory capacity starts in the gigabytes when we only need megabytes. They can run dual 4K display outputs when we only need to switch a couple of wires on and off. They need an SD card with a whole operating system on it just to boot. They suck up 3A of power, which is at the expensive end of USB power supplies. They’re also quite bulky compared to other options we’ll look at through this post.
A Pi, power supply, and SD card will quickly run to ~$50, which is 10x our cost target.
✅ Sense ✅ Affect
✅ Plug-and-Play ✅ Secure
🤯 Compute ✅ Connect
😐 Power ❌ Cost
Raspberry Pi Zero W
This is as small as the Raspberry range goes, but it’s still a similar position as the main Raspberry Pi models: completely overpowered for this purpose, with a 1GHz processor, 512MB of RAM, and HDMI video output. Thankfully, it’s less power hungry, so we’re back into the range of any-USB-power-will-do. It’s also physically a lot smaller: look at the relative size of those micro USB ports.
Raspberry Pi Zero W
It still needs a micro-SD card in addition to the base board, which takes the bill up to ~$15, still 3x our cost target. From a plug-and-play perspective, you’ll have to first spend time soldering on your own header strip, or pay a bit extra for one pre-soldered.
✅ Sense ✅ Affect
✅ Plug-and-Play ✅ Secure
🤯 Compute ✅ Connect
✅ Power ❌ Cost
Arduino and Adafruit
As we carve away ports and computer power that we don’t need, our next stop is the Arduino or Adafruit Feather range of boards.
Each of these ecosystems has a range of different connectivity options: boards that come with Wi-Fi, Bluetooth, LTE, NB-IoT, LoRA, or none of the above.
They have Wi-Fi on board, a USB port for power and programming, a stack of GPIO pins, and an analogue input. There’s no additional SD card required: there’s flash memory on the board itself. You can buy a version with pre-soldered headers, making that plug-and-play scenario easier.
The compute is right sized for basic sensors: an 80MHz processor with 4MB of flash memory attached.
The only thing on here that’s a little bit extraneous for our scenario is the Li-Po battery connection. (That’s the black connector at the top corner, next to the micro-USB.) The board can both run off a Li-Po battery, and recharge one, as there’s a charging circuit built into it as well. But, for our scenario where we said permanent power was ok, the charging circuit just adds more cost to the board.
Unfortunately, the boards are still up around $20, which is 4x our cost target. There’s also a lot of exposed electronics, so we’d need to factor a case into the price yet.
✅ Sense ✅ Affect
✅ Plug-and-Play ✅ Secure
✅ Compute ✅ Connect
✅ Power ❌ Cost
Discovering the ESP
What we’ve looked at so far in this post are different boards.
As I was researching around different boards, I kept running into phrases like “Arduino compatible”, and references to ATmega, ESP8266, or ESP32 chipsets.
It starts to get interesting when you split up the concept of a board, versus a chipset.
The chipset is the processor at the core of each of these boards: the smarts that makes it tick. There are a small number of very popular chipsets. There are then lots of different boards that package these chipsets up with other supporting systems and accessories to make them more accessible: USB serial drivers, voltage regulators, battery charge circuits, different breakout headers, and different physical form factors. It’s these extra systems that drive the cost up, and the brand recognition of the boards that drives the margin.
After some very nerdy reading 🤓, I got quite excited by the ESP range, specifically the ESP8266 and ESP32 chipsets. It turns out these are very popular with makers and manufacturers alike because they hit an interesting sweet spot of cost and capability. If you’ve got a Wi-Fi enabled smart plug in your house, there’s a decent chance that it has an ESP inside it.
These chips were never really designed as a general-purpose compute unit: the ESP8266 came first, and it was intended as a Wi-Fi modem, to be added on to some other device. It has enough compute to run a full TCP/IP stack, some leftover generic pins, and the ability to re-flash it with new firmware. There was originally only Chinese-language documentation, but the low price point and interesting feature set led the maker community off into investigating it further and translating the documentation. The manufacturer – Espressif Systems – saw the opportunity and jumped right in with the release of official SDKs and English-language documentation.
The ESP8266 was succeeded in 2016 by the ESP32 series, upgrading it to a 32-bit dual-core processor, more memory, Bluetooth support, and a stack more peripheral interfaces.
Both chipsets now have an excellent ecosystem of software around them. Of particular interest to me was the ESPHome project: it’s specifically targeted at generating firmware for ESP-based chipsets, to integrate with a wide range of sensors, and then link all of this back to Home Assistant, which is what we’re already using as the integration layer for everything on the property.
Now I could re-focus the search for boards to be based around these chipsets.
Two compelling solutions came to light:
ESP-01S
These modules are positively tiny, at just 25mm × 15mm.
They’re an ESP8266 chipset, with just 8 pins exposed: power, serial comms (mainly useful for programming them), and two GPIO ports. That’s not very much exposed, but it’s just enough to get the job done when you only need to detect one thing and switch another.
ESP-01S module
At only ~$1.50 each, they’re at an incredibly compelling price point. That’s a custom programmable microcontroller, with Wi-Fi connectivity, for under $2. 😲
One minor annoyance is that they need to be powered by 3.3V, so you can’t just attach 5V from an old USB charger. There are however two very prevalent base boards: a relay board, and a temperature/humidity board. Each of these supports the ESP-01S just plugging into them, and can be powered by 5-12V. You can pickup the ESP+relay, or ESP+temperature combo for ~$3.
One frill they’re missing in their no-frills approach is any kind of USB serial driver. That just means you’ll need a separate programmer module for the first-time flash. Once you’ve flashed them once, you should be able to do future updates over-the-air.
These modules are the absolute winner: they hit the perfect sweet spot of capability, ease-of-use, and cost. They’re more like a ready-to-go compute module than a dangling microprocessor in need of a breadboard.
They have an ESP32 processor at their core, nine exposed GPIO ports, a button, an LED, and an IR LED. They’re housed in a nice little package, so you don’t have exposed boards. They’re powered and programmed by USB-C. All for ~$5.
M5Stack Atom Lite
The pin configuration is documented right there on the case, making them incredibly easy to wire up quickly. The only piece that’s missing is to know which GPIOs support analog inputs, but you can cross-reference them back to the ESP32 pinout for all the per-pin specs.
Many sensors can be connected directly to the Atom Lite: plug in +3.3V or +5V, GND, and a signal wire to a GPIO pin. For quick construction, wires can be pushed directly into the exposed header sockets. For a more robust connection, you can add a Grove/PH2.0-4P connector to your sensor, and then plug it into the port at the bottom there, next to the USB-C.
The LED is actually a “Neopixel” which means that while it only uses up a single GPIO, it’s a digitally addressable tri-colour LED. We’ve used this to provide a multi-colour indicator right there on the device for quick diagnostics in-the-field.
✅ Sense ✅ Affect
🤩 Plug-and-Play ✅ Secure
✅ Compute ✅ Connect
✅ Power ✅ Cost
Pre-built solutions: Shelly1, Sonoff, and more
The prevalence of ESP chipsets continues to pre-built solutions as well.
The Shelly1 is a 230v/16A-capable relay, designed to fit in to tiny spaces like behind an existing light switch panel. It can be wired in with an existing physical switch, so you still have the instant experience of a normal light switch, but now it can be Wi-Fi monitored and controlled as well. Here it is with an Internationally Standard Sized Oreo for scale:
They’re Australian certified, but you’ll still need a sparky to actually install them for you if they’re hooked up to your 240v circuits. For lower voltage circuits – like switching irrigation – you can wire them up yourself with a screwdriver and zero soldering.
Here, an M5Stack Atom Lite ($6) is combined with a DS18B20 temperature probe (~$2, including 2m cable), a single resistor jammed across two pins, a leftover USB charge cable, and a little bit of electrical tape to hold it all together. For sub-$10, and no soldering, we have a Wi-Fi connected temperature sensor dipped into the hot water system, connected back to Home Assistant.
A similar setup connects to a flow meter in-line with a pressure pump. The flow meter draws power (3.3V and ground) straight off the M5Stack. For each litre of water that passes through, it’ll pulse 11 times. A resistor is jammed across the pulse return and ground to remove float noise. The flow sensor can cost anywhere from $5 to $500 depending on what pressure you want to be able to handle, for what type of fluid, and to what accuracy. Ours cost ~$20, so the whole setup was <$30. It’s not the nicest fanciest of engineering, but it was zero soldering, and it hasn’t missed a pulse yet.
ESP-01S modules with corresponding relays litter Dad’s desk by the bundle. Right now, he’s building out a relay control, with active feedback, for an otherwise dumb valve. It’s a complex valve that takes 15 seconds to move fully from one state to another, due to the pressure it can handle. This is a perfect scenario for local logic: we can use an ESP-01S to drive the valve for the right amount of time, and then validate the expected state against the feedback wire. A dumb valve becomes a smart valve for <$10.
