Our goal is to be able to quickly and cheaply integrate new sensors and controls. We want to be able to buy a flow meter / particulate matter sensor / pressure transducer / other crazy thing off AliExpress, and get it integrated into our control plane without it becoming a massive project.
Too many electronics and automation projects just die on the desk. We want the flexibility of our own electronics, but we want to be off the breadboard and into production as fast as we can.
As a software developer, the place I often getting tripped up is, amusingly, at the coding stage. Some fairly simple electronics suddenly need a lot of code to actually integrate well. Writing it yourself is a blackhole that’s easy to fall into.
Last year, I fell in love with ESPHome: the perfect piece of glue between ESP-based devices and Home Assistant.
I live in an inner-city apartment. Thereâs a concrete slab, brick walls, and no ceiling cavity access. Oh, and we rent, so drilling into things is frowned upon. The home automation potential in a scenario like this consists of some coloured lights, and watering about four plants on a timer. Itâs not exactly an inspiring IoT site. Thereâs also not much wrong with the humble light switch that works instantly, every time.
In complete contrast to this, my parents live on 100 acres / 40 hectares of land, a few hours out of the city. It’s quintessential rural Australia. Theyâre mostly off the grid: thereâs a skerrick of Telstra 4G signal, and some power, except when there isnât, which is surprisingly often. This is an absolute IoT playground, and weâve given that a fair crack over the seven years as theyâve established themselves there.
Opportunity
On a property like this, water is critical. There are the basic living requirements, the opportunity of gardens, and the very real risk of bushfires. All up, we have ~300,000 litres of water tanks on the property, and then a small dam and two creeks. We use sensors to track tank volumes. We measure flow rates at key plumbing points (for both instantaneous oversight and tracking cumulative usage). We use power meters to detect when pressure pumps are running.
Energy management is also important. Whilst there is the option of grid connectivity, we try to run as off-grid as possible, or at least export-only. This requires some smarts around load management. For example, instead of just having a hot water system that aggressively chases a specific temperature, we want the hot water systems to head for a temperature range, but only use excess solar production to do it. If thereâs a bit less sun for the day, itâs ok if the water it a few degrees cooler: donât burn down the batteries or import power just to hit a specific temperature.
And then thereâs a safety aspect. The property is on the top of an escarpment where storms can roll in fast from a few different directions. By running things like lightning sensors on the property, we can trigger our own localised alerts for approaching storms.
Challenges
The challenge is to convert all these possibilities into something real, that works, and doesnât cost an absolute mint. Over the years, we found this incredibly hard to solve for. Youâll find a solution for measuring tank volume, but it comes with its own LoRA gateway, a cloud dependency, and a new app. Youâll find a cheap Z-Wave temperature sensor, but itâs only really good for a room, and doesnât have a probe that you can put into the measurement point in a hot water system. Youâll find a flow meter, but itâs only an industrial solution that wants to talk RS485 serial. Who even does that anymoreâ Youâll find a garage door opener that works for the extra-high roller doors on the shed, but it has its own 433MHz RF remote.
Itâs easy to end up with a horrible mishmash of radio technologies, software platforms, and APIs, not to mention some scary pricing as you drift from the traditional world of home automation into the more dated world of industrial automation.
Goal
We want to be able to dream up crazy new ideas, pick and choose the right sensors, and then integrate them with relative ease and consistency. That means a balance between the ability to build new things ourselves, but not having to go and custom fabricate a circuit board and write new firmware just to add a new sensor.
Considerations
Sense
Most sensors give out some kind of analogue signal (like a pressure sensor where the output voltage varies from 0V to 5V depending on applied pressure), a pulse (like a flow meter that pulses once for every 500mL of water that flows through it), or a digital signal (like a temperate and humidity sensor with a 1-Wire output).
To handle all of these scenarios, weâll need some GPIO pins (the most basic of digital connections), and something with an analogue-to-digital-converter onboard (so that we can measure that variable pressure scenario).
Affect
Most of the outputs that weâre dealing with are straight up binary: turn a pump on, open a valve, flash a light. This means that they can be again driven by some GPIO pins, paired with an appropriately sized relay.
For more complex outputs, like sound, we can defer back to more consumer-grade hardware, like just broadcasting an announcement on the Google Home speaker in the kitchen.
Plug and Play
As much as weâre building things ourselves, we donât need to put too many barriers in front of ourselves. Weâre after a module that we can connect sensors to easily, without too much soldering, and certainly not having to build up our own circuit boards / Veroboard. We want to be out of the lab and into the environment as fast as possible.
Secure
Thereâs a persistent joke about how the âSâ in IoT stands for security.
Security was absolutely a consideration for us though: both on-property, and when it comes to remote access. When youâre switching things like power, or controlling a precious resource like water, you want to be confident that youâre the only person in control.
Our preference has been to keep connectivity and control local to the property, via authenticated connections, and then apply a separate remote access approach further up the stack. This means the hardware needs to have enough power and smarts to handle secured local connections, and not be dependent on its own path to the internet.
Compute
Some sensors require both precise timing and persistence to count things like pulses and turn them into natural measures. For example, a flow meter might give you a signal pulse for very 500mL of water that flows through it. If you miss a pulse and stop counting for a bit, youâre missing water. Our preference has been to count pulses on the hardware attached to the sensor, and then report back the natural values (L/min, L/day) whenever the network and central compute is available. We want to keep sensor-specific concepts, like pulses, at the sensor, and just send meaningful information over the network.
As much as we want things to be connected, they can still be somewhat smart in their own right. If a hardware module has both a temperature sensor and a relay to turn a heating element on and off, itâs perfectly capable of being a thermostat on its own, regardless of whatâs happening to the wider network or any centralised compute. It will be smarter when the central controls are online, because it can be aware of other datapoints like the solar charge status, but it doesnât have to be totally bound to the availability of those other systems to operate its basic function. Achieving this balance requires us to be able to run basic logic directly on the module.
The world is not static. If we want to run logic on these devices, and keep them secure, they need to be able to receive firmware updates over-the-air. We donât want to be clambering around under sheds with laptops to reprogram a thermostat.
Connect
Network standards are still the multi-billion-dollar question in IoT.
Early on, we deployed a number of Z-Wave and Zigbee based devices. These are two different mesh protocols, at the mid-point of their VHS vs. Betamax battle for dominance. Theyâre common in consumer automation solutions like smart switches, and good for extremely low power environments, like where you want to be able to throw a temperature sensor in the corner of a room with a coin-cell battery in it and then forget about it for a year. The sensor ecosystem is very consumer focussed (youâll find a million temperate sensors, but no tank pressure sensors). The communication protocol is constrained: by design, itâs a very low bandwidth data network, operating up to 40kbps for Z-Wave, or a whopping 250kbps for Zigbee. Range is limited to keep within the power limits, so typically as low as ~15-20m. Thereâs no common way of building on-device logic, and if you do manage to, then itâs incredibly hard to apply firmware updates over-the-air for either of them.
Our exploration continued into LoRA and NB-IoT, but weâve opted away from both for this property. Theyâd each be very compelling options if we wanted to cover the larger property, such as if it was more of a working farm with distributed infrastructure than predominantly bushland and gardens with clustered buildings.
Ultimately, we settled on Wi-Fi as our preferred connectivity on the property. Weâre already got great coverage throughout the house, cottage, shed, and key outdoor locations via a UniFi deployment. Whilst this is heavily centred on the built infrastructure, and not the full 100 acres of property, thatâs where most of the sense-and-act needs to occur anyway. Investing in the Wi-Fi coverage provides other benefits, like Wi-Fi calling for mobiles where weâre otherwise on the fringe of 4G coverage. The UniFi infrastructure is readily extensible, as Lars has proven on his property with the deployment of Llama Cam, and even extending the coverage to places without power. Finally, it gives us a pretty amazing management plane thatâs a lot nicer to work with than trying to diagnose a Z-Wave/Zigbee mesh.
UniFi infrastructure topology
Power
Weâre happy to depend on a power source for most devices: theyâre usually located near other loads that are already powered, or itâs easy enough to get power to them, such as dropping in a 12V cable out to a water tank at the same time as trenching the plumbing in. If we really want to run on battery, itâll be ok to have a larger 12v battery and a small solar panel or something: we donât need to try and run off a coin-cell battery for years on end.
Cost
The approach needs to be reasonably affordable if itâs going to grow over time: tens of dollars to add a new thing, not hundreds.
Whilst thereâs no room to optimise on sensor cost, it does give something to calibrate against for the target cost of the compute module. We wanted to target ~$5 per module. It felt like the right cost, relative to the sensors themselves.
Hardware Options
So many options! Let’s run through them against those considerations:
Raspberry Pi
By this point, most software developers are jumping in with âRaspberry Pi! You can even buy hats for them!â. Thatâs certainly a response I started from. It feels safe: it has ports that youâre used to, you can plug in screens, it runs a whole operating system, and it has an IO header exposed for those plug-and-play scenarios.
Raspberry Pi 4
Theyâre also completely overpowered for what we need here. These things are small PC replacements: theyâre full computers, not basic microcontrollers. Memory capacity starts in the gigabytes when we only need megabytes. They can run dual 4K display outputs when we only need to switch a couple of wires on and off. They need an SD card with a whole operating system on it just to boot. They suck up 3A of power, which is at the expensive end of USB power supplies. Theyâre also quite bulky compared to other options weâll look at through this post.
A Pi, power supply, and SD card will quickly run to ~$50, which is 10x our cost target.
â Sense â Affect
â Plug-and-Play â Secure
𤯠Compute â Connect
đ Power â Cost
Raspberry Pi Zero W
This is as small as the Raspberry range goes, but itâs still a similar position as the main Raspberry Pi models: completely overpowered for this purpose, with a 1GHz processor, 512MB of RAM, and HDMI video output. Thankfully, itâs less power hungry, so weâre back into the range of any-USB-power-will-do. Itâs also physically a lot smaller: look at the relative size of those micro USB ports.
Raspberry Pi Zero W
It still needs a micro-SD card in addition to the base board, which takes the bill up to ~$15, still 3x our cost target. From a plug-and-play perspective, youâll have to first spend time soldering on your own header strip, or pay a bit extra for one pre-soldered.
â Sense â Affect
â Plug-and-Play â Secure
𤯠Compute â Connect
â Power â Cost
Arduino and Adafruit
As we carve away ports and computer power that we donât need, our next stop is the Arduino or Adafruit Feather range of boards.
Each of these ecosystems has a range of different connectivity options: boards that come with Wi-Fi, Bluetooth, LTE, NB-IoT, LoRA, or none of the above.
They have Wi-Fi on board, a USB port for power and programming, a stack of GPIO pins, and an analogue input. Thereâs no additional SD card required: thereâs flash memory on the board itself. You can buy a version with pre-soldered headers, making that plug-and-play scenario easier.
The compute is right sized for basic sensors: an 80MHz processor with 4MB of flash memory attached.
The only thing on here thatâs a little bit extraneous for our scenario is the Li-Po battery connection. (Thatâs the black connector at the top corner, next to the micro-USB.) The board can both run off a Li-Po battery, and recharge one, as thereâs a charging circuit built into it as well. But, for our scenario where we said permanent power was ok, the charging circuit just adds more cost to the board.
Unfortunately, the boards are still up around $20, which is 4x our cost target. Thereâs also a lot of exposed electronics, so weâd need to factor a case into the price yet.
â Sense â Affect
â Plug-and-Play â Secure
â Compute â Connect
â Power â Cost
Discovering the ESP
What weâve looked at so far in this post are different boards.
As I was researching around different boards, I kept running into phrases like âArduino compatibleâ, and references to ATmega, ESP8266, or ESP32 chipsets.
It starts to get interesting when you split up the concept of a board, versus a chipset.
The chipset is the processor at the core of each of these boards: the smarts that makes it tick. There are a small number of very popular chipsets. There are then lots of different boards that package these chipsets up with other supporting systems and accessories to make them more accessible: USB serial drivers, voltage regulators, battery charge circuits, different breakout headers, and different physical form factors. Itâs these extra systems that drive the cost up, and the brand recognition of the boards that drives the margin.
After some very nerdy reading đ¤, I got quite excited by the ESP range, specifically the ESP8266 and ESP32 chipsets. It turns out these are very popular with makers and manufacturers alike because they hit an interesting sweet spot of cost and capability. If youâve got a Wi-Fi enabled smart plug in your house, thereâs a decent chance that it has an ESP inside it.
These chips were never really designed as a general-purpose compute unit: the ESP8266 came first, and it was intended as a Wi-Fi modem, to be added on to some other device. It has enough compute to run a full TCP/IP stack, some leftover generic pins, and the ability to re-flash it with new firmware. There was originally only Chinese-language documentation, but the low price point and interesting feature set led the maker community off into investigating it further and translating the documentation. The manufacturer â Espressif Systems â saw the opportunity and jumped right in with the release of official SDKs and English-language documentation.
The ESP8266 was succeeded in 2016 by the ESP32 series, upgrading it to a 32-bit dual-core processor, more memory, Bluetooth support, and a stack more peripheral interfaces.
Both chipsets now have an excellent ecosystem of software around them. Of particular interest to me was the ESPHome project: itâs specifically targeted at generating firmware for ESP-based chipsets, to integrate with a wide range of sensors, and then link all of this back to Home Assistant, which is what weâre already using as the integration layer for everything on the property.
Now I could re-focus the search for boards to be based around these chipsets.
Two compelling solutions came to light:
ESP-01S
These modules are positively tiny, at just 25mm Ă 15mm.
Theyâre an ESP8266 chipset, with just 8 pins exposed: power, serial comms (mainly useful for programming them), and two GPIO ports. Thatâs not very much exposed, but itâs just enough to get the job done when you only need to detect one thing and switch another.
ESP-01S module
At only ~$1.50 each, theyâre at an incredibly compelling price point. Thatâs a custom programmable microcontroller, with Wi-Fi connectivity, for under $2. đ˛
One minor annoyance is that they need to be powered by 3.3V, so you canât just attach 5V from an old USB charger. There are however two very prevalent base boards: a relay board, and a temperature/humidity board. Each of these supports the ESP-01S just plugging into them, and can be powered by 5-12V. You can pickup the ESP+relay, or ESP+temperature combo for ~$3.
One frill theyâre missing in their no-frills approach is any kind of USB serial driver. That just means youâll need a separate programmer module for the first-time flash. Once youâve flashed them once, you should be able to do future updates over-the-air.
These modules are the absolute winner: they hit the perfect sweet spot of capability, ease-of-use, and cost. They’re more like a ready-to-go compute module than a dangling microprocessor in need of a breadboard.
They have an ESP32 processor at their core, nine exposed GPIO ports, a button, an LED, and an IR LED. Theyâre housed in a nice little package, so you donât have exposed boards. Theyâre powered and programmed by USB-C. All for ~$5.
M5Stack Atom Lite
The pin configuration is documented right there on the case, making them incredibly easy to wire up quickly. The only piece that’s missing is to know which GPIOs support analog inputs, but you can cross-reference them back to the ESP32 pinout for all the per-pin specs.
Many sensors can be connected directly to the Atom Lite: plug in +3.3V or +5V, GND, and a signal wire to a GPIO pin. For quick construction, wires can be pushed directly into the exposed header sockets. For a more robust connection, you can add a Grove/PH2.0-4P connector to your sensor, and then plug it into the port at the bottom there, next to the USB-C.
The LED is actually a âNeopixelâ which means that while it only uses up a single GPIO, itâs a digitally addressable tri-colour LED. Weâve used this to provide a multi-colour indicator right there on the device for quick diagnostics in-the-field.
â Sense â Affect
𤊠Plug-and-Play â Secure
â Compute â Connect
â Power â Cost
Pre-built solutions: Shelly1, Sonoff, and more
The prevalence of ESP chipsets continues to pre-built solutions as well.
The Shelly1 is a 230v/16A-capable relay, designed to fit in to tiny spaces like behind an existing light switch panel. It can be wired in with an existing physical switch, so you still have the instant experience of a normal light switch, but now it can be Wi-Fi monitored and controlled as well. Here it is with an Internationally Standard Sized Oreo for scale:
Theyâre Australian certified, but youâll still need a sparky to actually install them for you if theyâre hooked up to your 240v circuits. For lower voltage circuits â like switching irrigation â you can wire them up yourself with a screwdriver and zero soldering.
Here, an M5Stack Atom Lite ($6) is combined with a DS18B20 temperature probe (~$2, including 2m cable), a single resistor jammed across two pins, a leftover USB charge cable, and a little bit of electrical tape to hold it all together. For sub-$10, and no soldering, we have a Wi-Fi connected temperature sensor dipped into the hot water system, connected back to Home Assistant.
A similar setup connects to a flow meter in-line with a pressure pump. The flow meter draws power (3.3V and ground) straight off the M5Stack. For each litre of water that passes through, itâll pulse 11 times. A resistor is jammed across the pulse return and ground to remove float noise. The flow sensor can cost anywhere from $5 to $500 depending on what pressure you want to be able to handle, for what type of fluid, and to what accuracy. Ours cost ~$20, so the whole setup was <$30. Itâs not the nicest fanciest of engineering, but it was zero soldering, and it hasnât missed a pulse yet.
ESP-01S modules with corresponding relays litter Dadâs desk by the bundle. Right now, heâs building out a relay control, with active feedback, for an otherwise dumb valve. Itâs a complex valve that takes 15 seconds to move fully from one state to another, due to the pressure it can handle. This is a perfect scenario for local logic: we can use an ESP-01S to drive the valve for the right amount of time, and then validate the expected state against the feedback wire. A dumb valve becomes a smart valve for <$10.