Sensors are Servers

The only servers in my stack are the sensors themselves.

I bought an old house and I want to start fixing things.

• there’s a humidity problem
• there’s a temperature leak or two
• there’s an opporunity to measure and to improve, or overengineer

It helps that I’m a bit of a nerd and this is a great excuse to learn about electronics. Before I knew it I had a raspberry in every room of my house and I started collecting my own data.

Hardware

For this problem, I chose to go with the rapsberry pi stack with sensors from pimoroni. I cannot recommend the pimoroni people enough if you’re interested in working with python. They offer a garden hat which is a solverless solution to attach sensors to a raspberry.

The main sensor that I am using is the BME 680. Take care not to confuse it with the BMP 280; the latter is much cheaper but does not offer a humidity reading. There are many other sensors that play nice on this platform that are worth a mention though:

• a distance sensor, very useful for checking open doors
• a light sensor can check for the sun or indoor lighting
• a thermal camera even though the resolution is a bit too low for me
• even a display that you can stick on the garden

Software

The sensors themselves come with packages that are installable from pypi which is great! The main problem though has to do with updating. I want the raspberries to post data somewhere (BigQuery) and if the endpoint changes I’d prefer that I do not need to manually update via USB.

After thinking about what I wanted the stack to look like I considered that the raspberries themselves act as if they are servers so I could go ahead and use ansible for it. It is an old tool, I remember we’ve used it to provision hadoop clusters 5 years ago, but it has all the features that I needed.

The only manual thing that you need to do is put an identity file on each raspberry and allow ssh traffic on each device. From there you can do a lot of things programmatically if you configure the right files.

SSH Config

# ~/.ssh/config
Host rpi-zero-room1
    HostName 123.456.789.1
    User pi
    IdentityFile ~/.ssh/home-rpi-keyfile
Host rpi-zero-room2
    HostName 123.456.789.2
    User pi
    IdentityFile ~/.ssh/home-rpi-keyfile
..

With this file configured you gain two benefits:

1. You can easily ssh into each raspberry from your laptop
2. You can rely on ansible being able to recognize any hosts in that file

Ansible Files

In my project folder I have the following file structure:

> tree -L 2
.
├── README.md
├── ansible
│   ├── hosts
│   ├── provision-cronjobs-rasp-electro.yml
│   ├── provision-cronjobs-rasp-zero.yml
│   ├── provision-files-rasp3.yml
│   └── provision-tools-rasp3.yml

The hosts file should look like:

[raspberries]
rpi-zero-room1 ansible_user=pi
rpi-zero-room2 ansible_user=pi
rpi-zero-room3 ansible_user=pi
..
rpi-zero-room9 ansible_user=pi

[different-sensors]
rpi-zero-other ansible_user=pi

Note that the names of the servers need to correspond with those in the config file and from here you can split up functionality per server. I have certain raspberries that have completely different sensors and you can split this from the hosts file.

The remaining .yml files contain instructions for the raspberry which are picked up and are run in parallel.

Example: Installation

Here’s an example .yml file that installs some required software.

- hosts: raspberries, different-sensors
  become: yes
  become_user: root
  tasks:
    - name: make sure that we have most recent apt
      command: apt-get update

    - name: install all the apt get stuff, incl fail2ban
      apt: name={{item}} state=installed
      with_items:
        - fail2ban
        - postfix
        - build-essential

    - name: pip install requirements globally
      pip: name={{item}} state=present
      with_items:
        - google-cloud-bigquery
        - ipython

The command might differ depending on the OS. Note that pypi is available via ansible!

Example: Cronjobs

Here’s an example .yml file that sets a cronjob.

- hosts: raspberries
  become: yes
  become_user: root
  tasks: 
    - name: remove old humidity/temperature cronjob
      cron:
        name="humiditemp"
        state=absent
        user=pi
    - name: add new humidity/temperature cronjob
      cron:
        name="humiditemp"
        minute="*"
        user=pi
        job="sudo /usr/bin/python /loggers/cron-scripts/measure.py"

Note that you need to remove any old cronjobs before adding new ones. Also note this won’t run for all hosts.

Updates

With ansible set up, updates are now super easy. Whenever the code is updated, I merely need to run:

ansible -i hosts all -m ping
ansible-playbook -i hosts provision-tools-rasp3.yml
ansible-playbook -i hosts provision-files-rasp3.yml
ansible-playbook -i hosts provision-cronjobs-rasp3.yml

This will update any raspberry connected to wifi.

Sensor in the Garden

I was considering installing a raspberry outside with solar panels and everything.

I’ll still do this someday, but hardware takes more time than software and you cannot copy/paste in real life. So instead of measuring the weather outside of my house, I figured scraping the weather APIs would be a much better idea.

It was. Adding to my stack was super easy. Google now has a cloud-cron service which allows you to have a serverless function called every minute.

Costs

Per day I log about.

\[ 3 \text{ sensors} \times 7 \text{ devices} \times 1440 \text{ readings/day} \approx 30 \text{K rows/day} \]

I’ve been running this for about 3 months.

\[ 30 \text{K rows/day} \times 90 \text{ days} \approx 2.7 \text{M rows}\]

This totals to about 150MB of data, with a dumb schema. That’s 600MB per year.

Price of Storage

\[ $0.020 \text{GB}^{-1} \times 0.6 \text{GB} \times 12 \text{ months} \approx $0.12 \text{ year}^{-1}\]

Sure, this amount will increase, but it takes a while before I start paying more than $1.

Price of Compute

\[ \frac{1,000,000 \text{ MB}}{600 \text{ MB}} \approx 1667 \text{ years before I start paying}\]

There is a free tier on google’s side, which suggests I don’t pay for the first TB each month.

End Product

BigQuery is really easy to access from R thanks to bigrquery. Setting up a dashboard locally took about half an hour and I can make all the charts that I desire from ggplot.

There’s a few things that I found interesting to read from these sensors:

• the humidity will spike in the rooms next to the bathrooms after somebody has had a shower
• the temperature does not fluctuate (in winter) in the room where the thermostat is
• the humidity increases when there’s just been rain, which suggests a leak
• the temperature increases when there are no clouds (this makes sense, I’ve got giant windows pointing south)

It’s fun to measure one’s house like this and it helps to motivate some investments too.

Final Tips

If you’re going to do this yourself; the heat from the raspberry influences the temperature/humidity sensor. So does sunlight!

Either do some hardware work and move the sensors away from the raspberry if you want to get accurate readings or consider physics.

\[ \text{actual temperature} = f(\text{sensor temp}, \text{cpu temp}) \]

Either way, consider that sensors are always biased. I haven’t fully figured out a good way around it just yet. The hardware can also get expensive if you want to have one such sensor per room in your house.

I also think it is worth mentioning that most investments I’ve made to the house don’t really require the measurements. Adding a curtain in front of a leaky door makes for less heatloss and you do not need automated measuring tools in order to do this right away. Still, electronics can be a fun hobby.

Acknowledgments

The real heros of this story are raspberry/pimoroni when it comes to hardware and python/linux when it comes to software. I couldn’t imagine just doing this 10 years ago and we’ve come a long way.