Thursday, June 18, 2015

MQTT Conversion, First Steps

Last post I talked about trying out mqtt and being impressed with the ease of using it. It really went well and I decided to convert my house monitoring system to using this tool. So I took the very first steps. This is going to be a complicated conversion because I already have a system running and I don't want to break it while I'm moving to something different, so I decided on a few things to get me started.

I'm going to divide the sensors, saving data, and presenting data into different things. For example, I'm starting with the weather station because it's the simplest, but still has multiple sensing devices. The entire house is published under the 'topic' Desert-Home, so the weather station will be Desert-home/Weather, and the two current sensors are the Acurite 5n1 and the barometer out on the fence. The interesting thing here is that they are radically different.

The 5n1 is detailed on about a hundred posts on this blog, but finally wound up being a sensor on the roof that transmits in the 933MHz range. I catch and decode the signal with an SDR (software defined radio) and some code I stole and heavily modified. The readings from this sensor will be published under Desert-Home/Weather/5n1.

The barometer is a device I built up myself and put in a Stevenson Screen out on a fence post in the yard. It has an XBee that sends a JSON string to a Pi. This will be published under Desert-Home/Weather/Barometer.

I can see each of these devices by simply subscribing to 'Desert-Home/Weather/#'. The pound sign means to get everything that comes in under 'Desert-Home/Weather' and that will include every sensor I come up with related to weather.

I also decided to use JSON strings as much as possible for a few reasons. Most important to me is the ease of reading a JSON string when you're monitoring the network to see why something is messing up. The JSON string has key:value pairs that tell you what it is and what the value is. This helps a lot when I'm looking for a problem. This will be painful since most of my sensors use another technique entirely and they will have to be converted. Another reason important to me is that I don't have to write parsers to get the data back into variables, There are libraries that will do that for me and all I have to do is use them. Maybe that will make up for the pain I go through converting the devices.

So, since the XBee network I have is on a different Pi than the weather station RF software, I get to try out a multi-machine solution as the very first thing I try. Step one is done; I brought up a mqtt server (mosquitto, see previous post) and added publish lines to the XBee receive code and the weather station decoding. It worked on the second try; I had a couple of syntax errors to fix.

Mosquitto caught the data and all I have to do to see it is use the tool mosquitto_sub to watch the data as it is generated. The command line:

 mosquitto_sub -h "" -t "Desert-Home/Weather/#" -v

shows me what is coming in as it happens:

Desert-Home/Weather/Barometer {"Barometer":{"temperature":"114.4","pressure":"1011.4","utime":"1434639446"}}
Desert-Home/Weather/5n1 {"sensorId":{"SID":"92","t":"1434664650"},"channel":{"CH":"A","t":"1434664650"},"messageCaught":{"MC":"0","t":"1434664650"},"battLevel":{"BAT":"7","t":"1434664650"},"windSpeed":{"WS":"6.0","t":"1434664650"},"windDirection":{"WD":"W","t":"1434664650"},"temperature":{"T":"114.6","t":"1434664632"},"humidity":{"H":"9","t":"1434664632"},"rainCounter":{"RC":"445","t":"1434664650"}}
Desert-Home/Weather/5n1 {"sensorId":{"SID":"92","t":"1434664650"},"channel":{"CH":"A","t":"1434664650"},"messageCaught":{"MC":"0","t":"1434664650"},"battLevel":{"BAT":"7","t":"1434664650"},"windSpeed":{"WS":"6.0","t":"1434664650"},"windDirection":{"WD":"W","t":"1434664650"},"temperature":{"T":"114.6","t":"1434664632"},"humidity":{"H":"9","t":"1434664632"},"rainCounter":{"RC":"445","t":"1434664650"}}
Desert-Home/Weather/5n1 {"sensorId":{"SID":"92","t":"1434664650"},"channel":{"CH":"A","t":"1434664650"},"messageCaught":{"MC":"0","t":"1434664650"},"battLevel":{"BAT":"7","t":"1434664650"},"windSpeed":{"WS":"6.0","t":"1434664650"},"windDirection":{"WD":"W","t":"1434664650"},"temperature":{"T":"114.6","t":"1434664632"},"humidity":{"H":"9","t":"1434664632"},"rainCounter":{"RC":"445","t":"1434664650"}}

Yes, the outside temperature is REALLY 114.4, remember I live in Arizona and this is the middle of June, but it's a dry heat, the humidity reading is 9%. As if that matters when the temperature is this high ...

But, also notice that I have both strings coming to the subscription even though they are very different devices, on different Pi's. Heck, they could be half way around the world and it wouldn't matter, it would still work. I tested that with a conspirator just this morning.

Now to follow my plans, I have to catch this data in some other process and save it to my database. I'll get to that soon, maybe even today, but I'm going to hold off on changing the presentation to the web for later because I have many different presentations to consider when I get to that point. See, currently I take everything presented from my data bases. If I'm using my Android app to look at the house, the data is coming out of the data base. When I send a command to close the garage doors, the command is executed and the app doesn't update until it is recorded in the data base. I may change that to reflect the latest reading from the mqtt server. We'll see, and then I'll probably change it.

The code is not in GitHub, I'm going to hold off until I get enough in there to make a significant difference. But, the example from the last post is exactly what I did with the difference above. This is a lot simpler and more versatile than I expected.


  1. You may want to consider that there is a single point of failure. That is the single broker. Eventually I would have a second broker run with a failover capability. We use this technique a lot in telecom. Not an immediate need but certainly down the road. Just point the publishers to both.


    1. There's always a single point of failure in a home setup, If the power doesn't die, the ethernet switch dies, or the wifi access point, or the ...

      However, It would be good to have a second broker to take up the slack if I work on a machine. I'll have to experiment with that.

  2. As I’m about to re-enter in the home automation (and as a long time reader of your blog), this mqqt series of posts are VERY usefull.
    With an Almond+, a Microserver and a few Arduinos to play with, mqqt seemed a good possibility to glue everything together…


    1. You're welcome, this is just getting started.

  3. Dave,

    I lurk on this blog a lot and appreciate all the work you do and share with us. I have messed around with MQTT and it is pretty awesome for this type of DIY home project. I just wanted to challenge you to take some time to make sure you have your topic choices well thought out. I say this in a way of maybe you want to have a higher 'parent' topic than just weather. sort of like 'desert-home/sensors/outside/weather/#'. This way you have more granularity.

    I might be making this confusing, but I think there is goodness in this. Then you can check all your sensors at once 'desert-home/sensors/#', or only check just the temperature of from your sensors with 'desert-home/sensors/+/+/temperature'. Same goes with switches. You could turn on/off all light in the whole house 'desert-home/switches/+/lights/#'. Could turn out one lamp in the living room 'desert-home/switches/livingRoom/lights/lamp1' or all the lights in the living room 'desert-home/switches/livingRoom/lights/#'.

    Possible examples:
    desert-home/switches/+/lights/# - switch all lights in every room.
    desert-home/switches/foyer/lock/# - switch all locks on/off
    desert-home/switches/outside/sprinkler/# - on/off

    desert-home/sensors/+/+/temperature - to get all temperatures


    Not sure if any of this made sense, but I really just wanted to get across , with some forward thinking, you can have more generalized topics that move towards getting very detailed. In the end I think that would bring about a more robust system.

    1. Totally understood you. This is one of the reasons why I have a blog, great ideas and pointers come around because people have done this before.

      I'm going to be revisiting the 'topic' thing over and over again as I bring this up, but it's one of those things (at least to me) that I need a little experience to see what I actually want to do. Even then, I screw it up and have to go back and change it.

      Thanks for the suggestion, I WILL take it seriously. And, if you think of something else, don't hesitate, post it. Other folk read this stuff and they'll want to know it also.

  4. You may want to check out mqttwarn ( a python script that connects mqtt with multiple services including many different databases.

    1. That thing is more complex than mqtt itself. Yes, it interfaces to almost everything in the world, but just trying to read the configuration left me spinning.

      Got something simpler?

    2. Nevermind, I think I can write one easier than I can understand that one.