Last post I talked about how I got Grafana working on my desktop talking to my database. It actually worked pretty well. Unfortunately, the laptop is NOT where I want it to run; I want it to be running on one of my Raspberry Pi machines. Since I have one of the Pi 3b's and, it isn't doing much of anything, I put it there.
Like almost everything related to linux, this was a pain. It seems that since I last played with that little machine, there have been two major updates to the OS. They went from Jessie to Pixel to Stretch. WTF?? Are the folks at Raspberry Pi running out of things to do?
Fine, I downloaded the minimal image of Stretch and wrote it onto an SD card, made a couple of changes to the boot init files to run headless and booted it up. That worked first try. They really did make some improvements in that area. No gritting your teeth and closing your eyes before rebooting; it just worked. Then I went looking for a Raspberry Pi version of Grafana ... Guess what? Grafana doesn't make one ! Fortunately, there's a person out there that took this job on for the rest of us. The github user 'fg2it' has created a github repository that holds Grafana for the various versions of Pi OS's and it's relatively easy to use them --- after you figure out what the heck he is doing. The repository he created is here <link>, and in the instructions he does a cool trick of updating where Grafana is installed from, then used apt-get to install it. I stumbled over that for a while before I got it working. Really, really clever way to do it.
Aside: I ran into instruction after instruction for installing Grafana on the Pi, and almost all of them referenced back to fg2it's repository but, tried to take the credit for all the work. The slimeballs only referenced fg2it in the various commands. You'll see what I mean when you go looking around. Thank you fg2it.
I installed the 'Stretch' version of Grafana on the Pi I had just updated and made the simple changes necessary and it worked. I could log into Grafana and create charts that I copied from the laptop version I had used previously. Everything worked pretty well. Next, I added some more charts and explored a bit. Here's the way it looks right this minute when I load the graphs up on my laptop:
This is produced by Grafana running on a Raspberry Pi displaying on a Windows laptop and getting stored data from a MySQL database running on a QNAP network appliance. Cool !
Yes, since it's only a web page, it works on a phone as well:
You do have to scroll down on the screen to see all four of the charts, but that's the way it should work on a small device.
There's a ton of features I haven't played with yet, and I'll get to some of those over the next few days, but first I want to protect my database and figure out some way of offering this up on the web. It may get complicated since this machine can't be seen from out there.
Also, this machine only has an SD card on it. I had to leave the usb solid state drive out of the installation to get this much running. That will have to change since I've never had good luck with the SD cards. Three - four months and they crap out leaving you with a days work getting it running again.
Note that I have no dependency on any cloud service at all. It's my database, and all the software runs right here in the house.
Take that Xively !
I live in the Arizona Desert, Southwestern USA. It gets hot here, and my power bills got out of hand. This is a journal of my various efforts to bring this problem under control using the cheapest technology I could find. Saving money shouldn't cost a fortune.
Showing posts with label Raspberry Pi. Show all posts
Showing posts with label Raspberry Pi. Show all posts
Tuesday, February 6, 2018
Wednesday, March 9, 2016
My Weather Station is Back Online
Yep, I got it working. About a week ago my weather station Raspberry Pi died, and predictably, I couldn't get it back up without major work <link>. It's all better now running on a Raspberry Pi model B. Yes, I know that isn't the latest of the Pi series, but I had it already, OK.
The Raspbian Jessie booted to a graphic user interface directly, not to a command line. This was a bit of a surprise because I usually bring up the Pi's I have headless so I don't have to hunt down enough equipment to bring them up on my TV. But, since I had the stuff handy from trying to get it working before I gave up and just reloaded, it came up and I configured it, then stopped the GUI. Loading the GUI makes the boot sequence much longer and I usually just use putty on a laptop to control these machines, I don't need it.
Contrary to many posts out there, the configuration screens worked just fine. I was able to assign a static IP address and finish the configuration easily. There was a little hunting through the various menus and icons to find things, but that's part and parcel of any first time installation.
But installing my stuff turned out to be somewhat harder than I expected because, as usual, every piece of supporting software had changed and I had to hunt down what I needed. I used the latest (as of the date on this post) version of libusb, it was 1.0.20, which was several versions later than the one that I would get from a normal apt-get on the Pi.
This version worked well and had a few 'gotchas's' in it that I had to stumble through to get it to work. There was enough documentation and forum postings out there that I found to help me through the process.
I also used the latest rtl-sdr. It was pretty easy as well, but also had a couple of things that I had to chase down. Then I loaded the version of rtl_433 that I had set aside so I wouldn't have to keep making my changes in that code. With the poking around I did in libusb and rtl-sdr, this piece was super simple.
I hooked them all together by starting things on the command line and it worked well. I was able to read the weatherhead RF signal and save it off to my database. The interface I have to move the data around for viewing worked just as it had before. The only thing left was to implement start up and restart using systemd instead of upstart.
For those of you that wonder why I needed all this, let me describe what's going on a bit. The usb implementation on the Pi is kernal based and not easily modified by the user. That means I had to get libusb and install it. libusb is a user space interface to the usb capabilities on Linux. Then, I had to be able to read one of those sdr radios so I could find the RF signal from the AcuRite weatherhead. I posted about this earlier <link>. Now that I had the signal, decoding it requires rtl_433 which is a piece of software that was developed to decode the signals of many of the wireless devices available such as thermostats, temperature sensors, door bells, weatherheads, etc.
So, it's software piece piled on software piece piled on ... Sure, I could have incorporated all the things I needed into one piece of software and just ran it, but that's not what leveraging software is all about. You use the piece that does the job you need and add special tweaks specific to your implementation. That saves time and doesn't cost much if anything.
With it working and the only piece left being restarting the processes, I dived into systemd. First though, a tiny bit of background. Unix (and linux came from unix regardless of what people say) has to have something to get it going. This used to be the init process. Init would load the various pieces of software and handle how they interacted. Over time init was replaced with various other things that had more features or less bugs. As is rampant in unix and its derivatives, everyone hated something about each of them. Raspberry Pi uses an operating system called 'Raspbian' which is Debian modified to work specifically on the Pi. There are a ton of other operating systems and the choice of which one to actually use is totally up to the person that installs it. I chose to use Raspian because it was the one the folk that produced the Pi chose to supply. I figured it would have the most support and documentation of various forms over time.
The Raspian that I installed with my first Pi used Sys-V init. This system five init is really old and complex to use. I hate complex, so I replaced it with the clearly superior init process call 'upstart'. Upstart is cool and easy to use once you install it. However, the Debian folk decided that the init process 'systemd' was better and moved to it. Raspberry folk went with the trend and moved to it as well.
That was the decision point that caused me to have to rebuild the weather station Pi from scratch. Upstart wouldn't allow the Pi to boot up at all with the new Jessie level OS that you will get when you update your Pi. But, I did all that and now I faced the dreaded systemd configuration. I looked at about a hundred web sites that discussed how to use systemd to handle starting and restarting a process, and quite honestly, was about to give up because everything was too complex and peppered with 'try this' to give me enough confidence to dive in.
At this point I want to mention that following Linux questions on a forum is so predictable. Someone has a problem and asks for help with it. The first response is to ask for more information about how the system is set up, what they saw that wasn't mentioned in the question, and usually some test or other that can't be run because the problem won't let them. Then a response asking them what they are doing. Then a response that says they should be doing it another way because that way is so much better. Then several posts about how it should be done regardless of the original intent. Lastly, the thread ends with no resolution because the discussion has wandered off into never, never land.
I saw that over and over. I finally stumbled across a blog posting (love blog postings) that whittled all the crap down to something amazingly simple compared to what I had been clicking through for hours <link>. Neil cut all the crap out and just showed how to handle a simple case. I didn't have to dredge through long discussions of 'units' or 'forking' or relative merits of daemons; I could just take his example and use it. It worked first try for one of the processes and third try for the other.
Then I configured logrotate to handle the log files I use for debugging and I was done. Finally.
I still have to back up the SD card and store the image on my NAS, and I'll probably back up the other Pi's just to be somewhat safer (obviously you can't actually be 'safe'). I'll also update my github repo for the various things I changed along the way, but that won't assure me of no problems the next time, just make it somewhat easier to recover.
What will really make it easier the next time is what I learned this time.
The Raspbian Jessie booted to a graphic user interface directly, not to a command line. This was a bit of a surprise because I usually bring up the Pi's I have headless so I don't have to hunt down enough equipment to bring them up on my TV. But, since I had the stuff handy from trying to get it working before I gave up and just reloaded, it came up and I configured it, then stopped the GUI. Loading the GUI makes the boot sequence much longer and I usually just use putty on a laptop to control these machines, I don't need it.
Contrary to many posts out there, the configuration screens worked just fine. I was able to assign a static IP address and finish the configuration easily. There was a little hunting through the various menus and icons to find things, but that's part and parcel of any first time installation.
But installing my stuff turned out to be somewhat harder than I expected because, as usual, every piece of supporting software had changed and I had to hunt down what I needed. I used the latest (as of the date on this post) version of libusb, it was 1.0.20, which was several versions later than the one that I would get from a normal apt-get on the Pi.
This version worked well and had a few 'gotchas's' in it that I had to stumble through to get it to work. There was enough documentation and forum postings out there that I found to help me through the process.
I also used the latest rtl-sdr. It was pretty easy as well, but also had a couple of things that I had to chase down. Then I loaded the version of rtl_433 that I had set aside so I wouldn't have to keep making my changes in that code. With the poking around I did in libusb and rtl-sdr, this piece was super simple.
I hooked them all together by starting things on the command line and it worked well. I was able to read the weatherhead RF signal and save it off to my database. The interface I have to move the data around for viewing worked just as it had before. The only thing left was to implement start up and restart using systemd instead of upstart.
For those of you that wonder why I needed all this, let me describe what's going on a bit. The usb implementation on the Pi is kernal based and not easily modified by the user. That means I had to get libusb and install it. libusb is a user space interface to the usb capabilities on Linux. Then, I had to be able to read one of those sdr radios so I could find the RF signal from the AcuRite weatherhead. I posted about this earlier <link>. Now that I had the signal, decoding it requires rtl_433 which is a piece of software that was developed to decode the signals of many of the wireless devices available such as thermostats, temperature sensors, door bells, weatherheads, etc.
So, it's software piece piled on software piece piled on ... Sure, I could have incorporated all the things I needed into one piece of software and just ran it, but that's not what leveraging software is all about. You use the piece that does the job you need and add special tweaks specific to your implementation. That saves time and doesn't cost much if anything.
With it working and the only piece left being restarting the processes, I dived into systemd. First though, a tiny bit of background. Unix (and linux came from unix regardless of what people say) has to have something to get it going. This used to be the init process. Init would load the various pieces of software and handle how they interacted. Over time init was replaced with various other things that had more features or less bugs. As is rampant in unix and its derivatives, everyone hated something about each of them. Raspberry Pi uses an operating system called 'Raspbian' which is Debian modified to work specifically on the Pi. There are a ton of other operating systems and the choice of which one to actually use is totally up to the person that installs it. I chose to use Raspian because it was the one the folk that produced the Pi chose to supply. I figured it would have the most support and documentation of various forms over time.
The Raspian that I installed with my first Pi used Sys-V init. This system five init is really old and complex to use. I hate complex, so I replaced it with the clearly superior init process call 'upstart'. Upstart is cool and easy to use once you install it. However, the Debian folk decided that the init process 'systemd' was better and moved to it. Raspberry folk went with the trend and moved to it as well.
That was the decision point that caused me to have to rebuild the weather station Pi from scratch. Upstart wouldn't allow the Pi to boot up at all with the new Jessie level OS that you will get when you update your Pi. But, I did all that and now I faced the dreaded systemd configuration. I looked at about a hundred web sites that discussed how to use systemd to handle starting and restarting a process, and quite honestly, was about to give up because everything was too complex and peppered with 'try this' to give me enough confidence to dive in.
At this point I want to mention that following Linux questions on a forum is so predictable. Someone has a problem and asks for help with it. The first response is to ask for more information about how the system is set up, what they saw that wasn't mentioned in the question, and usually some test or other that can't be run because the problem won't let them. Then a response asking them what they are doing. Then a response that says they should be doing it another way because that way is so much better. Then several posts about how it should be done regardless of the original intent. Lastly, the thread ends with no resolution because the discussion has wandered off into never, never land.
I saw that over and over. I finally stumbled across a blog posting (love blog postings) that whittled all the crap down to something amazingly simple compared to what I had been clicking through for hours <link>. Neil cut all the crap out and just showed how to handle a simple case. I didn't have to dredge through long discussions of 'units' or 'forking' or relative merits of daemons; I could just take his example and use it. It worked first try for one of the processes and third try for the other.
Then I configured logrotate to handle the log files I use for debugging and I was done. Finally.
I still have to back up the SD card and store the image on my NAS, and I'll probably back up the other Pi's just to be somewhat safer (obviously you can't actually be 'safe'). I'll also update my github repo for the various things I changed along the way, but that won't assure me of no problems the next time, just make it somewhat easier to recover.
What will really make it easier the next time is what I learned this time.
Thursday, March 3, 2016
How Can I Possibly Be Annoyed at the Raspberry Pi
Yes, they're all over the place and my Google news feed keeps telling me about how wonderful they are, but I almost threw one across the room today. The story is that my weather station Pi lost its SD card; they do that sometimes due to the heavy usage the cards aren't designed for. I've had it happen before, and thought I was prepared this time. You guessed it, I wasn't.
Yes, I have backups of the system, and backups of the software; the problem is that I didn't expect the folk that control the Raspberry Pi software to close some doors on me. You see, after I decided to update and upgrade the OS on the Pi, it wouldn't boot. I like to stay reasonably up to date on my stuff, but often wait a while to get the latest OS and tool changes. With the other non-technical projects I've been working on, it's been a few months since I took the time to update the little machines. During this period Raspberry Pi went from Debian Wheezy to Jessie.
Besides the usual wondering who the heck thinks up these silly names, I thought now would be a good time to update my systems. That was my first mistake; my second was when I didn't read ALL about the changes in Jessie. You see, they decided to use 'systemd' as their default init process and I switched to upstart quite a while ago because it was better. There's a few hundred pages out there on what's best and why, but I don't really care about that; I just want something that works, and I can keep updated.
Fine, I'd switch to the default for the Debian Jessie, but you can't update over a Debian Wheezy that has upstart on it; that makes the machine unbootable. To add insult to injury, this all happened when I was trying to update Wheezy. They don't update Wheezy, they switched me over to Jessie and just left me hanging.
Literally. Yep, hanging on a console message with the machine unable to boot.
I know how to get out of this mess and catch my machines up to the latest, but I really hate having to do it. At least it was my weather station and not the control system for the house. So my plans are to update to Jessie, then load all the packages I use to run this stuff, and finally to update my startup scripts to use the init they want me to use. Probably about three or four days of fooling around for the weather station, then I get to do it all over again for the house controller.
I want to be up to date with the latest because, eventually, the Pi Zero will be back in stock and I really want to play with it. No, I won't worry about updating something that will become a control part inside something else; I'll just use it like some other chip. I'll have to be careful to isolate any code on one away from the OS and come up with ways of starting processes that don't involve the OS, essentially treating them like little black boxes. At $5, I won't much care if they fail and have to be replaced. Sort of takes some of the fun out of them though.
The very next iteration of my devices will NOT use the unix facilities to start and stop processes or to monitor what they do. I'm going back to basics and working out ways to handle it that don't rely on some programmers data politics.
Subject change. A few folk have expressed disappointment that I haven't been posting about my tractor rebuilding, dog door, trenching, culvert, and other projects for the last few months. When I told them that the projects weren't technical, they said, "So?" I didn't have an answer because I've written about rattlesnakes, dogs, shades on the patio, etc.
Sorry tech weenies, I'm going to go back to my old ways and talk about surviving in the desert with a bunch of technology included. It just may be hydraulics, water flow, pruning techniques and such as well as little tiny computers and hacking into strange devices. Heck, I may well post about the technical aspects of using snake shot to control the varmints that try to kill me, as well as my continuing war with woodpeckers.
But my Pi 3B is already on order.
Yes, I have backups of the system, and backups of the software; the problem is that I didn't expect the folk that control the Raspberry Pi software to close some doors on me. You see, after I decided to update and upgrade the OS on the Pi, it wouldn't boot. I like to stay reasonably up to date on my stuff, but often wait a while to get the latest OS and tool changes. With the other non-technical projects I've been working on, it's been a few months since I took the time to update the little machines. During this period Raspberry Pi went from Debian Wheezy to Jessie.
Besides the usual wondering who the heck thinks up these silly names, I thought now would be a good time to update my systems. That was my first mistake; my second was when I didn't read ALL about the changes in Jessie. You see, they decided to use 'systemd' as their default init process and I switched to upstart quite a while ago because it was better. There's a few hundred pages out there on what's best and why, but I don't really care about that; I just want something that works, and I can keep updated.
Fine, I'd switch to the default for the Debian Jessie, but you can't update over a Debian Wheezy that has upstart on it; that makes the machine unbootable. To add insult to injury, this all happened when I was trying to update Wheezy. They don't update Wheezy, they switched me over to Jessie and just left me hanging.
Literally. Yep, hanging on a console message with the machine unable to boot.
I know how to get out of this mess and catch my machines up to the latest, but I really hate having to do it. At least it was my weather station and not the control system for the house. So my plans are to update to Jessie, then load all the packages I use to run this stuff, and finally to update my startup scripts to use the init they want me to use. Probably about three or four days of fooling around for the weather station, then I get to do it all over again for the house controller.
I want to be up to date with the latest because, eventually, the Pi Zero will be back in stock and I really want to play with it. No, I won't worry about updating something that will become a control part inside something else; I'll just use it like some other chip. I'll have to be careful to isolate any code on one away from the OS and come up with ways of starting processes that don't involve the OS, essentially treating them like little black boxes. At $5, I won't much care if they fail and have to be replaced. Sort of takes some of the fun out of them though.
The very next iteration of my devices will NOT use the unix facilities to start and stop processes or to monitor what they do. I'm going back to basics and working out ways to handle it that don't rely on some programmers data politics.
Subject change. A few folk have expressed disappointment that I haven't been posting about my tractor rebuilding, dog door, trenching, culvert, and other projects for the last few months. When I told them that the projects weren't technical, they said, "So?" I didn't have an answer because I've written about rattlesnakes, dogs, shades on the patio, etc.
Sorry tech weenies, I'm going to go back to my old ways and talk about surviving in the desert with a bunch of technology included. It just may be hydraulics, water flow, pruning techniques and such as well as little tiny computers and hacking into strange devices. Heck, I may well post about the technical aspects of using snake shot to control the varmints that try to kill me, as well as my continuing war with woodpeckers.
But my Pi 3B is already on order.
Tuesday, June 23, 2015
Hacking Into The Iris Door Sensor, Part 4, Resolution
The previous post on this project is here <link>.
Well, my partner in questionable activity in hacking the Iris Contact Switch and Key Fob has gotten his devices to work as well as mine, so it's time to close this project off for a while.
The reason my setup worked well and his didn't was because he didn't have a router on his network of Iris devices and I did. The Iris Smart Switch is a router and I have a handful of them scattered around the house. When he plugged one in and tried the devices, his started working and away he went. I just got word that he is switching his network over to completely local control.
Why is a router necessary? Frankly, I'm not sure at this point and I'll check into it more over time, but I have a couple of door switches working quite well, and the key fob controlling one of my smart switches. The system works nicely.
Between the two of us we managed to decode the various timers and such so you folk can pick up where we left off and implement these little devices in your own home for whatever you want. Nicely made (physically) product that was hampered only by the special code Iris put in them to force you to use their hub. I don't know why manufacturers insist on doing that, especially since there are folk like me that will take it as a challenge.
Here's the latest code with the various items. It will support the Iris smart switch, Key Fob and Door Switch. The code doesn't save status to a data base, or forward it to anything else, it just joins the devices and watches them; you'll need to adapt it to what you want to do.
The piece that is still missing is support for an accelerometer that is inside the key fob. I don't have a clue how to initialize and use it. I don't need it, it would just be nice to understand. Maybe after I get some of the other things I'm working on done I'll come back and take another look.
Have fun.
Well, my partner in questionable activity in hacking the Iris Contact Switch and Key Fob has gotten his devices to work as well as mine, so it's time to close this project off for a while.
The reason my setup worked well and his didn't was because he didn't have a router on his network of Iris devices and I did. The Iris Smart Switch is a router and I have a handful of them scattered around the house. When he plugged one in and tried the devices, his started working and away he went. I just got word that he is switching his network over to completely local control.
Why is a router necessary? Frankly, I'm not sure at this point and I'll check into it more over time, but I have a couple of door switches working quite well, and the key fob controlling one of my smart switches. The system works nicely.
Between the two of us we managed to decode the various timers and such so you folk can pick up where we left off and implement these little devices in your own home for whatever you want. Nicely made (physically) product that was hampered only by the special code Iris put in them to force you to use their hub. I don't know why manufacturers insist on doing that, especially since there are folk like me that will take it as a challenge.
Here's the latest code with the various items. It will support the Iris smart switch, Key Fob and Door Switch. The code doesn't save status to a data base, or forward it to anything else, it just joins the devices and watches them; you'll need to adapt it to what you want to do.
#! /usr/bin/python
'''
Hacking into the iris door sensor
Have fun
'''
from xbee import ZigBee
import datetime
import time
import serial
import sys, traceback
import shlex
import Queue
from struct import *
import binascii
import inspect
# line number for debugging
def getLineNumber():
return inspect.stack()[1][2]
# show data formatted so I can read it
def showData(data):
print "********** Message Contents"
for key, value in data.iteritems():
if key == "id":
print key, value
else:
print key, "".join("%02x " % ord(b) for b in data[key])
print "**********"
def showClusterData(lAddr,sAddr, clusterId, data):
print int(time.time()),
print "".join("%02x" % ord(b) for b in lAddr) + \
" " + \
"".join("%02x" % ord(b) for b in sAddr) + \
" clid "+"%04x" % clusterId + "-" + \
"".join("%02x " % ord(b) for b in data)
# this is a call back function for XBee receive.
# When a message comes in this function will
# get the data.
# I had to use a queue to make sure there was enough time to
# decode the incoming messages. Otherwise, in heavy traffic
# periods, I'd get a new message while I was still working on
# the last one.
def messageReceived(data):
#print "queueing message"
messageQueue.put(data)
def handleMessage(data):
try:
# if data['source_addr_long'] not in \
# ['\x00\x0d\x6f\x00\x04\x51\x07\x82',]:
# return
#print 'gotta packet'
#showData(data)
if (data['id'] == 'rx_explicit'):
#print "RX Explicit"
#showData(data)
clusterId = (ord(data['cluster'][0])*256) + ord(data['cluster'][1])
#print 'Cluster ID:', hex(clusterId),
if (data['profile']=='\x00\x00'): # The General Profile
print 'Cluster ID:', hex(clusterId),
print "profile id:", repr(data['profile'])
if (clusterId == 0x0000):
print ("Network (16-bit) Address Request")
#showData(data)
elif (clusterId == 0x0004):
# Simple Descriptor Request,
print("Simple Descriptor Request")
#showData(data)
elif (clusterId == 0x0005):
# Active Endpoint Request,
print("Active Endpoint Request")
#showData(data)
elif (clusterId == 0x0006):
print "Match Descriptor Request"
'''
the switch looks for clusters under profile
c216, and I respond with only 1 cluster 02
'''
showData(data)
time.sleep(2)
print "Sending match descriptor response"
zb.send('tx_explicit',
dest_addr_long = data['source_addr_long'],
dest_addr = data['source_addr'],
src_endpoint = '\x00',
dest_endpoint = '\x00',
cluster = '\x80\x06',
profile = '\x00\x00',
options = '\x01',
data = data['rf_data'][0:1] + '\x00\x00\x00\x01\x02'
)
# The contact switch is a bit slow, give it
# some time to digest the messages.
time.sleep(2)
zb.send('tx_explicit',
dest_addr_long = data['source_addr_long'],
dest_addr = data['source_addr'],
src_endpoint = '\x02',
dest_endpoint = '\x02',
cluster = '\x00\xf6',
profile = '\xc2\x16',
data = '\x11\x01\xfc'
)
time.sleep(2)
elif (clusterId == 0x0008):
# I couldn't find a definition for this
print("This was probably sent to the wrong profile")
elif (clusterId == 0x13):
# This is the device announce message.
print 'Device Announce Message'
# this will tell me the address of the new thing
# so I'm going to send an active endpoint request
print 'Sending active endpoint request'
epc = '\xaa'+data['source_addr'][1]+data['source_addr'][0]
print "".join("%02x " % ord(b) for b in epc)
zb.send('tx_explicit',
dest_addr_long = data['source_addr_long'],
dest_addr = data['source_addr'],
src_endpoint = '\x00',
dest_endpoint = '\x00',
cluster = '\x00\x05',
profile = '\x00\x00',
options = '\x01',
data = epc
)
elif (clusterId == 0x8000):
print("Network (16-bit) Address Response")
#showData(data)
elif (clusterId == 0x8005):
# this is the Active Endpoint Response This message tells you
# what the device can do, but it isn't constructed correctly to match
# what the switch can do according to the spec. This is another
# message that gets it's response after I receive the Match Descriptor
print 'Active Endpoint Response'
# elif (clusterId == 0x0006):
elif (clusterId == 0x8038):
print("Management Network Update Request");
else:
print ("Unimplemented Cluster ID", hex(clusterId))
print
elif (data['profile']=='\xc2\x16'): # Alertme Specific
if (clusterId == 0xee):
clusterCmd = ord(data['rf_data'][2])
status = ''
if (clusterCmd == 0x80):
if (ord(data['rf_data'][3]) & 0x01):
status = "ON"
else:
status = "OFF"
elif (clusterId == 0xef):
clusterCmd = ord(data['rf_data'][2])
status = data['rf_data'] # cut down on typing
if (clusterCmd == 0x81):
usage = unpack('<H', status[3:5])[0]
elif (clusterCmd == 0x82):
usage = unpack('<L', status[3:7])[0] / 3600
upTime = unpack('<L', status[7:11])[0]
#print ("%s Minute Stats: Usage, %d Watt Hours; Uptime, %d Seconds" %(name, usage/3600, upTime))
elif (clusterId == 0xf0):
showClusterData(data['source_addr_long'],data['source_addr'],clusterId,data['rf_data'])
# If the cluster cmd byte is 'xfb', it's a status
if data['rf_data'][2] == '\xfb':
status = data['rf_data'] # just to make typing easier
if status[3] == '\x1f':
print " Door Sensor",
print str(float(unpack("<h", status[8:10])[0])\
/ 100.0 * 1.8 + 32) + "F",
if ord(status[-1]) & 0x01 == 1:
print "reed switch open",
else:
print "reed switch closed",
if ord(status[-1]) & 0x02 == 0:
print "tamper switch open",
else:
print "tamper switch closed",
elif status[3] == '\x1c':
# Never found anything useful in this
print "Power Switch",
elif status[3] == '\x1d':
print " Key Fob",
print str(float(unpack("<h", status[8:10])[0])\
/ 100.0 * 1.8 + 32) + "F",
unpack('<I',status[4:8])[0]
print 'Counter', unpack('<I',status[4:8])[0],
elif status[3] == '\x1e':
# This indicates a door sensor
# with an invalid temperature reading
# the other items are OK
print " Door Sensor",
print "Temperature invalid",
if ord(status[-1]) & 0x01 == 1:
print "reed switch open",
else:
print "reed switch closed",
if ord(status[-1]) & 0x02 == 0:
print "tamper switch open",
else:
print "tamper switch closed",
#This may be the missing link to this thing
print 'sending missing link',
zb.send('tx_explicit',
dest_addr_long = data['source_addr_long'],
dest_addr = data['source_addr'],
src_endpoint = data['dest_endpoint'],
dest_endpoint = data['source_endpoint'],
cluster = '\x00\xf0',
profile = '\xc2\x16',
data = '\x11\x39\xfd'
)
pass
else:
print " Don't know this device yet",
print ''
else:
print " Unknow cluster command"
print ''
pass
elif (clusterId == 0x00f2):
showClusterData(data['source_addr_long'],data['source_addr'],clusterId,data['rf_data'])
print 'Tamper Switch Changed State to',
status = data['rf_data']
if ord(status[3]) == 0x02:
print "Open",
else:
print "Closed",
print ''
pass
elif (clusterId == 0x00f3):
showClusterData(data['source_addr_long'],data['source_addr'],clusterId,data['rf_data'])
print ' Key Fob Button',
status = data['rf_data']
print ord(status[3]),
if status[2] == '\x01':
print 'Closed',
elif status[2] == '\x00':
print 'Open',
else:
print 'Unknown',
print 'Counter', unpack('<H',status[5:7])[0],
print ''
pass
elif (clusterId == 0xf6):
showClusterData(data['source_addr_long'],data['source_addr'],clusterId,data['rf_data'])
print ''
print "Identify Message"
#extract vendor strings
v = data['rf_data']
vendorstr = " - Vendor:"
start = 21
datalen=len(v)
while(start < datalen):
slen=ord(v[start])
vendorstr = vendorstr + " " + v[start+1:start+1+slen]
start = start+slen+1
print vendorstr
print "Sending init message"
zb.send('tx_explicit',
dest_addr_long = data['source_addr_long'],
dest_addr = data['source_addr'],
src_endpoint = '\x00',
dest_endpoint = '\x02',
cluster = '\x00\xf0',
profile = '\xc2\x16',
data = '\x19\x41\xfa\x00\x01'
)
elif (clusterId == 0x0500): # This is the security cluster
showClusterData(data['source_addr_long'],data['source_addr'],clusterId,data['rf_data'])
showData(data)
# When the switch first connects, it come up in a state that needs
# initialization, this command seems to take care of that.
# So, look at the value of the data and send the command.
if data['rf_data'][3:7] == '\x15\x00\x39\x10':
print "sending initialization"
zb.send('tx_explicit',
dest_addr_long = data['source_addr_long'],
dest_addr = data['source_addr'],
src_endpoint = data['dest_endpoint'],
dest_endpoint = data['source_endpoint'],
cluster = '\x05\x00',
profile = '\xc2\x16',
data = '\x11\x80\x00\x00\x05'
)
# The switch state is in byte [3] and is a bitfield
# bit 0 is the magnetic reed switch state
# bit 3 is the tamper switch state
switchState = ord(data['rf_data'][3])
if switchState & 0x04:
print 'Tamper Switch Closed',
else:
print 'Tamper Switch Open',
if switchState & 0x01:
print 'Reed Switch Opened',
else:
print 'Reed Switch Closed',
print ''
pass
else:
print ("Unimplemented Cluster ID", hex(clusterId))
print
else:
print ("Unimplemented Profile ID")
elif(data['id'] == 'route_record_indicator'):
print("Route Record Indicator")
else:
print("some other type of packet")
print(data)
except:
print "I didn't expect this error:", sys.exc_info()[0]
traceback.print_exc()
def stopXBee():
print("XBee stop handler")
zb.halt()
ser.close()
####################### Actually Starts Here ################################
#------------ XBee Stuff -------------------------
# this is the /dev/serial/by-id device for the USB card that holds the XBee
ZIGBEEPORT = "/dev/serial/by-id/usb-FTDI_FT232R_USB_UART_A901QL3F-if00-port0"
ZIGBEEBAUD_RATE = 57600
# Open serial port for use by the XBee
ser = serial.Serial(ZIGBEEPORT, ZIGBEEBAUD_RATE)
# The XBee addresses I'm dealing with
BROADCAST = '\x00\x00\x00\x00\x00\x00\xff\xff'
UNKNOWN = '\xff\xfe' # This is the 'I don't know' 16 bit address
# create a queue to put the messages into so they can
# be handled in turn without one interrupting the next.
messageQueue = Queue.Queue(0)
# Create XBee library API object, which spawns a new thread
zb = ZigBee(ser, callback=messageReceived)
print "started"
while True:
try:
if messageQueue.qsize() > 0:
#print "getting message"
message = messageQueue.get()
handleMessage(message)
messageQueue.task_done();
sys.stdout.flush() # if you're running non interactive, do this
except KeyboardInterrupt:
print "Keyboard interrupt"
zb.halt()
ser.close()
break
except:
print "Unexpected error:", sys.exc_info()[0]
traceback.print_exc()
break
print ("After the while")
# just in case
zb.halt()
ser.close()
Remember, for this to work properly, you'll need one of the Smart Switches in the network, but the Smart Switch that can measure power usage as well as control it was what got me into looking at these devices in the first place. Remember to look at the previous posts on this project as well, I may have forgotten to mention something.The piece that is still missing is support for an accelerometer that is inside the key fob. I don't have a clue how to initialize and use it. I don't need it, it would just be nice to understand. Maybe after I get some of the other things I'm working on done I'll come back and take another look.
Have fun.
Friday, June 19, 2015
MQTT, Early Lessons Learned, and a Concern
So, I'm stepping through code trying to work out the best way of switching to mqtt for handling data from various devices and commands to control them and it DIED.
Yep, I couldn't publish anything even though everything was connected. Without anything being published, I naturally couldn't update the weather data base, and everything got lost. Losing data is no big deal, I'll get more; it's not having a clue what went wrong that sucks. I went and looked at the logging for mqtt and enabled it to syslog so I could at least see something, and that's when it started working again. Probably just a coincidence, but I'm leaving the broker logging enabled for a while. During this process I found a nice tool for interacting with mqtt from my Windows laptop mqtt-spy, it is a java application and allows me to stuff things into mqtt and monitor any of the 'topics', but all it told me was that nothing was getting published.
I'll just wait for it to happen again and see if I can get more information.
Otherwise, I decided to organize the topics differently. This is the first of what will probably be a large number of decisions like this ... mostly wrong. Instead of having the weather devices under the topic Desert-Home/Weather, they're now under Desert-Home/Device. I did this because I realized that weather is a collection of readings, not a device. So, the AcuRite 5n1 and the barometer are now Desert-Home/Device/5n1 and Desert-Home/Device/Barometer.
Yes, a comment on my last post by Grant got me to thinking.
As I was moving and changing code, I thought of something else; wouldn't it be cool to create a log of all the activity of the various things in one place on one machine so I could prowl through the day to day operation easily? Thus was invented mqttlogger.py. This new process subscribes to two topics: Desert-Home/Log and Desert-Home/Attention and simply writes what it receives to stdout. I redirect stdout to a file in the init configuration file and ta da, a log of whatever activity I want in one place. The topic Desert-Home/Attention is for stuff like the batteries going dead in the weather head or one of my battery operated devices. I'll eventually hook email to the Attention topic so I can be notified more easily.
The possible plan is to create a topic called Weather that has levels for the various readings that matter. There would be Desert-Home/Weather/CurrentTemp, HighestDaily, CurrentBarometer, RainfallToday, etc. Then I could pick and choose from the various values to present when I get that far. Frankly though, that's a lot of work and it may get modified to a JSON string that represents the weather reading right now. We'll see when I get there.
The weather station software has been converted to use mqtt now and is working reasonably well with the caveat that it may stop at any time because of the problem I mentioned up top. I can monitor the various things using mqtt-spy to see each device and the items involved including the log I created.
I'll post the code for the logger soon, I want to get a little time on it before I go and embarrass myself, and I want to give it a couple of days before cramming it into GitHub.
If it wasn't for that nagging possibility of it failing again I'd be really pleased.
Yep, I couldn't publish anything even though everything was connected. Without anything being published, I naturally couldn't update the weather data base, and everything got lost. Losing data is no big deal, I'll get more; it's not having a clue what went wrong that sucks. I went and looked at the logging for mqtt and enabled it to syslog so I could at least see something, and that's when it started working again. Probably just a coincidence, but I'm leaving the broker logging enabled for a while. During this process I found a nice tool for interacting with mqtt from my Windows laptop mqtt-spy, it is a java application and allows me to stuff things into mqtt and monitor any of the 'topics', but all it told me was that nothing was getting published.
I'll just wait for it to happen again and see if I can get more information.
Otherwise, I decided to organize the topics differently. This is the first of what will probably be a large number of decisions like this ... mostly wrong. Instead of having the weather devices under the topic Desert-Home/Weather, they're now under Desert-Home/Device. I did this because I realized that weather is a collection of readings, not a device. So, the AcuRite 5n1 and the barometer are now Desert-Home/Device/5n1 and Desert-Home/Device/Barometer.
Yes, a comment on my last post by Grant got me to thinking.
As I was moving and changing code, I thought of something else; wouldn't it be cool to create a log of all the activity of the various things in one place on one machine so I could prowl through the day to day operation easily? Thus was invented mqttlogger.py. This new process subscribes to two topics: Desert-Home/Log and Desert-Home/Attention and simply writes what it receives to stdout. I redirect stdout to a file in the init configuration file and ta da, a log of whatever activity I want in one place. The topic Desert-Home/Attention is for stuff like the batteries going dead in the weather head or one of my battery operated devices. I'll eventually hook email to the Attention topic so I can be notified more easily.
The possible plan is to create a topic called Weather that has levels for the various readings that matter. There would be Desert-Home/Weather/CurrentTemp, HighestDaily, CurrentBarometer, RainfallToday, etc. Then I could pick and choose from the various values to present when I get that far. Frankly though, that's a lot of work and it may get modified to a JSON string that represents the weather reading right now. We'll see when I get there.
The weather station software has been converted to use mqtt now and is working reasonably well with the caveat that it may stop at any time because of the problem I mentioned up top. I can monitor the various things using mqtt-spy to see each device and the items involved including the log I created.
I'll post the code for the logger soon, I want to get a little time on it before I go and embarrass myself, and I want to give it a couple of days before cramming it into GitHub.
If it wasn't for that nagging possibility of it failing again I'd be really pleased.
Thursday, June 18, 2015
MQTT Conversion, First Steps
Last post I talked about trying out mqtt and being impressed with the ease of using it. It really went well and I decided to convert my house monitoring system to using this tool. So I took the very first steps. This is going to be a complicated conversion because I already have a system running and I don't want to break it while I'm moving to something different, so I decided on a few things to get me started.
I'm going to divide the sensors, saving data, and presenting data into different things. For example, I'm starting with the weather station because it's the simplest, but still has multiple sensing devices. The entire house is published under the 'topic' Desert-Home, so the weather station will be Desert-home/Weather, and the two current sensors are the Acurite 5n1 and the barometer out on the fence. The interesting thing here is that they are radically different.
The 5n1 is detailed on about a hundred posts on this blog, but finally wound up being a sensor on the roof that transmits in the 933MHz range. I catch and decode the signal with an SDR (software defined radio) and some code I stole and heavily modified. The readings from this sensor will be published under Desert-Home/Weather/5n1.
The barometer is a device I built up myself and put in a Stevenson Screen out on a fence post in the yard. It has an XBee that sends a JSON string to a Pi. This will be published under Desert-Home/Weather/Barometer.
I can see each of these devices by simply subscribing to 'Desert-Home/Weather/#'. The pound sign means to get everything that comes in under 'Desert-Home/Weather' and that will include every sensor I come up with related to weather.
I also decided to use JSON strings as much as possible for a few reasons. Most important to me is the ease of reading a JSON string when you're monitoring the network to see why something is messing up. The JSON string has key:value pairs that tell you what it is and what the value is. This helps a lot when I'm looking for a problem. This will be painful since most of my sensors use another technique entirely and they will have to be converted. Another reason important to me is that I don't have to write parsers to get the data back into variables, There are libraries that will do that for me and all I have to do is use them. Maybe that will make up for the pain I go through converting the devices.
So, since the XBee network I have is on a different Pi than the weather station RF software, I get to try out a multi-machine solution as the very first thing I try. Step one is done; I brought up a mqtt server (mosquitto, see previous post) and added publish lines to the XBee receive code and the weather station decoding. It worked on the second try; I had a couple of syntax errors to fix.
Mosquitto caught the data and all I have to do to see it is use the tool mosquitto_sub to watch the data as it is generated. The command line:
mosquitto_sub -h "192.168.0.205" -t "Desert-Home/Weather/#" -v
shows me what is coming in as it happens:
The code is not in GitHub, I'm going to hold off until I get enough in there to make a significant difference. But, the example from the last post is exactly what I did with the difference above. This is a lot simpler and more versatile than I expected.
I'm going to divide the sensors, saving data, and presenting data into different things. For example, I'm starting with the weather station because it's the simplest, but still has multiple sensing devices. The entire house is published under the 'topic' Desert-Home, so the weather station will be Desert-home/Weather, and the two current sensors are the Acurite 5n1 and the barometer out on the fence. The interesting thing here is that they are radically different.
The 5n1 is detailed on about a hundred posts on this blog, but finally wound up being a sensor on the roof that transmits in the 933MHz range. I catch and decode the signal with an SDR (software defined radio) and some code I stole and heavily modified. The readings from this sensor will be published under Desert-Home/Weather/5n1.
The barometer is a device I built up myself and put in a Stevenson Screen out on a fence post in the yard. It has an XBee that sends a JSON string to a Pi. This will be published under Desert-Home/Weather/Barometer.
I can see each of these devices by simply subscribing to 'Desert-Home/Weather/#'. The pound sign means to get everything that comes in under 'Desert-Home/Weather' and that will include every sensor I come up with related to weather.
I also decided to use JSON strings as much as possible for a few reasons. Most important to me is the ease of reading a JSON string when you're monitoring the network to see why something is messing up. The JSON string has key:value pairs that tell you what it is and what the value is. This helps a lot when I'm looking for a problem. This will be painful since most of my sensors use another technique entirely and they will have to be converted. Another reason important to me is that I don't have to write parsers to get the data back into variables, There are libraries that will do that for me and all I have to do is use them. Maybe that will make up for the pain I go through converting the devices.
So, since the XBee network I have is on a different Pi than the weather station RF software, I get to try out a multi-machine solution as the very first thing I try. Step one is done; I brought up a mqtt server (mosquitto, see previous post) and added publish lines to the XBee receive code and the weather station decoding. It worked on the second try; I had a couple of syntax errors to fix.
Mosquitto caught the data and all I have to do to see it is use the tool mosquitto_sub to watch the data as it is generated. The command line:
mosquitto_sub -h "192.168.0.205" -t "Desert-Home/Weather/#" -v
shows me what is coming in as it happens:
Desert-Home/Weather/Barometer {"Barometer":{"temperature":"114.4","pressure":"1011.4","utime":"1434639446"}}
Desert-Home/Weather/5n1 {"sensorId":{"SID":"92","t":"1434664650"},"channel":{"CH":"A","t":"1434664650"},"messageCaught":{"MC":"0","t":"1434664650"},"battLevel":{"BAT":"7","t":"1434664650"},"windSpeed":{"WS":"6.0","t":"1434664650"},"windDirection":{"WD":"W","t":"1434664650"},"temperature":{"T":"114.6","t":"1434664632"},"humidity":{"H":"9","t":"1434664632"},"rainCounter":{"RC":"445","t":"1434664650"}}
Desert-Home/Weather/5n1 {"sensorId":{"SID":"92","t":"1434664650"},"channel":{"CH":"A","t":"1434664650"},"messageCaught":{"MC":"0","t":"1434664650"},"battLevel":{"BAT":"7","t":"1434664650"},"windSpeed":{"WS":"6.0","t":"1434664650"},"windDirection":{"WD":"W","t":"1434664650"},"temperature":{"T":"114.6","t":"1434664632"},"humidity":{"H":"9","t":"1434664632"},"rainCounter":{"RC":"445","t":"1434664650"}}
Desert-Home/Weather/5n1 {"sensorId":{"SID":"92","t":"1434664650"},"channel":{"CH":"A","t":"1434664650"},"messageCaught":{"MC":"0","t":"1434664650"},"battLevel":{"BAT":"7","t":"1434664650"},"windSpeed":{"WS":"6.0","t":"1434664650"},"windDirection":{"WD":"W","t":"1434664650"},"temperature":{"T":"114.6","t":"1434664632"},"humidity":{"H":"9","t":"1434664632"},"rainCounter":{"RC":"445","t":"1434664650"}}
Yes, the outside temperature is REALLY 114.4, remember I live in Arizona and this is the middle of June, but it's a dry heat, the humidity reading is 9%. As if that matters when the temperature is this high ...
But, also notice that I have both strings coming to the subscription even though they are very different devices, on different Pi's. Heck, they could be half way around the world and it wouldn't matter, it would still work. I tested that with a conspirator just this morning.
Now to follow my plans, I have to catch this data in some other process and save it to my database. I'll get to that soon, maybe even today, but I'm going to hold off on changing the presentation to the web for later because I have many different presentations to consider when I get to that point. See, currently I take everything presented from my data bases. If I'm using my Android app to look at the house, the data is coming out of the data base. When I send a command to close the garage doors, the command is executed and the app doesn't update until it is recorded in the data base. I may change that to reflect the latest reading from the mqtt server. We'll see, and then I'll probably change it.
The code is not in GitHub, I'm going to hold off until I get enough in there to make a significant difference. But, the example from the last post is exactly what I did with the difference above. This is a lot simpler and more versatile than I expected.
Wednesday, June 17, 2015
OK, Fine, I'll take a look at MQTT
A number of my readers have mentioned mqtt as a great tool for the kind of thing I'm doing around the house. A couple of them have pointed out that what I'm doing could be done much easier using mqtt and even sent me web sites to look at.
Naturally, I resisted. That's sort of the way I am, but also, the various web sites out there make it look so complex that I just didn't want to try and deal with it. Another stinking process with an API and things I had to learn to use it ... gag!
Then Glenn sent me a link to an Arduino implementation of publishing data to an mqtt server <link> and things clicked into place. What happened was the Arduino was too small and slow for fancy implementations, it had to fit in a couple of K and run on a 16MHz machine, so it was as simple as it could be. That got me to thinking --- there's really no protocol, you build it yourself, There's no registration, special language, ... none of the stuff that drives me nuts. You just use it.
Fine, I've eaten my words before, so I gave it a shot. AWESOME set of tools. This thing is going to be the center piece of a major update to my house control system.
OK, Glen, Brant, others too many to itemize, go ahead ... scream, "I told you so, but you wouldn't listen." I deserve it.
The way I started was to install the mqtt client software on my weather station Pi. I put a few lines in the python code to import, initialize an object and then a single line to send the data to a public mqtt server. Yes, that's literally all there was to publishing data, one single line of actual code.
Then I used the one of the tools to see if it actually got there and I could read it back.
If you've been following the weather station posts, these lines will look really familiar. The tool mosqitto_sub is a test subscriber that took the public server I used, 'test.mosquitto.org', and looked for the published data keyed under 'Desert-Home/Weather/String' and printed it to the console of the Pi. So, what was happening was that my weather station Pi was reading the data from a radio and sending it over the internet to a server. Then on a different Pi, I was reading the data back as it came in from the server I subscribed to. Three machines involved working over the internet and all I had to do was write one line of actual code to do it.
To try it out, install the mqtt client software for python and the command line tool I used above. No, I didn't misspell it, there really are two 't's in there.
sudo apt-get install mosquitto mosquitto-clients
sudo pip install paho-mqtt
This will actually install the mqtt broker and start it up, but you can stop it by the command:
sudo /etc/init.d/mosquitto stop
Then, in some code somewhere, put the lines:
import paho.mqtt.client as mqtt
The string "Desert-Home/Weather/String" is called a 'topic'. This is a hierarchical set of keys that you construct for your own purposes. I chose "Desert-Home" for obvious reasons, then "Weather" because it's for the weather station, then "String" because I was publishing the JSON string produced by my code that reads the AcuRite 5n1 weather head. I may create another item "OutsideTemperature" to hold the latest outside reading from the fence post sensor in which case it would be:
"Desert-Home/Weather/OutsideTemperature."
Then to get the data back as it is published to the server, use a line like:
mosquitto_sub -h "test.mosquitto.org" -t "Whatever/you/decided" -v
Slick isn't it. There's no mystery here, no special stuff you have to learn, it's all totally up to you how you use it. To make it more versatile, the '#' and '+' characters have special meaning that can help you debug and select things. Check the various sites out there for the usage of those symbols.
There's a ba-jillion web sites out there that talk about mqtt, so I'm not going to recap the light-weight, short transaction, fail safe blather that they cover ad nauseum, instead over the next few weeks, I'm going to convert my house monitor and control system to actually use it and try to show you how to follow my footsteps improving and changing it to suit your particular needs. I'm sure I'm going to mess it up and have to redo things a couple of hundred times, but this should, at least, be good for a laugh.
During this process I'm going to visit my sensor devices and update the code on them to my latest experience level and implement several changes to them as well. JSON strings as data output and input so I can get rid of parsers, that kind of stuff. I'll gateway the devices from the XBee network to messages that will be published to mqtt and caught by whatever needs to handle the data. I don't know what I'm going to do with the Iris devices yet, but something will come to me.
Stay tuned.
Naturally, I resisted. That's sort of the way I am, but also, the various web sites out there make it look so complex that I just didn't want to try and deal with it. Another stinking process with an API and things I had to learn to use it ... gag!
Then Glenn sent me a link to an Arduino implementation of publishing data to an mqtt server <link> and things clicked into place. What happened was the Arduino was too small and slow for fancy implementations, it had to fit in a couple of K and run on a 16MHz machine, so it was as simple as it could be. That got me to thinking --- there's really no protocol, you build it yourself, There's no registration, special language, ... none of the stuff that drives me nuts. You just use it.
Fine, I've eaten my words before, so I gave it a shot. AWESOME set of tools. This thing is going to be the center piece of a major update to my house control system.
OK, Glen, Brant, others too many to itemize, go ahead ... scream, "I told you so, but you wouldn't listen." I deserve it.
The way I started was to install the mqtt client software on my weather station Pi. I put a few lines in the python code to import, initialize an object and then a single line to send the data to a public mqtt server. Yes, that's literally all there was to publishing data, one single line of actual code.
Then I used the one of the tools to see if it actually got there and I could read it back.
pi@deserthome:~/src/other-things$ mosquitto_sub -h "test.mosquitto.org" -t "Desert-Home/#" -v
Desert-Home/Weather/String {"sensorId":{"SID":"92","t":"1434572606"},"channel":{"CH":"A","t":"1434572606"},"messageCaught":{"MC":"0","t":"1434572606"},"battLevel":{"BAT":"7","t":"1434572606"},"windSpeed":{"WS":"4.0","t":"1434572606"},"windDirection":{"WD":"SSE","t":"1434572587"},"temperature":{"T":"111.7","t":"1434572606"},"humidity":{"H":"11","t":"1434572606"},"rainCounter":{"RC":"445","t":"1434572587"}}
Desert-Home/Weather/String {"sensorId":{"SID":"92","t":"1434572606"},"channel":{"CH":"A","t":"1434572606"},"messageCaught":{"MC":"0","t":"1434572606"},"battLevel":{"BAT":"7","t":"1434572606"},"windSpeed":{"WS":"4.0","t":"1434572606"},"windDirection":{"WD":"SSE","t":"1434572587"},"temperature":{"T":"111.7","t":"1434572606"},"humidity":{"H":"11","t":"1434572606"},"rainCounter":{"RC":"445","t":"1434572587"}}
Desert-Home/Weather/String {"sensorId":{"SID":"92","t":"1434572623"},"channel":{"CH":"A","t":"1434572623"},"messageCaught":{"MC":"0","t":"1434572623"},"battLevel":{"BAT":"7","t":"1434572623"},"windSpeed":{"WS":"2.5","t":"1434572623"},"windDirection":{"WD":"SSW","t":"1434572623"},"temperature":{"T":"111.7","t":"1434572606"},"humidity":{"H":"11","t":"1434572606"},"rainCounter":{"RC":"445","t":"1434572623"}}
To try it out, install the mqtt client software for python and the command line tool I used above. No, I didn't misspell it, there really are two 't's in there.
sudo apt-get install mosquitto mosquitto-clients
sudo pip install paho-mqtt
This will actually install the mqtt broker and start it up, but you can stop it by the command:
sudo /etc/init.d/mosquitto stop
Then, in some code somewhere, put the lines:
import paho.mqtt.client as mqtt
mqttc = mqtt.Client()
mqttc.connect("test.mosquitto.org", 1883, 60)
mqttc.loop_start()
To create the object, initialize it and start it up. Then the single line somewhere you are logging or recording something like this:
mqttc.publish("Desert-Home/Weather/String",buff);
"Desert-Home/Weather/OutsideTemperature."
Then to get the data back as it is published to the server, use a line like:
mosquitto_sub -h "test.mosquitto.org" -t "Whatever/you/decided" -v
Slick isn't it. There's no mystery here, no special stuff you have to learn, it's all totally up to you how you use it. To make it more versatile, the '#' and '+' characters have special meaning that can help you debug and select things. Check the various sites out there for the usage of those symbols.
There's a ba-jillion web sites out there that talk about mqtt, so I'm not going to recap the light-weight, short transaction, fail safe blather that they cover ad nauseum, instead over the next few weeks, I'm going to convert my house monitor and control system to actually use it and try to show you how to follow my footsteps improving and changing it to suit your particular needs. I'm sure I'm going to mess it up and have to redo things a couple of hundred times, but this should, at least, be good for a laugh.
During this process I'm going to visit my sensor devices and update the code on them to my latest experience level and implement several changes to them as well. JSON strings as data output and input so I can get rid of parsers, that kind of stuff. I'll gateway the devices from the XBee network to messages that will be published to mqtt and caught by whatever needs to handle the data. I don't know what I'm going to do with the Iris devices yet, but something will come to me.
Stay tuned.
Tuesday, May 26, 2015
Hacking Into The Iris Door Sensor, Part 3, The code
The previous post in the project is here <link>
Last time I described the Key Fob that Lowe's sells and previous to that I showed the Door Switch, but I haven't shown you the way I got them to work. I'm still a little reluctant because my partner in this project hasn't succeeded in getting his devices to work properly, but it occurred to me that someone out there might pick up the code and give it a try. I know several folk have used my Smart Switch code to control and monitor things around the house, maybe they're just waiting for me to publish how to do it.
At any rate, this is how I got the switches and key fob to behave:
I have eight devices, five of the smart switches, two door switches, and a key fob, and all of them are working with the same monitoring code. The code is only to monitor the devices, there's no provisions for control, Actually, the only thing you can control is the smart switches, and I've already posted code for that, so this will show you how to monitor all the devices. This code works pretty easily, pull the battery out of the device, start the code, put the battery back in and push the button eight times quickly. The switch will contact the code and join on its own. Once joined, the door switch and key fob will send status every two minutes telling you they're alive and in range. The smart switches send status much more often with a cumulative status every minute. If you want to join other devices, just pull the battery, put it back, and push the button a bunch of times; joining is always enabled and the next device should work fine. Stopping the code (cntrl-C) and restarting it won't cause the devices to leave the network, they'll be fine when you restart the code. I did this a LOT when I was trying to get them to work.
I could go into intimate details of how the interaction works, but the code will show you this much better. What happens is that the switch sends an device announce message and the code will respond and then they exchange data. During this there are two specific message that are sent to Alertme devices to get them to accept the XBee coordinator; this is the secret part of the Iris system that makes these devices unique. The code I came up with for the smart switches wouldn't work because the door switch and key fob are slower devices and I was hitting them too quickly with the initialization messages. They didn't have time to recognize and react to them. A couple of sleep calls took care of that problem. I used the same setup on the XBee as when I hacked into the Smart Switch, you can look at it here <link>
Once running, you can press a switch on the key fob and it will transmit a messsage telling you that the button was pressed or released along with a counter that represents the time in milliseconds off the action. By subtracting the released time from the pressed time you can look for a long press as opposed to a short press. Nice feature.
The door switch has two switches, a tamper switch that will tell you if the cover is taken off and a reed switch that tells you the door has been opened. The tamper switch will tell you press and release like the key fob does, as will the reed switch. This means you can tell if the door is opened or closed as well as the cover off or on.
Like the smart switches, the key fob and door switch once joined, stay that way. You can pull the battery and put it back in without it having a problem. You can also drive away out of range and come back and everything works just fine. If you kill the coordinator by pulling the power or something, you might have to make them rejoin. This all depends on how long it was off and what happened during the off period. Sometimes the coordinator XBee will choose a different channel and then the switches will have trouble finding it. A simple power failure won't cause any problem, but reloading software to the controller could mean a visit to the devices to force them to rejoin. This kind of problem can be avoided by using the parameters on the XBee controller to limit it to a single predetermined channel. Doing this means the XBee comes up, establishes a network on the channel and the devices then just continue on as before.
So, you're going to get a lot of data as you add devices. Each device will interact at least every couple of minutes and every time you mess with them. For me, this meant missed messages. The traffic was pretty steady, but things didn't look right until I added a queue to the code to hold the messages and took them off one at a time to deal with. There's plenty of unused time to handle the messages, but they tend to come in in bunches and can cause problems. Adding a queue seemed to help that problem a lot; I don't see any missing or partial messages when I run it.
There's lots of debug and logging in the code as well as a test to help you isolate a single device. You'll see it in the code; I look for a list of devices to listen to and ignore the others. I had to do this to keep the other devices on the network from confusing me when I was trying figure out which bit meant what. Look for the test at the top of the code to decode the messages and adjust it as you need to. Also, this doesn't save anything from one run to the next. There's no database of devices that gets updated and the state of the devices isn't saved to compare with later. Once you're comfortable with the code, simply add whatever you want to handle things. I'm going to hook this code in with my database of house devices to save the states and decide exactly what I want to do with it later when I have a little more experience.
Here's the code:
Yes, it's a bit complex. More complex than other hacks I've posted. I had to do this to allow for multiple devices that had multiple endpoints and clusters involved. It still isn't too hard to understand, and you should be fine when you start messing with it.
You'll notice that I keep a list of devices that have contacted the code. This is to allow you to tell what happened with various devices as the interaction continues. I log the long address, time and relevant details as they happen to help you understand how to deal with the incoming data. When you have several devices, the log can become a bit overpowering since nothing stops sending, it just goes on forever. Here's a bit of the logging so you can get an idea what it looks like:
The first field is the Unix timestamp. Basically the number of seconds since 1970. The second is the long address of the device; this never changes. Then the cluster id followed by the data specific to the cluster. I decoded the times for the key fob and the temperature for the door switch because those could be fun to play with. Notice that the smart switches really send a lot of data, this will help you understand why I limit the number of devices with a test at the top of the code. If everything is there, you have a heck of a time picking out what you want to see. The list of know devices is the long address and the currently assigned short address. The short address will change from time to time as you restart the process or pull a battery somewhere.
Remember, this runs on a Raspberry Pi, I haven't tried it on anything else. I have my XBee plugged into a USB port on the Pi, if you're using the serial port, just change the port in the code, that should be all that is needed. If you don't know which usb port to use, there's a ton of articles on the web that will help you find the port, just look around a bit, or to see how I did it look here <link>. What I'm hoping for is a couple of folk that want to use these nicely priced and well made devices with their own code for some project to grab it and give me some feedback on the interaction. It would be nice to see someone else running this code.
Have fun.
The next part to this project is here <link>
Last time I described the Key Fob that Lowe's sells and previous to that I showed the Door Switch, but I haven't shown you the way I got them to work. I'm still a little reluctant because my partner in this project hasn't succeeded in getting his devices to work properly, but it occurred to me that someone out there might pick up the code and give it a try. I know several folk have used my Smart Switch code to control and monitor things around the house, maybe they're just waiting for me to publish how to do it.
At any rate, this is how I got the switches and key fob to behave:
I have eight devices, five of the smart switches, two door switches, and a key fob, and all of them are working with the same monitoring code. The code is only to monitor the devices, there's no provisions for control, Actually, the only thing you can control is the smart switches, and I've already posted code for that, so this will show you how to monitor all the devices. This code works pretty easily, pull the battery out of the device, start the code, put the battery back in and push the button eight times quickly. The switch will contact the code and join on its own. Once joined, the door switch and key fob will send status every two minutes telling you they're alive and in range. The smart switches send status much more often with a cumulative status every minute. If you want to join other devices, just pull the battery, put it back, and push the button a bunch of times; joining is always enabled and the next device should work fine. Stopping the code (cntrl-C) and restarting it won't cause the devices to leave the network, they'll be fine when you restart the code. I did this a LOT when I was trying to get them to work.
I could go into intimate details of how the interaction works, but the code will show you this much better. What happens is that the switch sends an device announce message and the code will respond and then they exchange data. During this there are two specific message that are sent to Alertme devices to get them to accept the XBee coordinator; this is the secret part of the Iris system that makes these devices unique. The code I came up with for the smart switches wouldn't work because the door switch and key fob are slower devices and I was hitting them too quickly with the initialization messages. They didn't have time to recognize and react to them. A couple of sleep calls took care of that problem. I used the same setup on the XBee as when I hacked into the Smart Switch, you can look at it here <link>
Once running, you can press a switch on the key fob and it will transmit a messsage telling you that the button was pressed or released along with a counter that represents the time in milliseconds off the action. By subtracting the released time from the pressed time you can look for a long press as opposed to a short press. Nice feature.
The door switch has two switches, a tamper switch that will tell you if the cover is taken off and a reed switch that tells you the door has been opened. The tamper switch will tell you press and release like the key fob does, as will the reed switch. This means you can tell if the door is opened or closed as well as the cover off or on.
Like the smart switches, the key fob and door switch once joined, stay that way. You can pull the battery and put it back in without it having a problem. You can also drive away out of range and come back and everything works just fine. If you kill the coordinator by pulling the power or something, you might have to make them rejoin. This all depends on how long it was off and what happened during the off period. Sometimes the coordinator XBee will choose a different channel and then the switches will have trouble finding it. A simple power failure won't cause any problem, but reloading software to the controller could mean a visit to the devices to force them to rejoin. This kind of problem can be avoided by using the parameters on the XBee controller to limit it to a single predetermined channel. Doing this means the XBee comes up, establishes a network on the channel and the devices then just continue on as before.
So, you're going to get a lot of data as you add devices. Each device will interact at least every couple of minutes and every time you mess with them. For me, this meant missed messages. The traffic was pretty steady, but things didn't look right until I added a queue to the code to hold the messages and took them off one at a time to deal with. There's plenty of unused time to handle the messages, but they tend to come in in bunches and can cause problems. Adding a queue seemed to help that problem a lot; I don't see any missing or partial messages when I run it.
There's lots of debug and logging in the code as well as a test to help you isolate a single device. You'll see it in the code; I look for a list of devices to listen to and ignore the others. I had to do this to keep the other devices on the network from confusing me when I was trying figure out which bit meant what. Look for the test at the top of the code to decode the messages and adjust it as you need to. Also, this doesn't save anything from one run to the next. There's no database of devices that gets updated and the state of the devices isn't saved to compare with later. Once you're comfortable with the code, simply add whatever you want to handle things. I'm going to hook this code in with my database of house devices to save the states and decide exactly what I want to do with it later when I have a little more experience.
Here's the code:
#! /usr/bin/python
'''
This is a hack into the operation of the Iris Smart Home Smart
Plug, Door Switch, and Key Fob. The evolution of this is discussed
on Desert-Home.com in detail.
Have fun
'''
from apscheduler.schedulers.background import BackgroundScheduler
from xbee import ZigBee
import logging
import datetime
import Queue
import time
import serial
import sys, traceback
import shlex
from struct import *
def printData(data):
print "********** Message Contents"
for key, value in data.iteritems():
if key == "id":
print key, value
else:
print key, "".join("%02x " % ord(b) for b in data[key])
print "**********"
def clusterData(lAddr,clusterId, data):
print int(time.time()),
print "".join("%02x" % ord(b) for b in lAddr) + \
" clid "+"%04x" % clusterId + "-" + \
"".join("%02x " % ord(b) for b in data)
# this is a call back function. When a message
# comes in this function will get the data
# I had to use a queue to make sure there was enough time to
# decode the incoming messages. Otherwise, in heavy traffic
# periods, I'd get a new message while I was still working on
# the last one.
def messageReceived(data):
#print "queueing message"
messageQueue.put(data)
def handleMessage (data):
try:
'''
I have a network of these devices and had to
add this test to keep the log down to a reasonable
size. If all the devices show up, it's hard to
tell what is going on. Just uncomment the check
and put whatever you want to watch in.
'''
'''if data['source_addr_long'] not in \
['\x00\x0d\x6f\x00\x03\xc2\x71\xcc',
'\x00\x0d\x6f\x00\x02\x83\xfa\x4e']:
return'''
#print ''
#print 'gotta packet',
#printData(data)
if (data['id'] == 'rx_explicit'):
#print "RX Explicit"
#printData(data)
clusterId = (ord(data['cluster'][0])*256) + ord(data['cluster'][1])
#print 'Cluster ID:', hex(clusterId),
#print "profile id:", repr(data['profile'])
if (data['profile']=='\x00\x00'): # The General Profile
print 'Cluster ID:', hex(clusterId),
print "profile id:", repr(data['profile'])
if (clusterId == 0x0000):
print ("Network (16-bit) Address Request")
#printData(data)
elif (clusterId == 0x0004):
# Simple Descriptor Request,
print("Simple Descriptor Request")
#printData(data)
elif (clusterId == 0x0005):
# Active Endpoint Request,
print("Active Endpoint Request")
#printData(data)
elif (clusterId == 0x0006):
print "Match Descriptor Request"
#printData(data)
time.sleep(2)
print "Sending match descriptor response"
zb.send('tx_explicit',
dest_addr_long = data['source_addr_long'],
dest_addr = data['source_addr'],
src_endpoint = '\x00',
dest_endpoint = '\x00',
cluster = '\x80\x06',
profile = '\x00\x00',
options = '\x01',
data = '\x04\x00\x00\x00\x01\x02'
)
# The contact switch is a bit slow, give it
# some time to digest the messages.
time.sleep(2)
zb.send('tx_explicit',
dest_addr_long = data['source_addr_long'],
dest_addr = data['source_addr'],
src_endpoint = '\x02',
dest_endpoint = '\x02',
cluster = '\x00\xf6',
profile = '\xc2\x16',
data = '\x11\x01\xfc'
)
time.sleep(2)
elif (clusterId == 0x0008):
# I couldn't find a definition for this
print("This was probably sent to the wrong profile")
elif (clusterId == 0x0013):
# This is the device announce message.
print 'Device Announce Message'
# this will tell me the address of the new thing
# so I'm going to send an active endpoint request
print 'Sending active endpoint request'
epc = '\xaa'+data['source_addr'][1]+data['source_addr'][0]
print "".join("%02x " % ord(b) for b in epc)
zb.send('tx_explicit',
dest_addr_long = data['source_addr_long'],
dest_addr = data['source_addr'],
src_endpoint = '\x00',
dest_endpoint = '\x00',
cluster = '\x00\x05',
profile = '\x00\x00',
options = '\x01',
data = epc
)
#printData(data)
elif (clusterId == 0x8000):
print("Network (16-bit) Address Response")
#printData(data)
elif (clusterId == 0x8038):
print("Management Network Update Request");
elif (clusterId == 0x8005):
# this is the Active Endpoint Response This message tells you
# what the device can do
print 'Active Endpoint Response'
printData(data)
elif (clusterId == 0x8004):
print "simple descriptor response"
else:
print ("Unimplemented Cluster ID", hex(clusterId))
print
elif (data['profile']=='\xc2\x16'): # Alertme Specific
if data['source_addr_long'] not in devices:
devices.setdefault(data['source_addr_long'], []).append(data['source_addr'])
# suppress printing for known clusters
# so I can look at it more closely
if clusterId not in [0X0500, 0x00ef, 0x00f0, 0x00f2, 0x00f3,0x00f6]:
printData(data)
print "Unhandled Message"
if (clusterId == 0xef):
clusterData(data['source_addr_long'],clusterId,data['rf_data'])
clusterCmd = ord(data['rf_data'][2])
status = data['rf_data'] # cut down on typing
if (clusterCmd == 0x81):
usage = unpack('<H', status[3:5])[0]
print " Current Usage:", usage
elif (clusterCmd == 0x82):
usage = unpack('<L', status[3:7])[0] / 3600
upTime = unpack('<L', status[7:11])[0]
print (" Switch Minute Stats: Usage, %d Watt Hours; Uptime, %d Seconds" %(usage/3600, upTime))
elif (clusterId == 0x00f6):
clusterData(data['source_addr_long'],clusterId,data['rf_data'])
print ''
print "Identify Message"
print "Sending init message"
zb.send('tx_explicit',
dest_addr_long = data['source_addr_long'],
dest_addr = data['source_addr'],
src_endpoint = '\x00',
dest_endpoint = '\x02',
cluster = '\x00\xf0',
profile = '\xc2\x16',
data = '\x19\x41\xfa\x00\x01'
)
elif (clusterId == 0x00f3):
clusterData(data['source_addr_long'],clusterId,data['rf_data'])
print ' Key Fob Button',
status = data['rf_data']
print ord(status[3]),
if status[2] == '\x01':
print 'Closed',
elif status[2] == '\x00':
print 'Open',
else:
print 'Unknown',
print 'Counter', unpack('<H',status[5:7])[0],
print ''
pass
elif (clusterId == 0x00f2):
clusterData(data['source_addr_long'],clusterId,data['rf_data'])
print 'Tamper Switch Changed State'
pass
elif (clusterId == 0x00f0):
clusterData(data['source_addr_long'],clusterId,data['rf_data'])
# If the cluster cmd byte is 'xfb', it's a status
if data['rf_data'][2] == '\xfb':
status = data['rf_data'] # just to make typing easier
if status[3] == '\x1f':
print " Door Sensor",
print str(float(unpack("<h", status[8:10])[0])\
/ 100.0 * 1.8 + 32) + "F",
elif status[3] == '\x1c':
# Never found anything useful in this
print "Power Switch",
elif status[3] == '\x1d':
print " Key Fob",
print str(float(unpack("<h", status[8:10])[0])\
/ 100.0 * 1.8 + 32) + "F",
unpack('<I',status[4:8])[0]
print 'Counter', unpack('<I',status[4:8])[0],
elif status[3] == '\x1e':
# I haven't figured out what this is yet
# it comes from a door switch and the temperature
# field is always ff ff, it may be an error
# indication.
pass
else:
print " Don't know this device yet",
print ''
pass
elif (clusterId == 0x0500): # This is the security cluster
clusterData(data['source_addr_long'],clusterId,data['rf_data'])
# When the switch first connects, it come up in a state that needs
# initialization, this command seems to take care of that.
# So, look at the value of the data and send the command.
if data['rf_data'][3:7] == '\x15\x00\x39\x10':
print "sending initialization"
zb.send('tx_explicit',
dest_addr_long = data['source_addr_long'],
dest_addr = data['source_addr'],
src_endpoint = '\x00',
dest_endpoint = '\x00',
cluster = '\x05\x00',
profile = '\xc2\x16',
data = '\x11\x80\x00\x00\x05'
)
# The switch state is in byte [3] and is a bitfield
# bit 0 is the magnetic reed switch state
# bit 3 is the tamper switch state
switchState = ord(data['rf_data'][3])
if switchState & 0x04:
print 'Tamper Switch Closed',
else:
print 'Tamper Switch Open',
if switchState & 0x01:
print 'Reed Switch Opened',
else:
print 'Reed Switch Closed',
print ''
pass
else:
print ("Unimplemented Profile ID")
elif(data['id'] == 'route_record_indicator'):
print("Route Record Indicator")
else:
print("some other type of packet")
print(data)
except:
print "I didn't expect this error:", sys.exc_info()[0]
traceback.print_exc()
def showDevices():
print "Known Devices ************"
for key in devices:
print "".join("%02x " % ord(b) for b in key)+':',
print "".join("%02x " % ord(b) for b in devices[key][0])
print "**************************"
def sendSwitch(whereLong, whereShort, srcEndpoint, destEndpoint,
clusterId, profileId, clusterCmd, databytes):
payload = '\x11\x00' + clusterCmd + databytes
zb.send('tx_explicit',
dest_addr_long = whereLong,
dest_addr = whereShort,
src_endpoint = srcEndpoint,
dest_endpoint = destEndpoint,
cluster = clusterId,
profile = profileId,
data = payload
)
def tryCommand():
# Try out commands here
print '********* Sending Test Command'
switchLongAddr ='\x00\x0d\x6f\x00\x03\xc2\x71\xcc'
if switchLongAddr in devices:
switchShortAddr = devices[switchLongAddr][0]
print ' Switch Status'
sendSwitch(switchLongAddr, switchShortAddr, '\x00', '\x02',
'\x00\xee', '\xc2\x16', '\x01', '\x01')
else:
print 'Waiting for short address'
#------------ XBee Stuff -------------------------
# this is the /dev/serial/by-id device for the USB card that holds the XBee
ZIGBEEPORT = "/dev/serial/by-id/usb-FTDI_FT232R_USB_UART_A901QL3F-if00-port0"
ZIGBEEBAUD_RATE = 9600
# Open serial port for use by the XBee
ser = serial.Serial(ZIGBEEPORT, ZIGBEEBAUD_RATE)
# The XBee addresses I'm dealing with
BROADCAST = '\x00\x00\x00\x00\x00\x00\xff\xff'
UNKNOWN = '\xff\xfe' # This is the 'I don't know' 16 bit address
#-------------------------------------------------
logging.basicConfig()
# Create XBee library API object, which spawns a new thread
zb = ZigBee(ser, callback=messageReceived)
# create a queue to put the messages into so they can
# be handled in turn without one interrupting the next.
messageQueue = Queue.Queue(0)
# A dictionary to put devices into as they show up
devices = {}
scheditem = BackgroundScheduler()
scheditem.add_job(showDevices, 'interval', seconds=60)
#scheditem.add_job(tryCommand, 'interval', seconds=15)
scheditem.start()
print ("started")
notYet = True;
firstTime = True;
while True:
try:
if (firstTime):
# this is in case I need some initialization in the
# future
firstTime = False
if messageQueue.qsize() > 0:
#print "getting message"
message = messageQueue.get()
handleMessage(message)
messageQueue.task_done();
time.sleep(0.1)
#print ("tick") # This is here to let you know it's alive
sys.stdout.flush() # if you're running non interactive, do this
except IndexError:
print "empty line"
except NameError as e:
print "NameError:",
print e.message.split("'")[1]
except KeyboardInterrupt:
print ("Keyboard interrupt")
break
except:
print ("I didn't expect this error:", sys.exc_info()[0])
traceback.print_exc()
break
print ("After the while loop")
# halt() must be called before closing the serial
# port in order to ensure proper thread shutdown
scheditem.shutdown(wait=False) # shut down the apscheduler
zb.halt()
ser.close()
You'll notice that I keep a list of devices that have contacted the code. This is to allow you to tell what happened with various devices as the interaction continues. I log the long address, time and relevant details as they happen to help you understand how to deal with the incoming data. When you have several devices, the log can become a bit overpowering since nothing stops sending, it just goes on forever. Here's a bit of the logging so you can get an idea what it looks like:
started 1432598997 000d6f0004510782 clid 00f0-09 9e fb 1f bf 9e 7a 0a b7 0b c4 01 af f4 03 02 Door Sensor 85.982F 1432598998 000d6f0003cf0e5b clid 00ef-09 00 81 00 00 Current Usage: 0 1432598999 000d6f000283fa4e clid 00f0-09 a2 fb 1d 3c a2 e5 03 bc 0b 00 00 a7 fb 03 00 Key Fob 86.072F Counter 65380924 1432599000 000d6f00025886b3 clid 00ef-09 00 81 87 00 Current Usage: 135 1432599000 000d6f00025886b3 clid 00f0-09 00 fb 1c 22 dd 95 83 66 32 00 00 b5 b0 01 00 Power Switch 1432599000 000d6f000237b25a clid 00ef-09 00 81 01 00 Current Usage: 1 1432599003 000d6f000258a4cc clid 00ef-09 00 81 00 00 Current Usage: 0 1432599005 000d6f0002547a5d clid 00ef-09 00 81 00 00 Current Usage: 0 1432599007 000d6f0002547a5d clid 00f0-09 00 fb 1c 2b 09 b3 1a 66 32 00 00 b7 fe 01 00 Power Switch 1432599008 000d6f0003cf0e5b clid 00ef-09 00 81 00 00 Current Usage: 0 1432599009 000d6f00025886b3 clid 00ef-09 00 82 67 a4 a5 72 80 e5 20 01 00 Switch Minute Stats: Usage, 148 Watt Hours; Uptime, 18933120 Seconds 1432599010 000d6f00025886b3 clid 00ef-09 00 81 87 00 Current Usage: 135 1432599010 000d6f000237b25a clid 00ef-09 00 81 01 00 Current Usage: 1 1432599013 000d6f000258a4cc clid 00ef-09 00 82 f7 ea 21 00 b8 bc 03 00 00 Switch Minute Stats: Usage, 0 Watt Hours; Uptime, 244920 Seconds 1432599013 000d6f000258a4cc clid 00ef-09 00 81 00 00 Current Usage: 0 1432599015 000d6f0002547a5d clid 00ef-09 00 81 00 00 Current Usage: 0 1432599018 000d6f0003cf0e5b clid 00ef-09 00 81 00 00 Current Usage: 0 1432599020 000d6f00025886b3 clid 00ef-09 00 81 87 00 Current Usage: 135 1432599020 000d6f000237b25a clid 00ef-09 00 82 a9 3d 8a 1f 40 e9 20 01 00 Switch Minute Stats: Usage, 40 Watt Hours; Uptime, 18934080 Seconds 1432599020 000d6f0003cf0e5b clid 00f0-09 00 fb 1c 34 a9 6e 64 fb 31 00 00 ba b7 01 00 Power Switch 1432599020 000d6f000237b25a clid 00ef-09 00 81 01 00 Current Usage: 1 1432599023 000d6f000237b25a clid 00f0-09 00 fb 1c 25 0d a5 83 4a 32 00 00 b4 b4 01 00 Power Switch 1432599023 000d6f000258a4cc clid 00ef-09 00 81 00 00 Current Usage: 0 1432599025 000d6f0002547a5d clid 00ef-09 91 82 40 35 ff 00 d4 ac 06 00 00 Switch Minute Stats: Usage, 1 Watt Hours; Uptime, 437460 Seconds 1432599025 000d6f0002547a5d clid 00ef-09 00 81 00 00 Current Usage: 0 1432599025 000d6f000258a4cc clid 00f0-09 00 fb 1c 18 11 f3 0e 66 32 00 00 bd b3 01 00 Power Switch 1432599028 000d6f0003cf0e5b clid 00ef-09 00 81 00 00 Current Usage: 0 1432599030 000d6f00025886b3 clid 00ef-09 00 81 87 00 Current Usage: 135 1432599030 000d6f00025886b3 clid 00f0-09 00 fb 1c 22 55 96 83 66 32 00 00 b6 b6 01 00 Power Switch 1432599030 000d6f000237b25a clid 00ef-09 00 81 01 00 Current Usage: 1 1432599033 000d6f000258a4cc clid 00ef-09 00 81 00 00 Current Usage: 0 1432599035 000d6f0002547a5d clid 00ef-09 00 81 00 00 Current Usage: 0 1432599036 000d6f0002547a5d clid 00f0-09 00 fb 1c 2b 81 b3 1a 66 32 00 00 b7 fd 01 00 Power Switch 1432599038 000d6f0003cf0e5b clid 00ef-09 00 82 38 42 54 00 bc 1b 19 00 00 Switch Minute Stats: Usage, 0 Watt Hours; Uptime, 1645500 Seconds 1432599038 000d6f0003cf0e5b clid 00ef-09 00 81 00 00 Current Usage: 0 1432599040 000d6f00025886b3 clid 00ef-09 00 81 87 00 Current Usage: 135 1432599040 000d6f000237b25a clid 00ef-09 00 81 01 00 Current Usage: 1 1432599043 000d6f000258a4cc clid 00ef-09 00 81 00 00 Current Usage: 0 1432599045 000d6f0002547a5d clid 00ef-09 00 81 00 00 Current Usage: 0 1432599048 000d6f0003cf0e5b clid 00ef-09 00 81 00 00 Current Usage: 0 1432599050 000d6f00025886b3 clid 00ef-09 00 81 87 00 Current Usage: 135 1432599050 000d6f0003cf0e5b clid 00f0-09 00 fb 1c 34 21 6f 64 fb 31 00 00 ba bb 01 00 Power Switch 1432599050 000d6f000237b25a clid 00ef-09 00 81 01 00 Current Usage: 1 1432599053 000d6f000237b25a clid 00f0-09 00 fb 1c 25 85 a5 83 4a 32 00 00 b4 a4 01 00 Power Switch 1432599054 000d6f000258a4cc clid 00ef-09 00 81 00 00 Current Usage: 0 1432599055 000d6f0002547a5d clid 00ef-09 00 81 00 00 Current Usage: 0 1432599055 000d6f000258a4cc clid 00f0-09 00 fb 1c 18 89 f3 0e 66 32 00 00 bd b1 01 00 Power Switch Known Devices ************ 00 0d 6f 00 02 83 fa 4e : 92 26 00 0d 6f 00 04 51 07 82 : a9 3b 00 0d 6f 00 02 58 a4 cc : e1 3c 00 0d 6f 00 02 58 86 b3 : 27 f6 00 0d 6f 00 02 37 b2 5a : 2b d1 00 0d 6f 00 02 54 7a 5d : 41 c7 00 0d 6f 00 03 cf 0e 5b : 79 c5 ************************** 1432599058 000d6f0003cf0e5b clid 00ef-09 00 81 00 00 Current Usage: 0 1432599060 000d6f00025886b3 clid 00ef-09 00 81 87 00 Current Usage: 135 1432599060 000d6f00025886b3 clid 00f0-09 00 fb 1c 22 cd 96 83 66 32 00 00 b6 c1 01 00 Power Switch
Remember, this runs on a Raspberry Pi, I haven't tried it on anything else. I have my XBee plugged into a USB port on the Pi, if you're using the serial port, just change the port in the code, that should be all that is needed. If you don't know which usb port to use, there's a ton of articles on the web that will help you find the port, just look around a bit, or to see how I did it look here <link>. What I'm hoping for is a couple of folk that want to use these nicely priced and well made devices with their own code for some project to grab it and give me some feedback on the interaction. It would be nice to see someone else running this code.
Have fun.
The next part to this project is here <link>
Saturday, May 23, 2015
Hacking Into The Iris Door Sensor, Part 2
The previous post on this project is here <link>
Yes, I've made some progress on this project, but first let me talk about why I'm bothering with this. A couple of folk have asked me why I hack into these devices in general; the answer is not an esoteric, "Because they're there." I enjoy the challenge, but mostly it's because I want to use them without a bunch of strings attached.
The Iris devices are fairly priced, but not if you have to subscribe to a service that cost $10+ a month for the rest of time. From dismantling them I can tell the construction is actually pretty well done even though I keep seeing complaints on the web about how they should be smaller, prettier or come in colors. I couldn't build one as nice for the price. Actually, I couldn't build one as nice for twice the price. The single disadvantage is that their usefulness is tied to a web site that is run by a corporation that can change their mind at any time about anything and leave me hanging. Been there, done that, want to avoid it in the future.
Enough philosophy, I'm going to talk about the Iris Key Fob a bit. When I got some reaction from the door switch, I wondered if the key fob would work in a similar way, so since I was going to town anyway, I stopped off at Lowe's and picked up another door switch (I have two now) and a key fob as well.
Edit: I originally forgot to put up a picture it the Key Fob assembled, here it is:
Here it is with the cover off.
Same good construction as the door switch with a pair of switches that you press to activate funtions remotely. Let's pull it apart.
This is the case with the board removed to show how the plastic that operates the buttons is done. This makes for a nice feel when you're pushing the switch. The only problem I saw is that it's easy to press both switches if you squeeze it. A little care can overcome that problem though. Here's one side of the board,
and this is the other
You can click on the picture and zoom in if you want.
The switches have tactile feedback so you know you pressed it and the battery seats firmly so you know it's in there. Nice switch, albeit a bit large for some people's taste.
This switch was designed to control the Iris system in a home. If you push one button, it arms the alarm, and the other switch disarms it. It also provides feedback to the system that the fob is in range. People describe it as the Iris Hub sends messages to the fob every two minutes and if it doesn't get a response, the hub arms the alarm. My reading of the messages that fly back and forth shows that the fob reports every two minutes to the hub, so that would mean two minutes without a message and the hub will arm itself. There may be something I'm missing, but I'll turn it up eventually.
I see this device as two remote control switches that I can do any darn thing I want with. It could be on and off for a machine, light, ceiling fan, whatever I want. I can have one in each room attached to the bed for ceiling lights, or turn off the smoker in the yard if I want to.
I can hear my readers saying, "Fine, nice pictures and philosophy; tell us how the interaction works and give us some code." Well, I'm going to make you wait a couple of days before I bury you in code because this gets somewhat complicated and I'm not sure I've got enough bugs out to trust it yet. I have two door switches, a key fob and five smart switches running on the same XBee controller quite reliably, but a reader who 'volunteered' to try the code also has had nothing but problems.
Where my stuff joins and just works, his decided to get flaky and mess with his mind. We're trading logs and code back and forth to try and figure out what's different.
More later, but you got a look at the internals of a device, so stop complaining.
The next post on this project is here <link>
Yes, I've made some progress on this project, but first let me talk about why I'm bothering with this. A couple of folk have asked me why I hack into these devices in general; the answer is not an esoteric, "Because they're there." I enjoy the challenge, but mostly it's because I want to use them without a bunch of strings attached.
The Iris devices are fairly priced, but not if you have to subscribe to a service that cost $10+ a month for the rest of time. From dismantling them I can tell the construction is actually pretty well done even though I keep seeing complaints on the web about how they should be smaller, prettier or come in colors. I couldn't build one as nice for the price. Actually, I couldn't build one as nice for twice the price. The single disadvantage is that their usefulness is tied to a web site that is run by a corporation that can change their mind at any time about anything and leave me hanging. Been there, done that, want to avoid it in the future.
Enough philosophy, I'm going to talk about the Iris Key Fob a bit. When I got some reaction from the door switch, I wondered if the key fob would work in a similar way, so since I was going to town anyway, I stopped off at Lowe's and picked up another door switch (I have two now) and a key fob as well.
Edit: I originally forgot to put up a picture it the Key Fob assembled, here it is:
Here it is with the cover off.
Same good construction as the door switch with a pair of switches that you press to activate funtions remotely. Let's pull it apart.
This is the case with the board removed to show how the plastic that operates the buttons is done. This makes for a nice feel when you're pushing the switch. The only problem I saw is that it's easy to press both switches if you squeeze it. A little care can overcome that problem though. Here's one side of the board,
and this is the other
You can click on the picture and zoom in if you want.
The switches have tactile feedback so you know you pressed it and the battery seats firmly so you know it's in there. Nice switch, albeit a bit large for some people's taste.
This switch was designed to control the Iris system in a home. If you push one button, it arms the alarm, and the other switch disarms it. It also provides feedback to the system that the fob is in range. People describe it as the Iris Hub sends messages to the fob every two minutes and if it doesn't get a response, the hub arms the alarm. My reading of the messages that fly back and forth shows that the fob reports every two minutes to the hub, so that would mean two minutes without a message and the hub will arm itself. There may be something I'm missing, but I'll turn it up eventually.
I see this device as two remote control switches that I can do any darn thing I want with. It could be on and off for a machine, light, ceiling fan, whatever I want. I can have one in each room attached to the bed for ceiling lights, or turn off the smoker in the yard if I want to.
I can hear my readers saying, "Fine, nice pictures and philosophy; tell us how the interaction works and give us some code." Well, I'm going to make you wait a couple of days before I bury you in code because this gets somewhat complicated and I'm not sure I've got enough bugs out to trust it yet. I have two door switches, a key fob and five smart switches running on the same XBee controller quite reliably, but a reader who 'volunteered' to try the code also has had nothing but problems.
Where my stuff joins and just works, his decided to get flaky and mess with his mind. We're trading logs and code back and forth to try and figure out what's different.
More later, but you got a look at the internals of a device, so stop complaining.
The next post on this project is here <link>
Monday, May 11, 2015
Yet Another Update to the AcuRite 5N1 Weather Sensor
The previous post on this project is here <link>
Just when you thought I'd finally shut up about the AcuRite weather sensor I decide to post again. I promise, at some point I'll actually be satisfied with this device, but not quite yet. Over the months I've been working with this I've received comments and emails from folk that tried it and discovered things. One reader had an especially interesting situation; he had a neighbor that had a sensor within range. This meant that the code would pick them both up and present data from both sensors. To add insult to injury, when he tried changing his station to channel C, he found out that rtl_433 had a bug that prevented his receiving data on that channel. He must have been really annoyed at that point.
Since wind and temperature matter to placement, this was causing him grief and he set off to fix it. He isolated the sensor ID, and while he was at it, decoded the battery level, channel and found a better way to read the RF that allowed channel C to work.
During his trials he also found that the checksum was allowing bad data into the decoding and would present unusual readings. I've seen this over time as well. Things like a wind gust of 147 mph, a temperature spike of 30 or so degrees for no reason, the usual outlier readings, but with weather, it's hard to tell what an outlier really is.
I totally stole his ideas and some of his code and incorporated into my weather station,; I haven't been running it too long, but it looks like the vast majority of the outlier readings are gone. There may be some turn up, but so far it's been good.
First, inspection of the data showed that the bytes holding the various readings were all even parity. I added code to check the parity and discard the message if it didn't pass. Note that I only check the readings bytes because the first, second and last bytes are sensor, status, and checksum; these don't incorporate parity. In the c code I add the decoded sensor id, battery level, and channel to the JSON string I output, then check for the proper values in the python code that receives it. I did it this way because some folk may want to catch multiple sensors and average or compare them; especially if they can see a neighbors device for free.
I also noticed at my house that I receive a couple of devices from other folk that are not AcuRite devices. These things can cause problems as well, especially since they could easily be using the same checksum algorithm.
All the changes meant that I had to get the very latest version of rtl_433 from the original github repository. This was required because the rtl_433 folk made a new decoding method available that works better with the AcuRite data stream. That decoder wasn't there in January, but it is now. So, I tried again to fork it and failed miserably, and wound up just updating my copy of rtl_433. But, another good thing happened as well. The folk that wrote rtl_sdr, the actual radio software have improved it a lot and it installed first time, and I got to remove that from my copy.
Now, you get rtl_sdr, install it, build it, and then get my rtl_433, install it and build it. That will give you the version I'm using with all the changes to support the newest findings. My python code for the weather station I run is in GitHub as well. The specific stuff I use is:
rtl_sdr <link>
rtl_433 <link>
weather station <link>
The reason I linked to a particular commit in GitHub is so you can have exactly the same code to work with. The two rtl projects are in flux and could change dramatically at any time. You can always get the latest version if you think it will serve you better.
The latest version of rtl_433 not only has the new decoder software, it also allows you to specify the device you expect to receive. That makes it nice for me since I only have the AcuRite device I'm interested in. I changed my start up command to support this feature:
rtl_433 -f 433.915e6 -R9 2>> saveweather.log | saveweather.py >> saveweather.log 2>&1
The '-R9' specifies the 5N1 device. Notice how I save the stderr output to a file? That helps me understand how many errors I'm getting and could help me fix a problem. Right now I'm also using the -D for debug option as well to get more information. Try '--help' on rtl_433 to see the other options it has.
I owe this latest version to Pcjunky (yes, I know his name, but I ain't gonna post it without his permission). He took my code and beat on it until it worked to his satisfaction. Thank's a lot.
Now, I want to show you the parity routine I ran across. Here's the code:
static int acuriteParity(uint8_t v){
// returns 1 if parity odd, 0 if parity even
v ^= v >> 4;
v &= 0xf;
return (0x6996 >> v) & 1;
}
This calculates the parity for a byte in around five cpu operations. It's an extremely clever implementation that I leave the explanation as an exercise for the student.
Have fun.
Just when you thought I'd finally shut up about the AcuRite weather sensor I decide to post again. I promise, at some point I'll actually be satisfied with this device, but not quite yet. Over the months I've been working with this I've received comments and emails from folk that tried it and discovered things. One reader had an especially interesting situation; he had a neighbor that had a sensor within range. This meant that the code would pick them both up and present data from both sensors. To add insult to injury, when he tried changing his station to channel C, he found out that rtl_433 had a bug that prevented his receiving data on that channel. He must have been really annoyed at that point.
Since wind and temperature matter to placement, this was causing him grief and he set off to fix it. He isolated the sensor ID, and while he was at it, decoded the battery level, channel and found a better way to read the RF that allowed channel C to work.
During his trials he also found that the checksum was allowing bad data into the decoding and would present unusual readings. I've seen this over time as well. Things like a wind gust of 147 mph, a temperature spike of 30 or so degrees for no reason, the usual outlier readings, but with weather, it's hard to tell what an outlier really is.
I totally stole his ideas and some of his code and incorporated into my weather station,; I haven't been running it too long, but it looks like the vast majority of the outlier readings are gone. There may be some turn up, but so far it's been good.
First, inspection of the data showed that the bytes holding the various readings were all even parity. I added code to check the parity and discard the message if it didn't pass. Note that I only check the readings bytes because the first, second and last bytes are sensor, status, and checksum; these don't incorporate parity. In the c code I add the decoded sensor id, battery level, and channel to the JSON string I output, then check for the proper values in the python code that receives it. I did it this way because some folk may want to catch multiple sensors and average or compare them; especially if they can see a neighbors device for free.
I also noticed at my house that I receive a couple of devices from other folk that are not AcuRite devices. These things can cause problems as well, especially since they could easily be using the same checksum algorithm.
All the changes meant that I had to get the very latest version of rtl_433 from the original github repository. This was required because the rtl_433 folk made a new decoding method available that works better with the AcuRite data stream. That decoder wasn't there in January, but it is now. So, I tried again to fork it and failed miserably, and wound up just updating my copy of rtl_433. But, another good thing happened as well. The folk that wrote rtl_sdr, the actual radio software have improved it a lot and it installed first time, and I got to remove that from my copy.
Now, you get rtl_sdr, install it, build it, and then get my rtl_433, install it and build it. That will give you the version I'm using with all the changes to support the newest findings. My python code for the weather station I run is in GitHub as well. The specific stuff I use is:
rtl_sdr <link>
rtl_433 <link>
weather station <link>
The reason I linked to a particular commit in GitHub is so you can have exactly the same code to work with. The two rtl projects are in flux and could change dramatically at any time. You can always get the latest version if you think it will serve you better.
The latest version of rtl_433 not only has the new decoder software, it also allows you to specify the device you expect to receive. That makes it nice for me since I only have the AcuRite device I'm interested in. I changed my start up command to support this feature:
rtl_433 -f 433.915e6 -R9 2>> saveweather.log | saveweather.py >> saveweather.log 2>&1
The '-R9' specifies the 5N1 device. Notice how I save the stderr output to a file? That helps me understand how many errors I'm getting and could help me fix a problem. Right now I'm also using the -D for debug option as well to get more information. Try '--help' on rtl_433 to see the other options it has.
I owe this latest version to Pcjunky (yes, I know his name, but I ain't gonna post it without his permission). He took my code and beat on it until it worked to his satisfaction. Thank's a lot.
Now, I want to show you the parity routine I ran across. Here's the code:
static int acuriteParity(uint8_t v){
// returns 1 if parity odd, 0 if parity even
v ^= v >> 4;
v &= 0xf;
return (0x6996 >> v) & 1;
}
This calculates the parity for a byte in around five cpu operations. It's an extremely clever implementation that I leave the explanation as an exercise for the student.
Have fun.
Saturday, May 9, 2015
Hacking Into The Iris Door Sensor
Several people have asked me about the Iris Door Sensor, and I haven't had much interest. My battery operated switch can do this and is totally programmable, but I figured, "What the heck." I went ahead and got one when I went to Lowe's to get another smart switch.
Now, this is one of those projects that will go on and on as I try to get it to work. Meanwhile, these are the first steps.
This is what one looks like:
I don't think Lowe's will mind that I stole the picture from their site. First thing I did was open it up and look inside:
The spot on the left is for the battery and there's a switch for joining it to a network. I tried to get it to join with my Iris network, but it didn't work, so being me, I took it apart. Here's the circuit board removed from the casing:
Not much on this side, so flipping it over:
This is a side view of the magnet it comes with:
Remember those little extrusions on the magnet case; I'll talk about them later. I got out the magnifying glass and took a look at the active components, the list isn't very long:
Battery - CR2 Lithium, 3V
Magnetic Reed Switch - Hamlin 59050-1-S-00-0 Normally Open
Magnet off center to be near the reed switch.
Antenna - Antenova a6150
Processor - Silicon Labs Ember 250
Temperature Sensor - Microchip MCP9801M
Nice set of high quality components. this thing is put together pretty well. The important part is the Ember 250 SOC (system on a chip) that runs the device. These are made specifically for ZigBee and are very nice devices; to bad the product comes from Alertme. They always ignore the ZigBee spec and go their own way in critical areas. This is going to make it tough to hack into.
I put together some code and turned off my normal network to keep down interference and gave it a try. I couldn't get it to join properly, but I was able to see it trying. Since this is an Alertme device, it has significant API differences from the ZigBee spec, so I started trying things. I got it to join eventually by a non-reproducible accident and saw it running.
The problem is that I can't get that to happen again. When it was working, I noticed that it was really sensitive to where the magnet was, so I looked at the reed switch specs and it's only got a 7 mm range. That means that the magnet needs to be right next to the sensor for it to close. The FAQ for the switch on the Lowe's site even talks about that. They blame it on polarity when it's really the fact that the magnet needs to be so close to the sensor; like polarity would matter on a reed switch. A neo magnet might make the range longer, but it wouldn't be as pretty.
Some of us may have problems with that on installation; seven millimeters is not very much, especially when it's about 4 mm from the switch to the case. Do you see why they have those little protusions on the case now? It gives some spacing and a little place for the magnet to rub on the sensor case when the door closes. Strange way to build it, but I'm not an engineer.
About how it works: When the battery is first inserted, it starts sending Device Announce messages and continues this until I answer it with an Active Endpoint Request (which it hasn't answered yet). When it gets the Active Endpoint, it sends a Match Descriptor request and I answer it with a Match Descriptor Response tailored to it. Then, it sends one of the odd Alertme messages to profile 0xC216 that I've seen before from Alertme devices.
This is the point where, with the smart switches, I send a set of canned responses back and the device would join. The door switch doesn't join, It will stay around for a while trying, but will eventually give up and move on to another channel to look for another controller.
The time I got it to join, it started sending messages to cluster 0x500, the security device cluster. While it was doing that I could actually see the switch change state because the values it sent back were changing based on magnet location. However, it was erratic and extremely slow at sensing the magnet. There was some LED flashing on the device that seemed to correspond to the position of the magnet, but I couldn't be sure.
I'm going to post the code I'm working with as it is right now, but it's not pretty. I hacked up a piece of code I developed to break into another ZigBee device and am making experimental changes to it trying to understand this door switch.
If you grab it, there's an 'if' statement up at the top where I get the packets from the XBee, I check the incoming source address and skip anything that isn't coming from the switch. I have a number of the Iris devices and they were messing up the experiment, so I just ignore them in the code. You'll have to change that to your own devices address.
#! /usr/bin/python
# Have fun
from xbee import ZigBee
import logging
import datetime
import time
import serial
import sys, traceback
import shlex
from struct import *
'''
Before we get started there's a piece of this that drove me nuts. Each message to a
Zigbee cluster has a length and a header. The length isn't talked about at all in the
Zigbee documentation (that I could find) and the header byte is drawn backwards to
everything I've ever dealt with. So, I redrew the header byte so I could understand
and use it:
7 6 5 4 3 2 1 0
X Disable Default Response 1 = return default message
X Direction 1 = server to client, 0 = client to server
X Manufacturer Specific
X Frame Type 1 = cluster specific, 0 = entire profile
So, to send a cluster command, set bit zero, and to get an attribute from a cluster
set bit 4. If you want to be sure you get a reply, set the default response. I haven't
needed the manufacturer specific bit yet.
'''
def printData(data):
print "********** Message Contents"
for key, value in data.iteritems():
if key == "id":
print key, value
else:
print key, "".join("%02x " % ord(b) for b in data[key])
print
def getAttributes(data, thisOne):
''' OK, now that I've listed the clusters, I'm going to see about
getting the attributes for one of them by sending a Discover
attributes command. This is not a ZDO command, it's a ZCL command.
ZDO = ZigBee device object - the actual device
ZCL = Zigbee device cluster - the collection of routines to control it.
Frame control field, bit field, first byte
bits 0-1, frame type field
00 entire profile
01 specific to a particular cluster
10-11 reserved (don't use)
Note, if you're sending commands, this should be 01
if you're reading attributes, it should be 00
bit 2, manufacturer specific, if this bit is set, include
the manufacturer code (below)
bit 3 direction, this determines if it is from a client to
server.
1 server to client
0 client to server
Note, server and client are specific to zigbee, not
the purpose of the machine, so think about this. For
example to turn an on/off switch on, you have to be the
server so this bit will be 01
bit 4 disable default response
bits 5-7 reserved (set to zero)
Manufacturer code, either 2 bytes or not there
Transaction sequence number, byte
Command identifier, byte
Frame payload, variable, the command bytes to do something
frame control bits = 0b00 (this means a BINARY 00)
manufacturer specific bit = 0, for normal, or one for manufacturer
So, the frame control will be 0000000
discover attributes command identifier = 0x0c
then a zero to indicate the first attribute to be returned
and a 0x0f to indicate the maximum number of attributes to
return.
'''
print "Sending Discover Attributes"
zb.send('tx_explicit',
dest_addr_long = data['source_addr_long'],
dest_addr = data['source_addr'],
src_endpoint = '\x00',
dest_endpoint = '\x01',
cluster = thisOne, # cluster I want to know about
profile = '\x01\x04', # home automation profile
# means: frame control 0, sequence number 0xaa, command 0c,
# start at 0x0000 for a length of 0x0f
data = '\x00' + '\xaa' + '\x0c'+ '\x00' + '\x00'+ '\x0f'
)
# this is a call back function. When a message
# comes in this function will get the data
def messageReceived(data):
try:
# This is the long address of my door switch device
# since I have several other devices and they are transmitting
# all the time, I'm excluding them and only allowing the
# door switch in
if data['source_addr_long'] != '\x00\x0d\x6f\x00\x03\xc2\x71\xcc':
return
print 'gotta packet',
#print data
if (data['id'] == 'rx_explicit'):
print "RF Explicit"
#printData(data)
clusterId = (ord(data['cluster'][0])*256) + ord(data['cluster'][1])
print 'Cluster ID:', hex(clusterId),
print "profile id:", repr(data['profile'])
if (data['profile']=='\x01\x04'): # Home Automation Profile
# This has to be handled differently than the general profile
# each response if from a cluster that is actually doing something
# so there are attributes and commands to think about.
#
# Since this is a ZCL message; which actually means this message is
# is supposed to use the ZigBee cluster library to actually do something
# like turn on a light or check to see if it's on, the command way down
# in the rf_data is important. So, the commands may be repeated in
# each cluster and do slightly different things
#
# I'm going to grab the cluster command out of the rf_data first so
# I don't have to code it into each cluster
#print "take this apart"
#print repr(data['rf_data'])
if (data['rf_data'][0] == '\x08'): # was it successful?
#should have a bit check to see if manufacturer data is here
cCommand = data['rf_data'][2]
print "Cluster command: ", hex(ord(cCommand))
else:
print "Cluster command failed"
return
# grab the payload data to make it easier to work with
payload = data['rf_data'][3:] #from index 3 on is the payload for the command
datatypes={'\x00':'no data',
'\x10':'boolean',
'\x18':'8 bit bitmap',
'\x20':'unsigned 8 bit integer',
'\x21':'unsigned 24 bit integer',
'\x30':'8 bit enumeration',
'\x42':'character string'}
print "Raw payload:",repr(payload)
# handle these first commands in a general way
if (cCommand == '\x0d'): # Discover Attributes
# This tells you all the attributes for a particular cluster
# and their datatypes
print "Discover attributes response"
if (payload[0] == '\x01'):
print "All attributes returned"
else:
print "Didn't get all the attributes on one try"
i = 1
if (len(payload) == 1): # no actual attributes returned
print "No attributes"
return
while (i < len(payload)-1):
print " Attribute = ", hex(ord(payload[i+1])) , hex(ord(payload[i])),
try:
print datatypes[payload[i+2]]
i += 3
except:
print "I don't have an entry for datatype:", hex(ord(payload[i+2]))
return
if (clusterId == 0x0000): # Under HA this is the 'Basic' Cluster
pass
elif (clusterId == 0x0003): # 'identify' should make it flash a light or something
pass
elif (clusterId == 0x0004): # 'Groups'
pass
elif (clusterId == 0x0005): # 'Scenes'
pass
elif (clusterId == 0x0006): # 'On/Off' this is for switching or checking on and off
print "inside cluster 6"
elif (clusterId == 0x0008): # 'Level'
pass
else:
print("Haven't implemented this yet")
elif (data['profile']=='\x00\x00'): # The General Profile
if (clusterId == 0x0000):
print ("Network (16-bit) Address Request")
#printData(data)
elif (clusterId == 0x0004):
# Simple Descriptor Request,
print("Simple Descriptor Request")
#printData(data)
elif (clusterId == 0x0008):
# I couldn't find a definition for this
print("This was probably sent to the wrong profile")
elif (clusterId == 0x0013):
# This is the device announce message.
print 'Device Announce Message'
printData(data)
print "sending Active Endpoint Request "
zb.send('tx_explicit',
dest_addr_long = data['source_addr_long'],
dest_addr = data['source_addr'],
src_endpoint = '\x00',
dest_endpoint = '\x00',
cluster = '\x05\x00',
profile = '\xc2\x16',
# The first item is a number to identify the message
# The next two are the short address of the device
data = '\x12' + data['source_addr'][1]+ data['source_addr'][0]
)
elif (clusterId == 0x8000):
print("Network (16-bit) Address Response")
#printData(data)
elif (clusterId == 0x8038):
print("Management Network Update Request");
elif (clusterId == 0x8005):
# this is the Active Endpoint Response This message tells you
# what the device can do
print 'Active Endpoint Response'
printData(data)
if (ord(data['rf_data'][1]) == 0): # this means success
print "Active Endpoint reported back is: {0:02x}".format(ord(data['rf_data'][5]))
print("Now trying simple descriptor request on an endpoint")
zb.send('tx_explicit',
dest_addr_long = data['source_addr_long'],
dest_addr = data['source_addr'],
src_endpoint = '\x00',
dest_endpoint = '\x00', # This has to go to endpoint 0 !
cluster = '\x05\x00', #simple descriptor request'
profile = '\xc2\x16',
data = '\x13' + data['source_addr'][1] + data['source_addr'][0] + '\x01'
)
elif (clusterId == 0x8004):
print "simple descriptor response"
try:
clustersFound = []
r = data['rf_data']
if (ord(r[1]) == 0): # means success
#take apart the simple descriptor returned
endpoint, profileId, deviceId, version, inCount = \
unpack('<BHHBB',r[5:12])
print " endpoint reported is: {0:02x}".format(endpoint)
print " profile id: {0:04x}".format(profileId)
print " device id: {0:04x}".format(deviceId)
print " device version: {0:02x}".format(version)
print " input cluster count: {0:02x}".format(inCount)
position = 12
# input cluster list (16 bit words)
for x in range (0,inCount):
thisOne, = unpack("<H",r[position : position+2])
clustersFound.append(r[position+1] + r[position])
position += 2
print " input cluster {0:04x}".format(thisOne)
outCount, = unpack("<B",r[position])
position += 1
print " output cluster count: {0:02x}".format(outCount)
#output cluster list (16 bit words)
for x in range (0,outCount):
thisOne, = unpack("<H",r[position : position+2])
clustersFound.append(r[position+1] + r[position])
position += 2
print " output cluster {0:04x}".format(thisOne)
clustersFound.append('\x0b\x04')
print "added special cluster"
print "Completed Cluster List"
except:
print "error parsing Simple Descriptor"
printData(data)
print repr(clustersFound)
for c in clustersFound:
getAttributes(data, c) # Now, go get the attribute list for the cluster
elif (clusterId == 0x0006):
print "Match Descriptor Request"
# Match Descriptor Request
printData(data)
# Now the Match Descriptor Response
print "Sending match descriptor response"
zb.send('tx_explicit',
dest_addr_long = data['source_addr_long'],
dest_addr = data['source_addr'],
src_endpoint = '\x00',
dest_endpoint = '\x00',
cluster = '\x80\x06',
profile = '\x00\x00',
#seq #, status, address,num endp, list
data = '\x00\x00\x00\x00\x01\x02')
else:
print ("Unimplemented Cluster ID", hex(clusterId))
print
elif (data['profile']=='\xc2\x16'): # Alertme Specific
printData(data)
'''print "Sending weird messages"
if (clusterId == 0x00f6):
payload3 = '\x11\x01\x01'
zb.send('tx_explicit',
dest_addr_long = data['source_addr_long'],
dest_addr = data['source_addr'],
src_endpoint = '\x00',
dest_endpoint = data['source_endpoint'],
cluster = data['cluster'],
profile = '\xc2\x16',
data = payload3
)
payload4 = '\x19\x01\xfa\x00\x01'
zb.send('tx_explicit',
dest_addr_long = data['source_addr_long'],
dest_addr = data['source_addr'],
src_endpoint = '\x00',
dest_endpoint = data['source_endpoint'],
cluster = data['cluster'],
profile = '\xc2\x16',
data = payload4
)'''
if (clusterId == 0x00ef):
pass
elif (clusterId == 0x00f0):
pass
else:
print ("Unimplemented Profile ID")
elif(data['id'] == 'route_record_indicator'):
print("Route Record Indicator")
else:
print("some other type of packet")
print(data)
except:
print "I didn't expect this error:", sys.exc_info()[0]
traceback.print_exc()
def sendSwitch(whereLong, whereShort, srcEndpoint, destEndpoint,
clusterId, profileId, clusterCmd, databytes):
payload = '\x11\x00' + clusterCmd + databytes
# print 'payload',
# for c in payload:
# print hex(ord(c)),
# print
# print 'long address:',
# for c in whereLong:
# print hex(ord(c)),
# print
zb.send('tx_explicit',
dest_addr_long = whereLong,
dest_addr = whereShort,
src_endpoint = srcEndpoint,
dest_endpoint = destEndpoint,
cluster = clusterId,
profile = profileId,
data = payload
)
#------------ XBee Stuff -------------------------
# this is the /dev/serial/by-id device for the USB card that holds the XBee
ZIGBEEPORT = "/dev/serial/by-id/usb-FTDI_FT232R_USB_UART_A901QL3F-if00-port0"
ZIGBEEBAUD_RATE = 9600
# Open serial port for use by the XBee
ser = serial.Serial(ZIGBEEPORT, ZIGBEEBAUD_RATE)
# The XBee addresses I'm dealing with
BROADCAST = '\x00\x00\x00\x00\x00\x00\xff\xff'
theSwitch = '\x00\x0d\x6f\x00\x03\x58\x05\xc2'
UNKNOWN = '\xff\xfe' # This is the 'I don't know' 16 bit address
#-------------------------------------------------
logging.basicConfig()
# Create XBee library API object, which spawns a new thread
zb = ZigBee(ser, callback=messageReceived)
print ("started")
notYet = True;
firstTime = True;
while True:
try:
time.sleep(5)
if (firstTime):
# sendSwitch(whereLong=theSwitch, whereShort=UNKNOWN, srcEndpoint='\x00',
# destEndpoint='\x00', clusterId='\x00\x00', profileId='\x00\x00',
# clusterCmd='\x00', databytes='\x00')
# print "sending Active Endpoint Request "
# zb.send('tx_explicit',
# dest_addr_long = theSwitch,
# dest_addr = '\x94\x65',
# src_endpoint = '\x00',
# dest_endpoint = '\x00',
# cluster = '\x00\x05',
# profile = '\x00\x00',
# # The first item is a number to identify the message
# # The next two are the short address of the device
# data = '\x12' + '\x65' + '\x94'
# )
# time.sleep(10)
# print "sending 'configure reporting'"
# zb.send('tx_explicit',
# dest_addr_long = theSwitch,
# dest_addr = '\x94\x65',
# src_endpoint = '\x00',
# dest_endpoint = '\x01',
# cluster = '\x00\x06', # cluster I want to deal with
# profile = '\x01\x04', # home automation profile
# data = '\x00' + '\xaa' + '\x06' + '\x00' + '\x00' + '\x00' + '\x10' + '\x00' + '\x00' + '\x00' + '\x40' + '\x00' + '\x00'
# )
firstTime = False
print ("tick")
sys.stdout.flush() # if you're running non interactive, do this
except KeyboardInterrupt:
print ("Keyboard interrupt")
break
except:
print ("I didn't expect this error:", sys.exc_info()[0])
traceback.print_exc()
break
print ("After the while loop")
# halt() must be called before closing the serial
# port in order to ensure proper thread shutdown
zb.halt()
ser.close()
I'm continuing to play with this switch as time, and new ideas permit, so I think the door switch can be conquered. However, if is as flaky as the initial results would indicate, I may not be using it. Having to have the magnet right up against the sensor and having so little elbow room would stop the switch from being useful on things like garage doors, doors that move due to weather or settling, wooden windows that may not shut all the way, that kind of thing. If someone has one that works well in an Iris setup can tell us about this switch, it would be great.
If you want, grab the code, experiment, play, and let me know the results you get.
The next post on this project is here <link>.
Subscribe to:
Posts (Atom)










