Saturday, May 23, 2015

Hacking Into The Iris Door Sensor, Part 2

The previous post on this project is here <link>

Yes, I've made some progress on this project, but first let me talk about why I'm bothering with this. A couple of folk have asked me why I hack into these devices in general; the answer is not an esoteric, "Because they're there." I enjoy the challenge, but mostly it's because I want to use them without a bunch of strings attached.

The Iris devices are fairly priced, but not if you have to subscribe to a service that cost $10+ a month for the rest of time. From dismantling them I can tell the construction is actually pretty well done even though I keep seeing complaints on the web about how they should be smaller, prettier or come in colors. I couldn't build one as nice for the price. Actually, I couldn't build one as nice for twice the price. The single disadvantage is that their usefulness is tied to a web site that is run by a corporation that can change their mind at any time about anything and leave me hanging. Been there, done that, want to avoid it in the future.

Enough philosophy, I'm going to talk about the Iris Key Fob a bit. When I got some reaction from the door switch, I wondered if the key fob would work in a similar way, so since I was going to town anyway, I stopped off at Lowe's and picked up another door switch (I have two now) and a key fob as well.

Here it is with the cover off.

Same good construction as the door switch with a pair of switches that you press to activate funtions remotely. Let's pull it apart.

This is the case with the board removed to show how the plastic that operates the buttons is done. This makes for a nice feel when you're pushing the switch. The only problem I saw is that it's easy to press both switches if you squeeze it. A little care can overcome that problem though. Here's one side of the board,

and this is the other

You can click on the picture and zoom in if you want.

The switches have tactile feedback so you know you pressed it and the battery seats firmly so you know it's in there. Nice switch, albeit a bit large for some people's taste.

This switch was designed to control the Iris system in a home. If you push one button, it arms the alarm, and the other switch disarms it. It also provides feedback to the system that the fob is in range. People describe it as the Iris Hub sends messages to the fob every two minutes and if it doesn't get a response, the hub arms the alarm. My reading of the messages that fly back and forth shows that the fob reports every two minutes to the hub, so that would mean two minutes without a message and the hub will arm itself. There may be something I'm missing, but I'll turn it up eventually.

I see this device as two remote control switches that I can do any darn thing I want with. It could be on and off for a machine, light, ceiling fan, whatever I want. I can have one in each room attached to the bed for ceiling lights, or turn off the smoker in the yard if I want to.

I can hear my readers saying, "Fine, nice pictures and philosophy; tell us how the interaction works and give us some code." Well, I'm going to make you wait a couple of days before I bury you in code because this gets somewhat complicated and I'm not sure I've got enough bugs out to trust it yet. I have two door switches, a key fob and five smart switches running on the same XBee controller quite reliably, but a reader who 'volunteered' to try the code also has had nothing but problems.

Where my stuff joins and just works, his decided to get flaky and mess with his mind. We're trading logs and code back and forth to try and figure out what's different.

More later, but you got a look at the internals of a device, so stop complaining.

Monday, May 11, 2015

Yet Another Update to the AcuRite 5N1 Weather Sensor

The previous post on this project is here <link>

Just when you thought I'd finally shut up about the AcuRite weather sensor I decide to post again. I promise, at some point I'll actually be satisfied with this device, but not quite yet. Over the months I've been working with this I've received comments and emails from folk that tried it and discovered things. One reader had an especially interesting situation; he had a neighbor that had a sensor within range. This meant that the code would pick them both up and present data from both sensors. To add insult to injury, when he tried changing his station to channel C, he found out that rtl_433 had a bug that prevented his receiving data on that channel. He must have been really annoyed at that point.

Since wind and temperature matter to placement, this was causing him grief and he set off to fix it. He isolated the sensor ID, and while he was at it, decoded the battery level, channel and found a better way to read the RF that allowed channel C to work.

During his trials he also found that the checksum was allowing bad data into the decoding and would present unusual readings. I've seen this over time as well. Things like a wind gust of 147 mph, a temperature spike of 30 or so degrees for no reason, the usual outlier readings, but with weather, it's hard to tell what an outlier really is.

I totally stole his ideas and some of his code and incorporated into my weather station,; I haven't been running it too long, but it looks like the vast majority of the outlier readings are gone. There may be some turn up, but so far it's been good.

First, inspection of the data showed that the bytes holding the various readings were all even parity. I added code to check the parity and discard the message if it didn't pass. Note that I only check the readings bytes because the first, second and last bytes are sensor, status, and checksum; these don't incorporate parity. In the c code I add the decoded sensor id, battery level, and channel to the JSON string I output, then check for the proper values in the python code that receives it. I did it this way because some folk may want to catch multiple sensors and average or compare them; especially if they can see a neighbors device for free.

I also noticed at my house that I receive a couple of devices from other folk that are not AcuRite devices. These things can cause problems as well, especially since they could easily be using the same checksum algorithm.

All the changes meant that I had to get the very latest version of rtl_433 from the original github repository. This was required because the rtl_433 folk made a new decoding method available that works better with the AcuRite data stream. That decoder wasn't there in January, but it is now. So, I tried again to fork it and failed miserably, and wound up just updating my copy of rtl_433. But, another good thing happened as well. The folk that wrote rtl_sdr, the actual radio software have improved it a lot and it installed first time, and I got to remove that from my copy.

Now, you get rtl_sdr, install it, build it, and then get my rtl_433, install it and build it. That will give you the version I'm using with all the changes to support the newest findings. My python code for the weather station I run is in GitHub as well. The specific stuff I use is:

rtl_sdr <link>
rtl_433 <link>
weather station <link>

The reason I linked to a particular commit in GitHub is so you can have exactly the same code to work with. The two rtl projects are in flux and could change dramatically at any time. You can always get the latest version if you think it will serve you better.

The latest version of rtl_433 not only has the new decoder software, it also allows you to specify the device you expect to receive. That makes it nice for me since I only have the AcuRite device I'm interested in. I changed my start up command to support this feature:

rtl_433 -f 433.915e6 -R9 2>> saveweather.log |  >> saveweather.log 2>&1

The '-R9' specifies the 5N1 device. Notice how I save the stderr output to a file? That helps me understand how many errors I'm getting and could help me fix a problem. Right now I'm also using the -D for debug option as well to get more information. Try '--help' on rtl_433 to see the other options it has.

I owe this latest version to Pcjunky (yes, I know his name, but I ain't gonna post it without his permission). He took my code and beat on it until it worked to his satisfaction. Thank's a lot.

Now, I want to show you the parity routine I ran across. Here's the code:

static int acuriteParity(uint8_t v){
    // returns 1 if parity odd, 0 if parity even
    v ^= v >> 4;
    v &= 0xf;
    return (0x6996 >> v) & 1;

This calculates the parity for a byte in around five cpu operations. It's an extremely clever implementation that I leave the explanation as an exercise for the student.

Have fun.

Saturday, May 9, 2015

Hacking Into The Iris Door Sensor

Several people have asked me about the Iris Door Sensor, and I haven't had much interest. My battery operated switch can do this and is totally programmable, but I figured, "What the heck." I went ahead and got one when I went to Lowe's to get another smart switch.

Now, this is one of those projects that will go on and on as I try to get it to work. Meanwhile, these are the first steps.

This is what one looks like:

I don't think Lowe's will mind that I stole the picture from their site. First thing I did was open it up and look inside:

The spot on the left is for the battery and there's a switch for joining it to a network. I tried to get it to join with my Iris network, but it didn't work, so being me, I took it apart. Here's the circuit board removed from the casing:

Not much on this side, so flipping it over:

This is a side view of the magnet it comes with:

Remember those little extrusions on the magnet case; I'll talk about them later. I got out the magnifying glass and took a look at the active components, the list isn't very long:

Battery - CR2 Lithium, 3V
Magnetic Reed Switch - Hamlin 59050-1-S-00-0 Normally Open
Magnet off center to be near the reed switch.
Antenna - Antenova a6150
Processor - Silicon Labs Ember 250
Temperature Sensor - Microchip MCP9801M

Nice set of high quality components. this thing is put together pretty well. The important part is the Ember 250 SOC (system on a chip) that runs the device. These are made specifically for ZigBee and are very nice devices; to bad the product comes from Alertme. They always ignore the ZigBee spec and go their own way in critical areas. This is going to make it tough to hack into.

I put together some code and turned off my normal network to keep down interference and gave it a try. I couldn't get it to join properly, but I was able to see it trying. Since this is an Alertme device, it has significant API differences from the ZigBee spec, so I started trying things. I got it to join eventually by a non-reproducible accident and saw it running.

The problem is that I can't get that to happen again. When it was working, I noticed that it was really sensitive to where the magnet was, so I looked at the reed switch specs and it's only got a 7 mm range. That means that the magnet needs to be right next to the sensor for it to close. The FAQ for the switch on the Lowe's site even talks about that. They blame it on polarity when it's really the fact that the magnet needs to be so close to the sensor; like polarity would matter on a reed switch. A neo magnet might make the range longer, but it wouldn't be as pretty.

Some of us may have problems with that on installation; seven millimeters is not very much, especially when it's about 4 mm from the switch to the case. Do you see why they have those little protusions on the case now? It gives some spacing and a little place for the magnet to rub on the sensor case when the door closes. Strange way to build it, but I'm not an engineer.

About how it works: When the battery is first inserted, it starts sending Device Announce messages and continues this until I answer it with an Active Endpoint  Request (which it hasn't answered yet). When it gets the Active Endpoint, it sends a Match Descriptor request and I answer it with a Match Descriptor Response tailored to it. Then, it sends one of the odd Alertme messages to profile 0xC216 that I've seen before from Alertme devices.

This is the point where, with the smart switches, I send a set of canned responses back and the device would join. The door switch doesn't join, It will stay around for a while trying, but will eventually give up and move on to another channel to look for another controller.

The time I got it to join, it started sending messages to cluster 0x500, the security device cluster. While it was doing that I could actually see the switch change state because the values it sent back were changing based on magnet location. However, it was erratic and extremely slow at sensing the magnet. There was some LED flashing on the device that seemed to correspond to the position of the magnet, but I couldn't be sure.

I'm going to post the code I'm working with as it is right now, but it's not pretty. I hacked up a piece of code I developed to break into another ZigBee device and am making experimental changes to it trying to understand this door switch.

If you grab it, there's an 'if' statement up at the top where I get the packets from the XBee, I check the incoming source address and skip anything that isn't coming from the switch. I have a number of the Iris devices and they were messing up the experiment, so I just ignore them in the code. You'll have to change that to your own devices address.

#! /usr/bin/python
# Have fun

from xbee import ZigBee 
import logging
import datetime
import time
import serial
import sys, traceback
import shlex
from struct import *
Before we get started there's a piece of this that drove me nuts.  Each message to a 
Zigbee cluster has a length and a header.  The length isn't talked about at all in the 
Zigbee documentation (that I could find) and the header byte is drawn backwards to
everything I've ever dealt with.  So, I redrew the header byte so I could understand
and use it:

7   6   5   4   3   2   1   0
            X                   Disable Default Response 1 = return default message
                X               Direction 1 = server to client, 0 = client to server
                    X           Manufacturer Specific 
                            X   Frame Type 1 = cluster specific, 0 = entire profile
So, to send a cluster command, set bit zero, and to get an attribute from a cluster
set bit 4.  If you want to be sure you get a reply, set the default response.  I haven't
needed the manufacturer specific bit yet.


def printData(data):
    print "********** Message Contents"
    for key, value in data.iteritems():
        if key == "id":
            print key, value
            print key, "".join("%02x " % ord(b) for b in data[key])

def getAttributes(data, thisOne):
    ''' OK, now that I've listed the clusters, I'm going to see about 
    getting the attributes for one of them by sending a Discover
    attributes command.  This is not a ZDO command, it's a ZCL command.
    ZDO = ZigBee device object - the actual device
    ZCL = Zigbee device cluster - the collection of routines to control it.
        Frame control field, bit field,  first byte
            bits 0-1, frame type field
                00 entire profile
                01 specific to a particular cluster
                10-11 reserved (don't use)
                Note, if you're sending commands, this should be 01
                if you're reading attributes, it should be 00
            bit 2, manufacturer specific, if this bit is set, include
                   the manufacturer code (below)
            bit 3 direction, this determines if it is from a client to 
                 1 server to client
                 0 client to server
                 Note, server and client are specific to zigbee, not
                 the purpose of the machine, so think about this.  For
                 example to turn an on/off switch on, you have to be the
                 server so this bit will be 01
            bit 4 disable default response
            bits 5-7 reserved (set to zero)
        Manufacturer code,  either 2 bytes or not there 
        Transaction sequence number, byte
        Command identifier, byte
        Frame payload,  variable, the command bytes to do something
        frame control bits = 0b00 (this means a BINARY 00)
        manufacturer specific bit = 0, for normal, or one for manufacturer
        So, the frame control will be 0000000
        discover attributes command identifier = 0x0c
        then a zero to indicate the first attribute to be returned
        and a 0x0f to indicate the maximum number of attributes to 
    print "Sending Discover Attributes"
        dest_addr_long = data['source_addr_long'],
        dest_addr = data['source_addr'],
        src_endpoint = '\x00',
        dest_endpoint = '\x01',
        cluster = thisOne, # cluster I want to know about
        profile = '\x01\x04', # home automation profile
        # means: frame control 0, sequence number 0xaa, command 0c,
        # start at 0x0000 for a length of 0x0f
        data = '\x00' + '\xaa' + '\x0c'+ '\x00' + '\x00'+ '\x0f'

# this is a call back function.  When a message
# comes in this function will get the data
def messageReceived(data):
        # This is the long address of my door switch device
        # since I have several other devices and they are transmitting
        # all the time, I'm excluding them and only allowing the
        # door switch in
        if data['source_addr_long'] != '\x00\x0d\x6f\x00\x03\xc2\x71\xcc':
        print 'gotta packet',
        #print data
        if (data['id'] == 'rx_explicit'):
            print "RF Explicit"
            clusterId = (ord(data['cluster'][0])*256) + ord(data['cluster'][1])
            print 'Cluster ID:', hex(clusterId),
            print "profile id:", repr(data['profile'])
            if (data['profile']=='\x01\x04'): # Home Automation Profile
                # This has to be handled differently than the general profile
                # each response if from a cluster that is actually doing something
                # so there are attributes and commands to think about.
                # Since this is a ZCL message; which actually means this message is 
                # is supposed to use the ZigBee cluster library to actually do something
                # like turn on a light or check to see if it's on, the command way down
                # in the rf_data is important.  So, the commands may be repeated in
                # each cluster and do slightly different things
                # I'm going to grab the cluster command out of the rf_data first so 
                # I don't have to code it into each cluster
                #print "take this apart"
                #print repr(data['rf_data'])
                if (data['rf_data'][0] == '\x08'): # was it successful?
                    #should have a bit check to see if manufacturer data is here
                    cCommand = data['rf_data'][2]
                    print "Cluster command: ", hex(ord(cCommand))
                    print "Cluster command failed"
                # grab the payload data to make it easier to work with
                payload = data['rf_data'][3:] #from index 3 on is the payload for the command
                datatypes={'\x00':'no data',
                            '\x18':'8 bit bitmap',
                            '\x20':'unsigned 8 bit integer',
                            '\x21':'unsigned 24 bit integer',
                            '\x30':'8 bit enumeration',
                            '\x42':'character string'}
                print "Raw payload:",repr(payload)
                # handle these first commands in a general way
                if (cCommand == '\x0d'): # Discover Attributes
                    # This tells you all the attributes for a particular cluster
                    # and their datatypes
                    print "Discover attributes response"
                    if (payload[0] == '\x01'):
                        print "All attributes returned"
                        print "Didn't get all the attributes on one try"
                    i = 1
                    if (len(payload) == 1): # no actual attributes returned
                        print "No attributes"
                    while (i < len(payload)-1):
                        print "    Attribute = ", hex(ord(payload[i+1])) , hex(ord(payload[i])),
                            print datatypes[payload[i+2]]
                            i += 3
                            print "I don't have an entry for datatype:", hex(ord(payload[i+2]))
                if (clusterId == 0x0000): # Under HA this is the 'Basic' Cluster
                elif (clusterId == 0x0003): # 'identify' should make it flash a light or something 
                elif (clusterId == 0x0004): # 'Groups'
                elif (clusterId == 0x0005): # 'Scenes'  
                elif (clusterId == 0x0006): # 'On/Off' this is for switching or checking on and off  
                    print "inside cluster 6"
                elif (clusterId == 0x0008): # 'Level'  
                    print("Haven't implemented this yet")
            elif (data['profile']=='\x00\x00'): # The General Profile
                if (clusterId == 0x0000):
                    print ("Network (16-bit) Address Request")
                elif (clusterId == 0x0004):
                    # Simple Descriptor Request, 
                    print("Simple Descriptor Request")
                elif (clusterId == 0x0008):
                    # I couldn't find a definition for this 
                    print("This was probably sent to the wrong profile")
                elif (clusterId == 0x0013):
                    # This is the device announce message.
                    print 'Device Announce Message'
                    print "sending Active Endpoint Request "
                        dest_addr_long = data['source_addr_long'],
                        dest_addr = data['source_addr'],
                        src_endpoint = '\x00',
                        dest_endpoint = '\x00',
                        cluster = '\x05\x00',
                        profile = '\xc2\x16',
                        # The first item is a number to identify the message
                        # The next two are the short address of the device
                        data = '\x12' + data['source_addr'][1]+ data['source_addr'][0]

                elif (clusterId == 0x8000):
                    print("Network (16-bit) Address Response")
                elif (clusterId == 0x8038):
                    print("Management Network Update Request");
                elif (clusterId == 0x8005):
                    # this is the Active Endpoint Response This message tells you
                    # what the device can do
                    print 'Active Endpoint Response'
                    if (ord(data['rf_data'][1]) == 0): # this means success
                        print "Active Endpoint reported back is: {0:02x}".format(ord(data['rf_data'][5]))
                    print("Now trying simple descriptor request on an endpoint")
                        dest_addr_long = data['source_addr_long'],
                        dest_addr = data['source_addr'],
                        src_endpoint = '\x00',
                        dest_endpoint = '\x00', # This has to go to endpoint 0 !
                        cluster = '\x05\x00', #simple descriptor request'
                        profile = '\xc2\x16',
                        data = '\x13' + data['source_addr'][1] + data['source_addr'][0] + '\x01'
                elif (clusterId == 0x8004):
                    print "simple descriptor response"
                        clustersFound = []
                        r = data['rf_data']
                        if (ord(r[1]) == 0): # means success
                            #take apart the simple descriptor returned
                            endpoint, profileId, deviceId, version, inCount = \
                            print "    endpoint reported is: {0:02x}".format(endpoint)
                            print "    profile id:  {0:04x}".format(profileId)
                            print "    device id: {0:04x}".format(deviceId)
                            print "    device version: {0:02x}".format(version)
                            print "    input cluster count: {0:02x}".format(inCount)
                            position = 12
                            # input cluster list (16 bit words)
                            for x in range (0,inCount):
                                thisOne, = unpack("<H",r[position : position+2])
                                clustersFound.append(r[position+1] + r[position])
                                position += 2
                                print "        input cluster {0:04x}".format(thisOne)
                            outCount, = unpack("<B",r[position])
                            position += 1
                            print "    output cluster count: {0:02x}".format(outCount)
                            #output cluster list (16 bit words)
                            for x in range (0,outCount):
                                thisOne, = unpack("<H",r[position : position+2])
                                clustersFound.append(r[position+1] + r[position])
                                position += 2
                                print "        output cluster {0:04x}".format(thisOne)
                            print "added special cluster"
                            print "Completed Cluster List"
                        print "error parsing Simple Descriptor"
                    print repr(clustersFound)
                    for c in clustersFound:
                        getAttributes(data, c) # Now, go get the attribute list for the cluster
                elif (clusterId == 0x0006):
                    print "Match Descriptor Request"
                    # Match Descriptor Request
                    # Now the Match Descriptor Response
                    print "Sending match descriptor response"
                        dest_addr_long = data['source_addr_long'],
                        dest_addr = data['source_addr'],
                        src_endpoint = '\x00',
                        dest_endpoint = '\x00',
                        cluster = '\x80\x06',
                        profile = '\x00\x00',
                        #seq #, status, address,num endp, list
                        data = '\x00\x00\x00\x00\x01\x02')
                    print ("Unimplemented Cluster ID", hex(clusterId))
            elif (data['profile']=='\xc2\x16'): # Alertme Specific
                '''print "Sending weird messages"
                if (clusterId == 0x00f6):
                    payload3 = '\x11\x01\x01'
                        dest_addr_long = data['source_addr_long'],
                        dest_addr = data['source_addr'],
                        src_endpoint = '\x00',
                        dest_endpoint = data['source_endpoint'],
                        cluster = data['cluster'],
                        profile = '\xc2\x16',
                        data = payload3
                    payload4 = '\x19\x01\xfa\x00\x01'
                       dest_addr_long = data['source_addr_long'],
                       dest_addr = data['source_addr'],
                       src_endpoint = '\x00',
                       dest_endpoint = data['source_endpoint'],
                       cluster = data['cluster'],
                       profile = '\xc2\x16',
                       data = payload4
                if (clusterId == 0x00ef):
                elif (clusterId == 0x00f0):
                print ("Unimplemented Profile ID")
        elif(data['id'] == 'route_record_indicator'):
            print("Route Record Indicator")
            print("some other type of packet")
        print "I didn't expect this error:", sys.exc_info()[0]
def sendSwitch(whereLong, whereShort, srcEndpoint, destEndpoint, 
                clusterId, profileId, clusterCmd, databytes):
    payload = '\x11\x00' + clusterCmd + databytes
    # print 'payload',
    # for c in payload:
        # print hex(ord(c)),
    # print
    # print 'long address:',
    # for c in whereLong:
        # print hex(ord(c)),
    # print
        dest_addr_long = whereLong,
        dest_addr = whereShort,
        src_endpoint = srcEndpoint,
        dest_endpoint = destEndpoint,
        cluster = clusterId,
        profile = profileId,
        data = payload

#------------ XBee Stuff -------------------------
# this is the /dev/serial/by-id device for the USB card that holds the XBee
ZIGBEEPORT = "/dev/serial/by-id/usb-FTDI_FT232R_USB_UART_A901QL3F-if00-port0"
# Open serial port for use by the XBee

# The XBee addresses I'm dealing with
BROADCAST = '\x00\x00\x00\x00\x00\x00\xff\xff'
theSwitch = '\x00\x0d\x6f\x00\x03\x58\x05\xc2'
UNKNOWN = '\xff\xfe' # This is the 'I don't know' 16 bit address


# Create XBee library API object, which spawns a new thread
zb = ZigBee(ser, callback=messageReceived)

print ("started")
notYet = True;
firstTime = True;
while True:
        if (firstTime):
#           sendSwitch(whereLong=theSwitch, whereShort=UNKNOWN, srcEndpoint='\x00', 
#                   destEndpoint='\x00', clusterId='\x00\x00', profileId='\x00\x00', 
#                   clusterCmd='\x00', databytes='\x00')
#            print "sending Active Endpoint Request "
#            zb.send('tx_explicit',
#                dest_addr_long = theSwitch,
#                dest_addr = '\x94\x65',
#                src_endpoint = '\x00',
#                dest_endpoint = '\x00',
#                cluster = '\x00\x05',
#                profile = '\x00\x00',
#                # The first item is a number to identify the message
#                # The next two are the short address of the device
#                data = '\x12' + '\x65' + '\x94'
#            )
#            time.sleep(10)
#            print "sending 'configure reporting'"
#            zb.send('tx_explicit',
#                dest_addr_long = theSwitch,
#                dest_addr = '\x94\x65',
#                src_endpoint = '\x00',
#                dest_endpoint = '\x01',
#                cluster = '\x00\x06', # cluster I want to deal with
#                profile = '\x01\x04', # home automation profile
#                data = '\x00' + '\xaa' + '\x06' + '\x00' + '\x00' + '\x00' + '\x10' + '\x00' + '\x00' + '\x00' + '\x40' + '\x00' + '\x00'
#            )

            firstTime = False
        print ("tick")
        sys.stdout.flush() # if you're running non interactive, do this

    except KeyboardInterrupt:
        print ("Keyboard interrupt")
        print ("I didn't expect this error:", sys.exc_info()[0])

print ("After the while loop")
# halt() must be called before closing the serial
# port in order to ensure proper thread shutdown

I told you it wasn't pretty, so stop complaining.

I'm continuing to play with this switch as time, and new ideas permit, so I think the door switch can be conquered. However, if is as flaky as the initial results would indicate, I may not be using it. Having to have the magnet right up against the sensor and having so little elbow room would stop the switch from being useful on things like garage doors, doors that move due to weather or settling, wooden windows that may not shut all the way, that kind of thing. If someone has one that works well in an Iris setup can tell us about this switch, it would be great.

If you want, grab the code, experiment, play, and let me know the results you get.

The next post on this project is here <link>.

Sunday, May 3, 2015

My New Network Attached Storage

Well, I did it. I finally broke down and bought a NAS (network attached Storage). I researched these things for days looking for something that would fit my needs and couldn't make up my mind. I was tempted to just take a Raspberry Pi and add a disk drive, but I wanted more capability in the device than that would have given me. I asked other folk, and it came down to two manufacturers of this kind of device: Synology and QNAP. Frankly, they both sound great and have more good reviews than bad.

I settled on QNAP, not because it won out in the specification, but because I caught a nice sale. TigerDirect had the QNAP TS-231 on sale for $30 bucks off, so I ordered it. I got two 3TB WD red drives from Amazon and set it up. It's up in my attic data center working its little heart out.

When I was looking at what could be run on it, MySQL was listed. That meant that I could put my house databases up there and get them off the Pi's I have running. It also meant that I could expand the database and actually log data locally for my house.

Wow, that would mean all my griping about cloud services would end.

First step, set up a MySQL database on the server; that took about 10 minutes. Then understand how to put data on it; that took a day. Then convert my code to use the new databases on the NAS; that took a stinking week. The very basic use of MySQL is different enough from the way to use SQLite3 that I had to learn a whole lot to get it working.

But, when I did, it really sped things up. I'm not clear on why exactly; I'm making data base transactions across an Ethernet link instead of locally, but it runs faster. Since the NAS is set up Raid 1, I have 3TB to play with and I expect the databases to use a hunk of that so it may slow down over time as the data bases grow.

My impetus for finally biting the bullet on one of these devices was storage failures on the Raspberry Pi. That's the most significant problem I've had with the Pi, the SD cards wear out, and I had worse luck using a USB stick. Well, I won't be wearing the cards out as fast, and I don't need the extra space of a USB stick anymore. However, the SD cards will still wear out because all the system logs and such are beating on them.

To this end, I'm going to put the operating system on the NAS as well, but I haven't gotten that far yet. The idea is to put everything I can on the NAS and only boot the Pi from the SD card. That should make it last a while. Sort of the best of both worlds; the separate processor and hardware of the Pi combined with the storage and reliability of a NAS. I'll inherit the possibility of things quitting if the Ethernet fails, but my house already relies on Ethernet anyway. I have the various Arduinos set up to work in a default mode, so that problem hasn't gotten any worse ... I think.

One of the side-effects (affects?? I can never remember which one to use) is that I cleaned up a bunch of code, removed things I really wasn't using any more and restructured a lot of my processes. So, most of the processes that wrote data to the various cloud services I looked into are gone now. The code is still out there on GitHub (the internet is forever), but it's not in the latest update. If you need it, just look at a previous commit.

Another thing that changed is since I'm not updating the various clouds, the live graphs I have scattered in earlier posts may not work any more. I replaced some of them with pictures to keep it somewhat coherent, but gave up after a few. Also, the architecture of the overall house control and monitoring is a bit different. I guess I'll delete the page on that and try to create a new picture and description soon.

I'm keeping my legacy feeds on Xively running for now since the majority of my data is there. I wouldn't mind losing that data, I'll make more, but I want to annoy them as long a possible with a personal feed. They've been hinting that they may remove the personal feeds for a while now, and I just plan on staying there until they kick me off.

But, this thing is so cool. It just works and accepts transactions without complaining at all. MySOL is way capable and can do things I hadn't even thought of.

So, Dave's place has Arduinos running things like the garage, pool and air conditioning that talk to a Raspberry Pi that saves it's status to a NAS. There's also a weather head that sends RF data to another Raspberry Pi that saves weather data cumulatively to the NAS as well.

The code for the new way of doing things is out on GitHub, if you want to grab something.

Unfortunately, this exercise has given me a bunch of ideas on things I can do to improve the operation. Now that I can essentially have unlimited Raspberry Pi devices doing things and coordinating activities of other devices, I can put the house configuration file on the NAS and read it from any Ethernet connected device, I can separate the web services from everything else and increase house Ethernet security. I can save data on each room using the new temperature sensor in various ways. I can ... I can go completely nuts,

If I haven't already.

Friday, April 24, 2015

Battery Operated Temperature Sensor: OK, Let's Build This Thing

The previous post in this series is here <link>.

Well, I've been testing, trying new ideas and even added a button to this thing for the last several weeks. I don't think I'm going to learn anything new in the short term, so I decided to solder everything down and turn it into a package that I can actually use.

I got some of these:

Those of you that have been following this project understand now why I wanted to confine the parts to half a breadboard. I can put all the parts on this and then have something I can put in some kind of enclosure. I stripped all the parts off the breadboard and put them on the new board.

Yep, I got the regulator, XBee, Arduino Mini Pro, sensor, and a push button on there. I mounted the active components in sockets so they'd be easy to change if I needed to. I also put a plug on the battery pack so I can unplug it if needed. I still have the monitor circuit for watching the battery level, that way I can tell when the batteries are starting to go flat.

It's a nice little package, even if I do have to say so myself. If I put the battery underneath the board, it is pretty compact as well:

I think I'll put it in a travel container for a bar of soap to play with it around the house for a few months. Here's the schematic for the final result:

I'm going to the 3V version of the MCP1700 as soon as they come in from Mouser. Those little devices are perfect for battery applications. The 3V version (opposed to the 3.3V) worked well with this combination of components because the battery drops to just under 2.9V before it shuts off. That way I've dropped the batteries down to a volt each, and there's essentially nothing left in there.

Notice the voltage divider on the battery input? That combination of values works well to monitor the battery voltage, but I had to calibrate it in the code. It's pretty easy, get your meter, measure the battery voltage and then adjust the code until you get the same reading ... done. This will hold essentially forever, but if you want to check it yearly or so, it's easy.

This particular device (there WILL be others) is going beside my bed. I'll have the temperature of my bedroom as well as a button to turn the light off eliminating the X10 controller I have there now. Remember, this is a smart device, I can program it to do other things over time, and probably will. I think having one in the guest room with a door sensor attached would be fun. I may put one in the attic to monitor the heat up there, and one hooked to a moisture sensor out in the garage would be great for keeping track of the water heater.

How long will the batteries last? I don't know exactly, but it should be many months, a little less than a year. I can't measure the power usage, it's too low for anything I have. I removed the led from the Arduino board and feed the power behind the power regulator, so I have no losses there. I suspect it's using around 10 micro amps when asleep and not much when it's running. I'm going to set it to 5 seconds awake and 115 seconds asleep which will cut the drain in half compared to the level I ran it at in testing. If it runs down too soon, there's still several things I can do, but they won't actually gain me very much. Slower clock speed, mosfet switch for the power to the sensor and XBee, that kind of thing. The one thing that will really increase the time between battery changes is to simply use four batteries. The regulator I used can take it and it's not much more space. I'll try the three battery setup for as long as it runs to get a feel for what's going on.

My new NAS came in and is installed, so I guess I'll move on to something on it now. I'll build more of these as time permits, I have enough XBees for a bunch.

Friday, April 17, 2015

Battery Powered Temperature Sensor: Remote Control Added

The previous post in this series is here <link>.

So, I crawled into bed the other night and reached over to turn off the light and it didn't work. Poked the X10 control button several times and nothing. Crap! I had to get out of bed, go to another room and turn off the light from there. The next morning I replaced the button battery in the control and everything was fine again.

But this got me to thinking; this is the only X10 device I have left from years ago and it's time to think about doing something different. X10 failed completely in my house except for the room farthest away from everything else and it has been flaky for some time now. But, how would I implement a remote control that can just hang on the wall for months and get used maybe once a day?

Put a button on my battery operated temperature sensor! It's still in testing, but it appears to be working great, All I'd need to do is put a button on it and cause an XBee message to be sent to my house controller and then I can choose some switch device to put on the light circuit and have my button beside the bed. It would measure the temperature periodically and just wait for a button press to send the signal.

Sure, I could use my cell phone with the (brand spanking new) app I made for the house, A tablet, or a web browser. I could even call my neighbor and tell them to turn the light off for me, but I want a simple button on the nightstand that I can press without any hassle and turn the silly light off. But, how to do it without draining the battery?

Interrupts, yes, that's the ticket.

Remember from the previous (ton) of posts on this device, I put the device to sleep for 55 seconds at a time and wake it up for 5 seconds to transmit the battery level and temperature, all I should have to do is to hook a button to an interrupt pin and add some code to send a message to the house controller. At the controller I can add some code to do anything I have a controller for. I could open the garage door with it if I want to.

So, a few hours later after about 25 false starts, I got it working. I connected a little switch between pin 2 and ground of the Arduino board I'm using and it causes an interrupt which breaks the arduino out of sleep and then I just send the normal temperature message which now includes a new field to indicate that the button was pushed. At the house controller, I parse out the field, look at the value and do whatever I program in.

The reason it took several tries to get it working was because I was making it too hard. I thought I'd have to have a lot of special code to support the interrupt and proceeded down that path. It turned out the JeeLab routine I'm using for sleep already supports other interrupts, and all I had to do was check a variable and add a simple interrupt handler and everything worked ... well mostly.  Here's a code fragment showing how I sensed that the button had been pressed:

 if (millis() - savedmillis > AWAKETIME){
    Serial.print("was awake for ");
    Serial.println(millis() - savedmillis);
    delay(100); // delay to allow the characters to get out
    savedmillis = millis();
    digitalWrite(xbeeSleepReq, SLEEP); // put the XBee to sleep
    while (digitalRead(xbeeCTS) != SLEEP){} // Wait 'til it actually goes to sleep
    unsigned long timeSlept = 0;
    int result = 1;
    while (timeSlept < SLEEPTIME){
      attachInterrupt(0, buttonThing, LOW);
      result = Sleepy::loseSomeTime((unsigned int)7000);
      if (result == 0) // this is something other than a watchdog
      timeSlept = (millis() - savedmillis);
    Serial.print("was asleep for ");
    Serial.println(millis() - savedmillis);
    savedmillis = millis();
    if (result == 0)
      Serial.println("Woke up on a button press");
    digitalWrite(xbeeSleepReq, AWAKE); // wake that boy up now

I simply look at the return value from loseSomeTime() and if it's 0, then some interrupt other than the watchdog timer happened. I print a message for debugging and go to the XBee send code. The trick is in the interrupt handler:

// This is for the remote control buttons
void buttonThing(){
  buttonPressed = true;

The variable 'buttonPressed' is simply a global variable that I set to true and look at in the XBee send routine. If the variable is true, I set a new field in the mesage to say so, if it's false, I say 'nothing' in the field. I'll eventually work out what would be the best thing to put in the message, but that parts easy once you get it working. Notice that I attachInterrupt() for interrupt zero, which means button two on the Arduino, just before going to sleep, and detach it after the button is pressed. That prevents multiple interrupts from button bounce. The disadvantage to doing it this way is that it won't respond to a button press during the 5 seconds the board is awake. I don't currently consider this a problem, but that might change if I want to use it for something else.

So, now if I push the button, it gets an interrupt on pin 2, breaks out of sleep, sends the temperature and an indicator that the button was pushed to the house controller. The house controller records the temperature, then looks at the new field and does whatever I want.

The Xbee send code looks like this

void sendStatusXbee(){
  xbeeReadyWait(); // Make sure the XBee is ready

  char *command;
  if (buttonPressed)
    command = "toggle";
    command = "nothing";
  buttonPressed = false;
  sprintf(Dbuf, "{\"%s\":{\"name\":\"%s\",\"temperature\":\"%s\",\"command\":\"%s\",\"voltage\":\"%s\"}}\n", 
            deviceType, // this happens to be a temperature sensor
            deviceName, // originally read from the XBee
            dtostrf(readTemp(), 4, 1, t),
            dtostrf(readVcc(), 5, 3, v) // This is a text conversion of a float
  Serial.print(Dbuf); // notice this is only for the serial port
  sendXbee(Dbuf);     // out to the XBee
  Serial.println("Message sent");

Not much to it, just a field called 'command' in the JSON string that I can look at when the message comes in. Currently I'm sending the word 'toggle' if the button was pressed and 'nothing' if it wasn't; I love being able to read the messages, so that was descriptive enough for me.

There's another interrupt pin on the Arduino, pin 3, and I can use that also for two buttons. Maybe an on-off toggle. I could also multiplex it by putting a keypad on there. First button wakes up the Arduino and then watch for letters, digits, whatever and compose a more complex command. For now, It's just a button that turns off my bedside lights. 

Tomorrow though, it'll be a button that turns off the bedside light, turns off the outside lights, sets the thermostat to the temperature I use at night, closes the garage doors (if left open), and maybe turns off the water heater power. Call it my 'going to bed now' button.

Now, a bit about the sensor testing. I had to put in a voltage divider to measure the input power because it was driving me nuts monitoring the power after the regulator. To do this, use really high value resistors for the divider and then a .1 cap on the analog pin to ground to lower the impedance. The arduino has an input impedance of around 10K on the input pins, and if your external circuitry is too high, it will give the wrong readings. The easiest way to overcome this problem and still not use very much power is to use a cap. I used 10 Meg and 1 Meg, which put my quiescent current below a micro amp, that should be fine.

So watching it run on the almost dead batteries I installed, it started failing at around 3.6 volts. The regulator on the Pro Mini I'm using was junk. First, it didn't regulate well, the voltage wandered around 3.9 V and eventually dropped off to nothing when the supply went to 3.6. This is not what the spec sheet says. Fine, I have a solution to that, I'll just put a MCP1700 regulator in place and started it back up. Take a look at the specs for the MCP1700, it's a totally awesome low current, low dropout regulator designed for this kind of thing.The new regulator worked great and I've been running with 'dead' batteries for several days now.

I did kind of mess up though. I should have ordered a different MCP1700. I chose the 3.3V version, and I think I should have gone for a lower voltage. Fortunately, they cost me 37 cents each, so I haven't lost much money if I decide to go lower. I'll know more when I hit the cut off point on the 3.3V and how it performs when I bottom out the batteries.

Just for giggles, here's a picture of it right now:

I thought it was a mess of wires a few weeks ago, now you can't see some of the components for the wires connecting things. I haven't tried to condense it yet, I'll get to that later. The new switch is between the batteries and the other stuff so I can get to it. I plan on actually using it for a while to be sure it will do the job for me.

Overall though, this little bundle of wires has really performed well. This learning curve has actually been fun and gave me ideas for other things I can build around the house. Remember, it currently senses temperature, it could easily sense anything else you want to throw at it. Suppose you put a pressure pad in front of the driveway, you could tell when someone pulled in. A light sensor to tell you if the lights in the shed were left on. A moisture sensor beneath the water heater to tell you if it started leaking. An air flow sensor in one of the ducts running through the ceiling. All of these things could be at my beck and call.

One more step in my quest to conquer the world.

The next post in this series is here <link>.

Sunday, April 12, 2015

Getting Back To The Weather Station

I know I've worn out my welcome about the AcuRite 5n1weather station, sorry. This time it's a bit different and I'm really, really open to input and ideas. I decided that the console that came with the weather sensor is too strange and quirky to deal with long term and set up a Raspberry Pi to act as my weather station. I used a radio and directly receive the transmissions from the weatherhead <link> instead of the USB interface I worked on forever <link>

I have a lot of work to go on it, but I'll update it over time as I figure out what I actually want. To date, I set up a little SQLite3 database and record each of the readings there on a periodic basis. Currently, I update each record either as it comes in, or every minute. The reason for the difference is that my barometer and outside temperature sensor is on my XBee network and handled by a different machine, so I only interrogate it on a minute basis.

Yes, the little Pi weather station is a separate machine and I have it talk to the Pi that controls my house to get the two readings.

Actually, let me elaborate on this a bit. Remember way back when I posted about bringing up an HTML interface for the various processes I run using CherryPy <link>? That little effort worked incredibly well. For most processes I have at least two interfaces, one that I can read and one that is formatted in JSON that a machine can read. So, to check on a particular process, I just type the IP address and a port number into the browser and the process responds so I can see whatever I set up. This interaction doesn't care what machine it's on or where the request came from. Using this I have the weather software on one Pi, then get the readings for barometric pressure and outdoor temperature from another one. It's so cool.

I may expand on this idea and have several Pi's running doing different things for me. I could put the XBee network on a machine by itself and have other things talk to it to get the data. Sure it raises the risk of failure because it increases the number of machines that can fail, but It also means I have all the power of a Pi to do a task and not have to worry about things bogging it down. I may move all the upload processes to a single machine so that stuff is separate as well.

But enough of that, I'm here to talk about the weather station. I keep data on a periodic basis for: Rain counter, Outside Temperature on the fence, Outside Temperature at the weatherhead, Barometric Pressure, Wind Speed, Wind Direction, and Humidity. I also keep a daily record, written at midnight that hold the Barometric pressure at that moment, the high and low temperature for the day, and the rain counter at that moment. I create this record once a day, so I can use it as a daily data point on a graph.

From this I can calculate the daily rainfall, and when I have enough data, the weekly, monthly, etc for my location. I have enough data to graph the temperature on a daily basis, and even use the midnight record to go longer term. That stuff, of course, will depend on gathering the data over time. Or, I could download the data from Xively and fill it in for the past if it becomes relevant.

This looks like it will eat up about a megabyte a week in space, and I have roughly 4 Gig available on the Pi, so I have some time to get an NAS online to hold the data long term.

But, as usual, I have no idea what the heck I'm doing. I don't have a clue what I may want next month, or what is the best way to gather the stuff that I don't know about yet. The station is running right now, but it isn't hooked into the web yet so I can't show it off. I'll get to that in a week or so.

I looked into loading the stuff up to Weather Underground, but that didn't excite me very much. I've been fighting cloud providers for a few years now and I'm not impressed too much anymore. With the terabytes of NAS storage available these days and the ease of bringing up a significant little computer, who needs the headaches of specialized protocols and changing terms of service?

So, you folk that have weather stations and such, what am I missing?

Oh, the code isn't on GitHub yet, I'm not sure how I want to arrange it there. I may set up another repository for this instead of adding it to what's already there. I need to think about this a while. But, if anyone wants to take a look, let me know and I'll move more quickly and make it available.

The next post on this project is here <link>.