Tuesday, October 28, 2014

OK, Back to the ZigBee protocol and XBees ... AGAIN

I managed to hack into the Iris Smart Switch from Lowe's and they've been working fine, but there's always been this nagging little annoyance bothering me.  The Alertme Switch that Lowe's sells is NOT ZigBee compliant no matter what they may tell you.  In hacking at it I pointed out a few things that were not according to spec (yes, I've actually read that massive spec related to home automation), and I've been wondering what it would be like to work with an actual ZigBee device.

A reader set me up with one of these:


This is a Centralite 4256050-ZHAC and has an impressive set of capabilities.  They claim that it will work with any controller that is compliant with the ZigBee HA (Home Automation) specification.  That sounds like a challenge to me.  If it is compliant, I should be able to figure out how to work it using an XBee; so away I went.

After almost six DAYS of poking messages at the switch, I was only a tiny bit further along than I was when I started.  This silly thing simply wouldn't join with the XBee so I could see anything it did.  Then I stumbled across a note that said it had a special key.  Key?  It needs a key?  OK, I can try this.  It started working.  I was able to send something and get an answer back; sure the answer was an error, but it was an answer.  Thus began my exploration of the ZigBee protocol in earnest; I was going to make this switch work.

Once again, this isn't one of those posts where I tried for weeks and finally gave up because there just wasn't enough information out there, the machine was too slow, or someone kept something secret; I made it work and will give you the code and XBee configuration to follow in my footsteps down below.  But first I want to talk about the ZigBee protocol and its relationship to XBees a bit.

First, this is an incredibly complex protocol and not for the faint of heart.  Just learning some of the jargon is daunting, much less trying to put it to use.  Sure, there are libraries out there, but have you looked at the prices of those things?  I simply can't afford to mess with them at that price.  Also, the libraries are as hard to understand as the protocol, AND it has the overhead of the library that has to be learned also.  I kept wondering if the XBee somehow could help with this.  Turns out the XBee really can do ZigBee, there just aren't may people that have tried.  Actually, I couldn't find anyone besides me that actually had.

There are lots of pictures and explanations out there about the ideas behind ZigBee, and some of them are even correct, but it was still hard for me to understand.  Let me give you some basics.  The protocol has profiles, these are collections of specs and suggestions for the operation of a system.  Things like Home Automation, Electric Lights (that Hue thingie), Smart Energy, and a device can support one or more of these things.  I'm interested in Home Automation, they call it HA, and that's where I concentrated.  Within this they separate data that you read or change and call them attributes.  These attributes are put within clusters.  Don't get confused, this isn't really that tough.

Within the HA profile, there are some defined clusters and they have numbers that identify them.  Let's take cluster 0x0006, the on-off cluster.  This will have an attribute, the state of the device, and it is numbered 0x0000 and has a datatype of boolean; it tells you if the switch is on or off.  To read this attribute you send a command to the cluster asking for it and the cluster returns the number identifier, datatype and value of the attribute.  See, clusters have commands to operate on them and attributes that you can use.

To tell if the switch is on, send a cluster command 0x00 (read attribute) to cluster 0x0006 (on/off) and the device will send back a command 0x01 (read attribute response) to cluster 0x0006 with the attribute identifier, datatype, value.  Cool.

In the message you send, you also specify the endpoint you want the reply to be sent to and the endpoint you are sending to.  What's an endpoint?  An endpoint is simply a collection of clusters.  On the centralite switch, it has endpoints 0, the general one, and 1, specific to this device.  The general endpoint is where stuff that you want to deal with of a general nature goes and endpoint 1 is where you send stuff that deals with turning the light on and off.

Thus, you have to worry about profiles, endpoints, clusters, clusters commands, and attributes.  Actually it's not that bad, it's just hard to ferret out of the thousands of pages of documentation.  But, you ask, how does the XBee help me?  The XBee eliminates about half of the documentation from being necessary for us to mess with.  It handles all the interactions to set up a network, keep track of device routing, radio initialization, that stuff.  It also gives us a simpler (not simple) message format to use so we don't have to worry about the six or seven layers of protocol, we work totally at the application level and just let it do the rest.  Heck, it even handles the encryption for us.

Combine an XBee to handle the low level link stuff and our own code to handle the application, and you have a reasonable way of controlling these switches that are on the market.  Let me show you the command above in python:

zb.send('tx_explicit',
 dest_addr_long = switchLongAddr,
 dest_addr = switchShortAddr,
 src_endpoint = '\x00',
 dest_endpoint = '\x01',
 cluster = '\x00\x06', # cluster I want to deal with
 profile = '\x01\x04', # home automation profile
 data = '\x00'+'\xaa'+'\x00'+'\x00'+'\x00'
)

There's the long address, it's 32 bits long and doesn't change, ever.  The short address, it's 16 bits long and changes every time the switch joins with the controller; yes, even after a power failure.  The source endpoint.  This is zero because I didn't want to deal with more than one in my code; all the responses come back to endpoint zero.  The destination endpoint which is one on this switch. The cluster id of six as I mentioned above.  The profile 0x0104 which is the number for the HA protocol. And, some data.  The data is one byte of control bits, a byte transaction sequence number that I set to 0xaa so it would be easy recognize, the cluster command 0x00, and the attribute id of 0x0000.  The reason it is shown as ascii characters is a characteristic of the python XBee library implementation.

This may be confusing at first, but trust me, it actually makes sense once you get into it a bit.

This message will send a response to profile id 0x104, endpoint 00, cluster 0x0006, with a payload of 0x00, 0x00, 0x10, 01 if the light is on.  The first two bytes are the attribute id, the next byte is the datatype (0x10 means boolean) and the last byte is 1, meaning the switch is closed.

Are you getting an idea of how this works?  Now, I can hear you asking, "How the heck do I find out these values?"  They're documented in the Cluster Specification document, and there are messages that will itemize the endpoints and clusters within them that the device supports.  So, you send a message to the device to get the endpoints, it tells you what they are, then for each endpoint you ask what the attributes are and it responds.  You look at this stuff, see what you need and use it.

Actually makes sense in a deranged computer scientist sort of way.  But, let's talk about the setup for an XBee specifically to support the Home Automation profile.  That's what I wanted, to be able to turn this switch on and off.  First, it's different from the setup used on the Iris switch so don't think about that, take this as new.

Software Zigbee API Coordinator
Zigbee Stack Profile  (ZS) 2
Encryption Enable  (EE) 1
Encryption Options (EO) 0
Encryption Key  (KY) 5a6967426565416c6c69616e63653039
Network Encryption Key (NK) 0
API Enable (AP) 1
API Output Mode (AO) 3

Yes, you have to use the key.  That part took me the first week of messing with this to find out.  Of course, now that I know what to look for, it would take me about a minute to get it, but that's how we learn.  The difference in the encryption setup is what prevents this switch and the Iris switch from working with the same controller.  You can't have it both ways at once.  If anyone thinks of a way around this, let me know.

Once you have the XBee setup like this you can take the Centralite switch, press the button and hold it, then plug it in the wall.  When the led turns on, let go of the switch and it will join with the XBee automatically.  Yes, that's all there is to joining.  The two devices take care of it themselves and all you have to do is discover the device and start using it.  This is very different from the Alertme devices where we have to mess around with special secret commands to get it to work.  This device actually complies with the specification.

In the code below, I send a message asking for route information and grab the switch's address out of the response.  Then, I send it a command to set up reporting for the light and just wait for someone to tell the code what to do with the light. The commands are:

0 - Turn the switch off
1 - Turn the switch on
2 - Toggle the switch
3 - Dim the switch
4 - Brighten the switch
5 - Tell me the status of the switch
6 - Send messages and print responses about the switch.

Yes, the switch is capable of dimming a light.  The last command goes through a portion of the Zigbee discovery process to find out which endpoints are supported and what clusters and attributes are in them.  It's voluminous, but it's the first time I was actually able to see what the various buzz words actually represented.  This is the kind of thing I did to conquer the way the switch works.

#! /usr/bin/python

'''
This is an examination of a REAL ZigBee device.  The CentraLite 4256050-ZHAC

It has an impressive array of capabilities that I don't delve into in depth in
this examination, but it responds properly to the various ZigBee commands and holds
the clusters necessary to control a switch.

Nice little device
'''

# This is the super secret home automation key that is needed to 
# implement the HA profile.
# KY parameter on XBee = 5a6967426565416c6c69616e63653039
# Have fun

from xbee import ZigBee 
import logging
import datetime
import time
import serial
import sys, traceback
import shlex
from struct import *
'''
Before we get started there's a piece of this that drove me nuts.  Each message to a 
Zigbee cluster has a transaction sequence number and a header.  The transaction sequence
number isn't talked about at all in the Zigbee documentation (that I could find) and 
the header byte is drawn  backwards to everything I've ever dealt with.  So, I redrew 
the header byte so I could understand and use it:

7 6 5 4 3 2 1 0
      X          Disable Default Response 1 = don't return default message
        X        Direction 1 = server to client, 0 = client to server
          X      Manufacturer Specific 
              X  Frame Type 1 = cluster specific, 0 = entire profile
         
So, to send a cluster command, set bit zero.  If you want to be sure you get a reply, clearthe default response.  I haven't needed the manufacturer specific bit yet.
'''
switchLongAddr = '12'
switchShortAddr = '12'

'''
 This routine will print the data received so you can follow along if necessary
'''
def printData(data):
 for d in data:
  print d, ' : ',
  for e in data[d]:
   print "{0:02x}".format(ord(e)),
  if (d =='id'):
   print "({})".format(data[d]),
  print

def getAttributes(data, thisOne):
 ''' OK, now that I've listed the clusters, I'm going to see about 
 getting the attributes for one of them by sending a Discover
 attributes command.  This is not a ZDO command, it's a ZCL command.
 ZDO = ZigBee device object - the actual device
 ZCL = Zigbee cluster - the collection of routines to control it.
  
  frame control bits = 0b00 (this means a BINARY 00)
  manufacturer specific bit = 0, for normal, or one for manufacturer
  So, the frame control will be 0000000
  discover attributes command identifier = 0x0c
  
  then a zero to indicate the first attribute to be returned
  and a 0x0f to indicate the maximum number of attributes to 
  return.
 '''
 print "Sending Discover Attributes, Cluster:", repr(thisOne)
 zb.send('tx_explicit',
  dest_addr_long = data['source_addr_long'],
  dest_addr = data['source_addr'],
  src_endpoint = '\x00',
  dest_endpoint = '\x01',
  cluster = thisOne, # cluster I want to know about
  profile = '\x01\x04', # home automation profile
  # means: frame control 0, sequence number 0xaa, command 0c,
  # start at 0x0000 for a length of 0x0f
  data = '\x00' + '\xaa' + '\x0c'+ '\x00' + '\x00'+ '\x0f'
  )

# this is a call back function.  When a message
# comes in this function will get the data
def messageReceived(data):
 global switchLongAddr
 global switchShortAddr
 
 try:
  #print 'gotta packet',
  #printData(data)  # uncomment this to see the data returned
  
  # Since this is a test program, it will only support one switch
  # go get the long and short address out of the incoming packet
  # for more than one switch, this won't work
  switchLongAddr = data['source_addr_long']
  switchShortAddr = data['source_addr']
  
  if (data['id'] == 'rx_explicit'):
   #print "RF Explicit"
   #printData(data)
   clusterId = (ord(data['cluster'][0])*256) + ord(data['cluster'][1])
   print 'Cluster ID:', hex(clusterId),
   print "profile id:", repr(data['profile']),
   if (data['profile']=='\x01\x04'): # Home Automation Profile
    # This has to be handled differently than the general profile
    # each response if from a cluster that is actually doing something
    # so there are attributes and commands to think about.
    #
    # Since this is a ZCL message; which actually means this message is 
    # is supposed to use the ZigBee cluster library to actually do something
    # like turn on a light or check to see if it's on, the command way down
    # in the rf_data is important.  So, the commands may be repeated in
    # each cluster and do slightly different things
    #
    # I'm going to grab the cluster command out of the rf_data first so 
    # I don't have to code it into each cluster
    #print "take this apart"
    #print repr(data['rf_data'])
    if (data['rf_data'][0] == '\x08'): # was it successful?
     #should have a bit check to see if manufacturer data is here
     cCommand = data['rf_data'][2]
     print "Cluster command: ", hex(ord(cCommand))
    else:
     print "Cluster command failed"
     return
    # grab the payload data to make it easier to work with
    payload = data['rf_data'][3:] #from index 3 on is the payload for the command
    datatypes={'\x00':'no data',
       '\x10':'boolean',
       '\x18':'8 bit bitmap',
       '\x20':'unsigned 8 bit integer',
       '\x21':'unsigned 24 bit integer',
       '\x30':'8 bit enumeration',
       '\x42':'character string'}
    #print "Raw payload:",repr(payload)
    # handle these first commands in a general way
    if (cCommand == '\x0d'): # Discover Attributes
     # This tells you all the attributes for a particular cluster
     # and their datatypes
     print "Discover attributes response"
     if (payload[0] == '\x01'):
      print "All attributes returned"
     else:
      print "Didn't get all the attributes on one try"
     i = 1
     if (len(payload) == 1): # no actual attributes returned
      print "No attributes"
      return
     while (i < len(payload)-1):
      print "    Attribute = ", hex(ord(payload[i+1])) , hex(ord(payload[i])),
      try:
       print datatypes[payload[i+2]]
       i += 3
      except:
       print "I don't have an entry for datatype:", hex(ord(payload[i+2]))
       return
       
    if (clusterId == 0x0000): # Under HA this is the 'Basic' Cluster
     pass
    elif (clusterId == 0x0003): # 'identify' should make it flash a light or something 
     pass
    elif (clusterId == 0x0004): # 'Groups'
     pass
    elif (clusterId == 0x0005): # 'Scenes'  
     pass
    elif (clusterId == 0x0006): # 'On/Off' this is for switching or checking on and off  
     #print "inside cluster 6"
     if cCommand in ['\x0a','\x01']:
      # The very last byte tells me if the light is on.
      if (payload[-1] == '\x00'):
       print "Light is OFF"
      else:
       print "Light is ON"
     pass
    elif (clusterId == 0x0008): # 'Level'  
     pass
    else:
     print("Haven't implemented this yet")
   elif (data['profile']=='\x00\x00'): # The General Profile
    if (clusterId == 0x0000):
     print ("Network (16-bit) Address Request")
     #printData(data)
    elif (clusterId == 0x0008):
     # I couldn't find a definition for this 
     print("This was probably sent to the wrong profile")
    elif (clusterId == 0x0004):
     # Simple Descriptor Request, 
     print("Simple Descriptor Request")
     print("I don't respond to this")
     #printData(data)
    elif (clusterId == 0x0013):
     # This is the device announce message.
     print 'Device Announce Message'
     #printData(data)
     # This is a newly found device, so I'm going to tell it to 
     # report changes to the switch.  There are better ways of
     # doing this, but this is a test and demonstration
     print "sending 'configure reporting'"
     zb.send('tx_explicit',
      dest_addr_long = switchLongAddr,
      dest_addr = switchShortAddr,
      src_endpoint = '\x00',
      dest_endpoint = '\x01',
      cluster = '\x00\x06', # cluster I want to deal with
      profile = '\x01\x04', # home automation profile
      data = '\x00' + '\xaa' + '\x06' + '\x00' + '\x00' + '\x00' + '\x10' + '\x00' + '\x00' + '\x00' + '\x40' + '\x00' + '\x00'
     )
    elif (clusterId == 0x8000):
     print("Network (16-bit) Address Response")
     #printData(data)
    elif (clusterId == 0x8032):
     print "Route Record Response"
    elif (clusterId == 0x8038):
     print("Management Network Update Request");
    elif (clusterId == 0x8005):
     # this is the Active Endpoint Response This message tells you
     # what the device can do
     print 'Active Endpoint Response'
     printData(data)
     if (ord(data['rf_data'][1]) == 0): # this means success
      print "Active Endpoint reported back is: {0:02x}".format(ord(data['rf_data'][5]))
     print("Now trying simple descriptor request on endpoint 01")
     zb.send('tx_explicit',
      dest_addr_long = data['source_addr_long'],
      dest_addr = data['source_addr'],
      src_endpoint = '\x00',
      dest_endpoint = '\x00', # This has to go to endpoint 0 !
      cluster = '\x00\x04', #simple descriptor request'
      profile = '\x00\x00',
      data = '\x13' + data['source_addr'][1] + data['source_addr'][0] + '\x01'
     )
    elif (clusterId == 0x8004):
     print "simple descriptor response"
     try:
      clustersFound = []
      r = data['rf_data']
      if (ord(r[1]) == 0): # means success
       #take apart the simple descriptor returned
       endpoint, profileId, deviceId, version, inCount = \
        unpack('<BHHBB',r[5:12])
       print "    endpoint reported is: {0:02x}".format(endpoint)
       print "    profile id:  {0:04x}".format(profileId)
       print "    device id: {0:04x}".format(deviceId)
       print "    device version: {0:02x}".format(version)
       print "    input cluster count: {0:02x}".format(inCount)
       position = 12
       # input cluster list (16 bit words)
       for x in range (0,inCount):
        thisOne, = unpack("<H",r[position : position+2])
        clustersFound.append(r[position+1] + r[position])
        position += 2
        print "        input cluster {0:04x}".format(thisOne)
       outCount, = unpack("<B",r[position])
       position += 1
       print "    output cluster count: {0:02x}".format(outCount)
       #output cluster list (16 bit words)
       for x in range (0,outCount):
        thisOne, = unpack("<H",r[position : position+2])
        clustersFound.append(r[position+1] + r[position])
        position += 2
        print "        output cluster {0:04x}".format(thisOne)
       clustersFound.append('\x0b\x04')
       print "added special cluster"
       print "Completed Cluster List"
     except:
      print "error parsing Simple Descriptor"
      printData(data)
     print repr(clustersFound)
     for c in clustersFound:
      getAttributes(data, c) # Now, go get the attribute list for the cluster
    elif (clusterId == 0x0006):
     #print "Match Descriptor Request"
     # Match Descriptor Request
     #printData(data)
     pass
    else:
     print ("Unimplemented Cluster ID", hex(clusterId))
     print
   else:
    print ("Unimplemented Profile ID")
  elif(data['id'] == 'route_record_indicator'):
   #print("Route Record Indicator")
   pass
  else:
   print("some other type of packet")
   print(data)
 except:
  print "I didn't expect this error:", sys.exc_info()[0]
  traceback.print_exc()
  
if __name__ == "__main__":
 #------------ XBee Stuff -------------------------
 # this is the /dev/serial/by-id device for the USB card that holds the XBee
 ZIGBEEPORT = "/dev/serial/by-id/usb-FTDI_FT232R_USB_UART_A600eDiR-if00-port0"
 ZIGBEEBAUD_RATE = 9600
 # Open serial port for use by the XBee
 ser = serial.Serial(ZIGBEEPORT, ZIGBEEBAUD_RATE)


 # The XBee addresses I'm dealing with
 BROADCAST = '\x00\x00\x00\x00\x00\x00\xff\xff'
 UNKNOWN = '\xff\xfe' # This is the 'I don't know' 16 bit address

 #-------------------------------------------------
 logging.basicConfig()

  
 # Create XBee library API object, which spawns a new thread
 zb = ZigBee(ser, callback=messageReceived)
 print "started at ", time.strftime("%A, %B, %d at %H:%M:%S")
 notYet = True;
 firstTime = True;
 while True:
  try:
   if (firstTime):
    print("Wait while I locate the device")
    time.sleep(1)
    # First send a route record request so when the switch responds
    # I can get the addresses out of it
    print "Broadcasting route record request "
    zb.send('tx_explicit',
     dest_addr_long = BROADCAST,
     dest_addr = UNKNOWN,
     src_endpoint = '\x00',
     dest_endpoint = '\x00',
     cluster = '\x00\x32',
     profile = '\x00\x00',
     data = '\x12'+'\x01'
    )
    # if the device is already properly joined, ten seconds should be
    # enough time for it to have responded. So, configure it to
    # report that light has changed state.
    # If it hasn't joined, this will be ignored.
    time.sleep(5)
    print "sending 'configure reporting'"
    zb.send('tx_explicit',
     dest_addr_long = switchLongAddr,
     dest_addr = switchShortAddr,
     src_endpoint = '\x00',
     dest_endpoint = '\x01',
     cluster = '\x00\x06', # cluster I want to deal with
     profile = '\x01\x04', # home automation profile
     data = '\x00' + '\xaa' + '\x06' + '\x00' + '\x00' + '\x00' + '\x10' + '\x00' + '\x00' + '\x00' + '\x40' + '\x00' + '\x00'
    )
    firstTime = False
   print "Enter a number from 0 through 8 to send a command"
   str1 = raw_input("")
   # Turn Switch Off
   if(str1[0] == '0'):
    print 'Turn switch off'
    zb.send('tx_explicit',
     dest_addr_long = switchLongAddr,
     dest_addr = switchShortAddr,
     src_endpoint = '\x00',
     dest_endpoint = '\x01',
     cluster = '\x00\x06', # cluster I want to deal with
     profile = '\x01\x04', # home automation profile
     data = '\x01' + '\x01' + '\x00'
    )
   # Turn Switch On
   if(str1[0] == '1'):
    print 'Turn switch on'
    zb.send('tx_explicit',
     dest_addr_long = switchLongAddr,
     dest_addr = switchShortAddr,
     src_endpoint = '\x00',
     dest_endpoint = '\x01',
     cluster = '\x00\x06', # cluster I want to deal with
     profile = '\x01\x04', # home automation profile
     data = '\x01' + '\x01' + '\x01'
    )
   # Toggle Switch
   elif (str1[0] == '2'):
    zb.send('tx_explicit',
     dest_addr_long = switchLongAddr,
     dest_addr = switchShortAddr,
     src_endpoint = '\x00',
     dest_endpoint = '\x01',
     cluster = '\x00\x06', # cluster I want to deal with
     profile = '\x01\x04', # home automation profile
     data = '\x01' + '\x01' + '\x02'
    )
   # This will dim it to 20/256 over 5 seconds
   elif (str1[0] == '3'):
    print 'Dim it'
    zb.send('tx_explicit',
     dest_addr_long = switchLongAddr,
     dest_addr = switchShortAddr,
     src_endpoint = '\x00',
     dest_endpoint = '\x01',
     cluster = '\x00\x08', # cluster I want to deal with
     profile = '\x01\x04', # home automation profile
     data = '\x01'+'\xaa'+'\x00'+'\x25'+'\x32'+'\x00'
    )
   # This will brighten it up to 100% over 5 seconds
   elif (str1[0] == '4'):
    print 'Bright'
    zb.send('tx_explicit',
     dest_addr_long = switchLongAddr,
     dest_addr = switchShortAddr,
     src_endpoint = '\x00',
     dest_endpoint = '\x01',
     cluster = '\x00\x08', # cluster I want to deal with
     profile = '\x01\x04', # home automation profile
     data = '\x01'+'\xaa'+'\x00'+'\xff'+'\x32'+'\x00'
    )
   elif (str1[0] == '5'):
    print 'Report Switch Status'
    zb.send('tx_explicit',
     dest_addr_long = switchLongAddr,
     dest_addr = switchShortAddr,
     src_endpoint = '\x00',
     dest_endpoint = '\x01',
     cluster = '\x00\x06', # cluster I want to deal with
     profile = '\x01\x04', # home automation profile
     data = '\x00'+'\xaa'+'\x00'+'\x00'+'\x00'
    )
   elif (str1[0] == '6'):
    print 'Get Report from Switch'
    zb.send('tx_explicit',
     dest_addr_long = switchLongAddr,
     dest_addr = switchShortAddr,
     src_endpoint = '\x00',
     dest_endpoint = '\x00',
     cluster = '\x00\x05', # cluster I want to deal with
     profile = '\x00\x00', # home automation profile
     data = switchShortAddr[1]+switchShortAddr[0]
    )
  except IndexError:
   print "empty line, try again"
  except KeyboardInterrupt:
   print "Keyboard interrupt"
   break
  except NameError as e:
   print "NameError:",
   print e.message.split("'")[1]
   traceback.print_exc(file=sys.stdout)
  except:
   print "Unexpected error:", sys.exc_info()[0]
   traceback.print_exc(file=sys.stdout)
   break
   
  sys.stdout.flush() # if you're running non interactive, do this

 print ("After the while loop")
 # halt() must be called before closing the serial
 # port in order to ensure proper thread shutdown
 zb.halt()
 ser.close()

No, it isn't pretty, but it has comments.  It should be easy for folk to read and try out, and is the first example of a direct interface to a ZigBee compliant device I've ever seen.  This should take some of the mystery out of the protocol and the controllers that use it.  This code could be expanded to work one of the thermostats, receive from a panic button, or even one of those simple alarm switches.  The XBee does all the really hard stuff and saves us from worrying about it.

Have fun

60 comments:

  1. the zigbee protocol reminds me slightly of the BACnet protocol for HVAC equipment but it appears to be a lot simpler to use due to the protocol stack already being implemented in the radios.

    ReplyDelete
  2. This may be a stupid question, but I'm going to ask it anyways. Having the key value as the standard HA profile key, does that mean you can't really encrypt your local zigbee network? I must admit I haven't really dug into the standard yet, but it would be a concern to me if everyone using a zigbee standard are all using the same encryption key. Or am I mistaken on the idea of this key being the encryption key?

    ReplyDelete
    Replies
    1. Heh, you're right. I wondered if anyone would notice. If you prowl around the web, you'll find a number of places where this is pointed out in glorious detail. It's not as dangerous as it sounds though since the key that is shown is used in a key exchange and the actual key that encrypts the data is not sent in the clear, so normal traffic can't be read easily. However, if you get a device to join the network, the key above will travel in the clear and can be captured. From that point it's a complex interaction to break in, but it can be done.

      See, everything needs somewhere to start. If the key were kept secret (like the EL profile right now) then someone would steal it and the word would be out. If it is public (like this one) everybody ignores it because it's already known. Sure, there's solutions to this, but it makes the devices hard to implement since something has to be kept safe,and it's already too late since there are tons of devices out there already.

      Net, the HA network is not as safe as one could hope, but it is protected from casual observation, and short range. Doing it this way allows tools to be built that can monitor the network to solve problems, something that would be much harder if more security were added, and several orders of magnitude better than the older implementations of home automation.

      No, I don't have a remote door lock, and my garage doors are on my own proprietary network protected by my personal, super secret key ... and a dog.

      Delete
  3. Yet another great discovery post Dave.
    Very interesting read and comments.
    Grant, like we say in the networking business "the encryption key is there to keep the honest people out".
    Dave, I'm curious if you have tried your code with the IRIS plugin yet? I'm curious how "conforming" they are?
    Thanks again.
    Glenn.

    ReplyDelete
    Replies
    1. There's a tiny sentence or two up above. Nope, the Iris switch and true ZigBee compliant devices can't share the same XBee controller. At least I can't find any way. There's a pretty significant difference between the two of them that I can't see anyway around.

      Delete
  4. Found this in some of my XBee documents. Thought you might find it of value:

    2.3.1.3
    Transaction Sequence Number
    The transaction sequence number field is 8 bits in length and specifies an
    identification number for the transaction so that a response-style command frame
    can be related to a request-style command frame. The application object itself
    shall maintain an 8-bit counter that is copied into this field and incremented by
    one for each command sent. When a value of 0xff is reached, the next command
    shall re-start the counter with a value of 0x00.
    The transaction sequence number field can be used by a controlling device, which
    may have issued multiple commands, so that it can match the incoming responses
    to the relevant command.

    Also, in your comments for your source code you showed the bit numbers of the byte numbered from left to right. This is very common in network transmission work. It is done so the most significant bit is sent first and the lower order bits are sent last. Since these are the lowest values and hence the most recent values the will then be the most closely aligned with the current readings. IE they carry the least amount of delay. Sending them in the other order would allow the current values to shift quite a bit from the by the time the whole byte is received. Again very common in data acquisition, SCADA, etc. technologies.
    Might just have to order a couple of these off eBay and try them out.
    Glenn.

    ReplyDelete
    Replies
    1. Yes, I know that the network types like to label things that way. It sort of reminds me of the electrical engineers that draw current flow from the positive to the negative. It's just backwards to me. I want to look at a diagram and come away from it with a hexadecimal number that I can type into a program, not the exact reverse. I also think of current being the direction the electrons travel, not the direction the absence of an electron travels.

      What's funny about the transaction id is that it is used in the cluster packets (ZCL), but isn't documented there in many cases. You just have to know it's there for some of the specs. Me, I just stick something in there that I can recognize when I dump the packets to see what I did wrong.

      Thanks, and anything else you find that's interesting, let me know.

      Delete
  5. Hi Dave - this is great stuff! One simple question, once you set the encryption key in your controller, all of your xbee devices started using encryption with that key?

    -Mark R.

    ReplyDelete
    Replies
    1. Mark, the way encryption works on the XBee is a real pain to explain and I know I would mess it up. So, take a look in the XBee user's guide (just google it and you'll a hundred or so links) for the in depth description. Sure, it's a lot of jargon, but they have pictures to help.

      In this particular case, I set the value in the controller and the switches already have it in them; that's why I could talk to them at first. The switches expected everything to be encrypted and wouldn't listen to me. So, no, everything had to have the key; I just didn't have to put it in the switch myself because it was already there.

      Delete
  6. This stuff works, to turn light switch off , you set data = '1' + '1' + '0', could you please explain these
    fields , I am probably not interpreting or unable to correlate with Zigbee Cluster specs.
    Is '0' the command Id 'off' and is '1' + '1' the attribute Id [16 bits] OR
    is '0' the 'off' value of command '1', in which case, what is the remaining '1'.
    Appreciate ur help.

    ReplyDelete
    Replies
    1. I'm working from old memory here, but it's part of the general format of the frame control field. Look at the command frame formats piece for the ZCL header (section 2.3). The first 01 is actually a bunch of bits and only the first bit is needed for direction. The Zigbee document shows the bits in reverse order to what I expected with the LSB on the left. When you read that section, remember that 0b00 isn't a hex number, it means 0 BINARY 00, or no bits set in a 2 bit field. That drove me nuts. The other one is a transaction ID, I should have set it to "AA" like I did the others, but I was sick of it at that point. then the command to cluster 6 (on/off cluster) telling the light to turn on or off.

      When you send an on/off to cluster 6, it's a command, not setting an attribute. That's another part of the confusion. You might think of it as an objects method rather than an objects attribute.

      I know this is clear as mud, but I based the explanation on the way the document reads so I wouldn't lose you.

      Delete
  7. Dave,

    I've been trying to extend your code to include other Iris devices - actually, only one: the door sensor. I have a new (ie, not a member of Iris) Smart Switch and a new door sensor, and I was able to register and control the Smart Switch from your code. The door sensor seems to join, but it looks like it just sends a request, and the python script sends four messages back to it. The fact that these have hardcoded data, I guess, is why the Smart Switch responds and the door sensor doesn't.

    Can you tell from the data in the request what the device is looking for to join the network, and then hopefully send door contact state changes? Is this strictly a trial-and-error thing?

    thanks for any help! Busting this door sensor will free me from IrisSmartHome.com completely.

    ReplyDelete
    Replies
    1. I just, I mean just as in about 10 minutes ago, got one of the door sensors. I tried it and got exactly the same thing you did. However, I'm up to my neck in things I HAVE to do right now and won't get to it until tomorrow or the next day.

      So, don't hold your breath, but I will be looking at it real soon.

      Meanwhile, No, it isn't entirely trial and error, but it is a LOT of guessing to break into one of these things. If I get into it, I'll post about it. You're welcome to drop me a note by email and I'll keep you informed of my progress as I step through it. You can also send me the things you find out and we can work together to beat it into submission.

      Delete
  8. Hi
    I am a beginner in the field of Zigbee. Your posts are great!!
    I wanted to know if there were any open source libraries for implementation of Zigbee Home Automation profile.
    I want to control ZHA devices using Xbee module.
    I am aware of the fact that there is no common API for devices belonging to different developers.
    However, understanding the implementation of any one would be of a great use if I want to write my own HA library.
    Is it possible to get the source code for any one.

    ReplyDelete
    Replies
    1. I've never found one, and the source for the commercial ones is protected. The commercial ones are extremely complex to use and cost over a $1000 (US) to license. Akiba at FreakLabs gave an open source ZigBee library a valiant try a few years ago, but ran into licensing problems and quit the effort.

      Delete
    2. So, is there no way of controlling the ZHA certified devices without obtaining the license ?
      If there is,then how?

      Delete
    3. Er, uh, I got the HA appliance switch to work with my own code in this post. So, yes, there is a way. That's the reason for this post.

      Zigbee isn't easy to work with

      Delete
  9. Does nxp provide source code in their sdk if we purchase their chip as well as home automation device?
    Is it possible for us to get the functions included for communication relevant to home automation?

    ReplyDelete
    Replies
    1. I have no idea, The way most of the manufacturers do it is to sell you a development system that includes access to the libraries after you sign an license agreement. These development systems cost over a thousand dollars and the code is hard to understand.

      Each supplier is a little different from the others and offers slightly different stuff.

      Delete
  10. Hello!

    I have been trying the get my xbee zb coordinator to see my zigbee ha certified devices. I have quite a few adhoco s1 devices to play with. I have managed to make the xctu program to see the adhoco devices for a short a while and the connection established light to turn on on adhoco. But after 30 seconds the xbee coordinator losses the connection and adhoco device's light starts to blink for an error.

    Is it so that in your case the xbee zb automatically accepts centralite zigbee device to the mesh network and you don't have issue any extra commands through xbee to approve the centralite to join to the zigbee mesh.

    Next I changed the ZS value to 1 (according to the zigbee profile documentation the profile 2 is back ward compatible to profile 1) and the result was that the adhoco device kept showing and disappearing in XCTU's network view but XCTU/xbee couldn't create permanent connection to the adhoco device. XCTU was able to tell the adhoco devices long and short address and the mode (router/end device). Any idea how to make the xbee coordinator to establishe permanent connection to the adhoco device.

    ReplyDelete
    Replies
    1. I've never worked with an adhoco device, so I don't really know. However, this sounds exactly like what the Alertme devices do. If the end device doesn't get a specially formatted message from the controller, it leaves the network and starts looking for a different controller. These symptoms fit your situation exactly.

      Delete
    2. The Adhoco devices have a Zigbee Alliance's PROHA certification and I have understood that this should guarantee the automatic joining etc. functions. Right?
      http://www.zigbee.org/zigbee-products-2/#zigbeecertifiedproducts/productdetails3/5553fe4df7f0b6063927b095/

      Is there any examples about these kind of messages or info in the zigbee alliances documents. All ideas are more than welcome. Next I will start logging the communication between adhoco and xbee. Thanks to you I can build on your work and your example codes:)

      Delete
    3. The Alertme devices are certified also, but that didn't stop them from putting in special code to limit how the devices are used. Apparently the certification is somewhat lacking in its requirements. I have only found reference to the 'manufacturer specific' messages in the Zigbee docs. It's really annoying.

      Your best bet is to log the interaction and try to duplicate it. You can also monitor a working system to see what happens; that's hard to do though.

      Delete
    4. I will try the logging the interaction. I'm also considering of purchasing a Samsung smartthings hub and try to make that work with the adohoco devices. One thing I can also try to do, is to configure the xbee as a router and log the interaction when the xbee tries to joint to the Adhoco coordinator.

      If I manage to get something working I will write a short update here. Anyway big thanks to you for sharing your experience and code!

      Delete
  11. Hi,

    I'm trying to send a command to a Philips Hue Lux lightbulb using a series 2 Xbee.

    I am able to get the bulb to join the network and get an active descriptor response:

    >> { type: 145,
    remote64: '0017880100e42160',
    remote16: 'af28',
    sourceEndpoint: '00',
    destinationEndpoint: '00',
    clusterId: '8005',
    profileId: '0000',
    receiveOptions: 1,
    data: }


    And then a device joined response:

    >> { type: 145,
    remote64: '0017880100e42160',
    remote16: 'af28',
    sourceEndpoint: '00',
    destinationEndpoint: '00',
    clusterId: '0013',
    profileId: '0000',
    receiveOptions: 2,
    data: }

    The light flashes off and then on quickly with these commands. But for the life of me I cannot figure out the appropriate endpoint to send a 0006 or 0008 cluster command. I have searched for days trying to figure out how to decipher the data buffers - I can see that on the 0013 response it returns the short and long addresses and I suppose the capabilities, but I find it hard to decipher the active endpoints data buffer. I would really appreciate any help

    ReplyDelete
    Replies
    1. Argh! It removed the data. Here are the buffers:

      For 8005:
      30 81 30 30 00

      For 0013:
      00 28 af 60 21 e4 00 01 88 17 00 8e

      Delete
    2. Finally figured it out. Hooray! This has been a confusing ride. I was especially a bit confused because I wasn't sure if the bulb was actually "joined" after sending a 0013. To add to the confusion, the bulb was continually sending "0006" requests that I wasn't sure if I needed to respond to (especially after reading CapnBry's stuff at http://jeelabs.net/boards/6/topics/285, where some of it is specific to an AlertMe switch).

      And to add more to the confusion, in that 0006 there was a profile of "5e c0" <===> "c05e" -- i.e., the ZigBee Light Link profile. I don't want to deal with that, I only want 0104 - home automation. My fear was that the bulb would only do ZLL, but I read that the Hue uses "0xC05E, used for touchlink commissioning commands. All other ZCL commands are sent using HA profile" and so kept on trying.

      Anyway the roadblock was a simple issue -- I did not know the endpoint of the bulb and did not know how to get it. To get the endpoint you send a Match Descriptor Request (0x0006) to profile 0. I had:

      type: 11
      id: 01
      dest64: 0017880100e42160
      dest16: af28
      sourceEndpoint: 0
      destEndpoint: 0
      clusterId: 0006
      profile: 0000
      data: 0128af040101080000

      Check out the ZigBee specification document for formatting the data in this command. In my case it's:

      - the very frustrating 01 byte that Dave mentioned (I found that I needed to add this bit waaay after I started fiddling with this - wouldn't have found it if Dave hadn't mentioned it)
      - 28af - little endian 16 bit address of bulb (af28)
      - 0401 -- little endian home automation profile (0104)
      - 01 -- the number of inbound clusters I am looking to see are supported on the bulb
      - 0800 - little endian cluster support I'm looking for (0008, which is dimming in HA profile)
      - 00 -- the number of outbound clusters support I'm looking for

      In this case I'm specifically looking to see which endpoint might support the HA profile and the dimming cluster, if any. It responded with an 8006 and this data:

      01 00 28 af 01 0b

      I guess the data is:

      01 - frustrating byte again
      00 - ?
      28af - <==> af28
      01 - number of endpoints listed
      0b - the endpoint!

      Which is confusing because I don't think it matches the the ZigBee specification - it seems out of order. But no complaints here as that gave me the endpoint I need. Dave's code for switching the light on and off works well in this case after switching out the endpoint.

      Delete
    3. Hi Greg, what's the structure of match descriptor request? Did you use XCTU to send it? What was the cluster and data payload? I assume profile was x0104 and endpoint was x00?

      Delete
  12. Hi Dave, great post.

    I have a relatively simple question for you. I want to extend/repeat my ZHA Zigbee network created by SmartThings as the distance of the hub is terrible and I need it to reach about 100 foot outside (it doesn't even get through one thick wall at the moment)

    Is it possible to use an Xbee with an antenna on an Arduino as nothing more than a ZHA repeater/extender?

    ReplyDelete
    Replies
    1. Yes, it absolutely is. The problem is getting the channel it's working on and then getting it to join the network as a router device. The XBee supports the HA protocol pretty well, it's all the controllers out there seem to have something built into them to limit the devices they'll talk to.

      Delete
    2. Hi Dave, just to let you know I managed to do this with home from someone on the SmartThings forum who'd set up the Xbee XCTU settings to let it join a SmartThings hub (encryption settings mainly) it joins straight away and acts as a router, no extra CPU or anything needed, just power the thing however you want and give it the router firmware and you've added a powerful Zigbee repeater to your system! :D

      Delete
  13. First just want to say THANKS !

    When Zigbee came out I cheered .. "Open Standard" .. yea .. well just like promises from politicians, in reality not. BTW I invested quite a bit in dev. kits. Somewhere there is a box full of XBee's and even two that I designed and even had prototype boards made.

    I'm going to be rewiring my 1950's house this year, and it seems the manufactures have done all they can to loose money.

    ReplyDelete
    Replies
    1. Based on somewhat limited experience, it appears the big culprit in weaseling the Zigbee protocol is AlertMe. Their devices, albeit well made, sneak around the standard and have to handled specially. There are true Zigbee devices out there, but not many of them.

      Delete
  14. Thanks you for the article and for the detail outlined in this strain of comments. It's been a big help to understanding the flow of the protocol and getting my exploratory app going to help me really know what endpoints support vs what the smartthings hub has been showing me.
    With the help of this article I was able to get my Cree bulbs synced up with the XBee, list information from them with ZDO and control them with ZCL using the Home Automation profile.
    The problem I'm having is that I'm going to bed, there's no activity, and when I get up I get make a transmission failure address not found. The devices are still in the XBee routing table, but it isn't able to communicated with them. If I factory reset the lights, they rejoin appropriately and work fine until the next time I leave them alone for a while. I'm not yet sure how long of a time frame that is.
    The question I have is... am I supposed to be sending the device a command after it joins to bind it to my network? Or has someone else seen an conquered this issue?
    I thought at first that my old hub was reclaiming the device some way, but after experimenting a bit this does not seem to be the case.

    ReplyDelete
    Replies
    1. The address cache on the end device is purged every so often if there is no communication. I can't remember how long the period is, but I've seen it happen a few times. The XBees take care of this between XBees, but they don't if it's some other device. I get around this by getting the status of the device every couple of minutes or so.

      I stumbled across this by accident once when my controller got hit by a power failure. Before that I never saw a problem because I was interrogating devices on a regular basis, or had them set up to send data on a fairly short schedule.

      You should be able to make the end devices wake up and reestablish the channel by sending that message that makes them all reply. I can't remember what it's called right now, buy you should be able to find it pretty easily.

      Delete
    2. Oh, you can also get around this by putting a router from the same manufacturer in the network. I saw this happen on some Alertme devices a while back.

      Delete
  15. Dave, thanks for all this information -- I've learned a lot following this the past year or so. I've generally been able to resolve issues, but I've been stumped for a while now on attempting to get the latest Iris Smart Plug (3210-L) to work. I believe this is similar to the Centralite plug above. Anyway, I appear to be able to transmit and receive to the plug with XCTU (I can turn the outlet on and off and I get an Explicit RX Indicator response). When I control smart plug with python, I can still turn the outlet off and on, but I don't receive any messages from the smart plug. I also don't get an responses when I send the Active Endpoint Request or other TX frames. I've configured the xbee as you show above. Any suggestions? Thanks.

    ReplyDelete
    Replies
    1. OK, I'm not trying to make you feel stupid, but it appears your aren't getting anything back at all. If you can see the response using XCTU, you should be able to see the response with python as well. I mean, you're sending the exact same thing in python that you are with XCTU, so it should work the same way.

      Go take a close look at how you've wired up and implemented the receive side of the XBee, maybe try sending a message with message status turned on so you get the status back, something.

      And, you do know you owe me some code when you get it working right? I'm going to get to the new generation of Iris in time, but there's too much on my plate right now to get to it.

      Delete
    2. Dave -- I'm using a Sparkfun XBee Explorer to connect to the PC. I've used this to communicate with other xbees in the past and haven't had a problem. After reading your reply, I tried a different Explorer and I immediately started receiving data from the smart plug. I swapped back to the original Explorer and it now works too!

      I'm using your code above and it seems to work with the newest smart plug. I'll look into this more and post if I learn anything new about the smart plug. Thanks again for your help! Brian

      Delete
    3. That's awesome. Now it also brings up a very interesting point: we should be able to control all the Zigbee devices from Lowes if we just examine them and then do a few adjustments to the code.

      Now, if they only had a zigbee wall switch; that would be awesome.

      Delete
    4. Yes, I use the word 'awesome' way too much.

      Delete
    5. hi dave ,
      i m trying to control smarthings motion sensor with xbee as coordinator if you can check my forum i really need help

      https://community.smartthings.com/t/controle-smartthings-devices-with-xbee/61678/8

      thanks,

      Delete
    6. Marwen, I took a look at the thread there and the guy that is helping you went astray a bit. He keeps thinking you are using the SmartThings controller and you aren't. But, He's exactly right, working with the devices is a conversation, you send something, it sends something back. Some devices can be set up to periodically send you stuff as well.

      If you don't meet the conversation requirements, the devices go to sleep and usually don't want to wake up. I know basically nothing about the devices you're working with and can't be much help since I can't actually try anything to see what happens.

      Delete
  16. Hi,

    Thanks so much for writing this. I have a centralite (iris) door sensor I'm working with. Is it the Match Descriptor response that allows the sensor to join the network? I see the 0x91 request in XCTU, but I've not responded to one. It appears that the response is coded in the C sample code on github, but I do not see it in the Python example.

    Thanks,
    Bobby

    ReplyDelete
    Replies
    1. It's been too long for me to answer directly without dragging the stuff out and running tests all over again. I'm going to get into that much later this year, but right now they're in a box up in the attic.

      I think you may be mixing the 0x91 message in the XBee with some of the commands in the protocol though. When I worked with the Centralite, it just joined all on it own.

      Delete
    2. This comment has been removed by the author.

      Delete
    3. Hello Robert and Dave;

      I'm using the same door sensor from the Centralite but it seems that it's constantly dropping from the zigbee network.

      You can find a detailed explanation of the issue here: https://www.digi.com/support/forum/63359/zigbee-device-frequently-joining-dropping-from-coordinator I have also uploaded a video that shows the case.

      Would you please shed some light on the issue according to your experience with the sensor?

      Best.

      Delete
    4. I have never used one of the door switches from Centralite, but it looks like they make it rejoin each time it turns on. End devices (opposed to routers) shut off completely to save battery and then turn on and report.

      Delete
  17. Dave thank you for your answer.

    I have just opened the plastic cover of the sensor as it mentioned on the manual for starting the join process. And when I closed it back, it stills disconnects and reconnects for a while (changing 16 bit network address).

    I have two door sensors and they behave exactly the same.

    Do you have any idea why this is happening? Is it a configuration (security) issue?

    Best

    ReplyDelete
    Replies
    1. Do they actually work, as in report that the door is open or closed? If they don't, they haven't really joined' they hung up trying to join instead. I noticed that you sometimes need a router somewhere in the network for certain types of devices to join. I don't know why it works that way, but it does.

      Are these the new Lowe's Iris door switches?

      Delete
    2. No these door sensors are from Centralite. I have added my forum post at digi https://www.digi.com/support/forum/63359/zigbee-device-frequently-joining-dropping-from-coordinator

      I noticed that, the end device (door sensor) sends Cluster ID: 0x0013 again and again (DEVICE ANNOUNCE FROM ENDDEVICE)

      And for a couple of minutes later, device goes to sleep mode I guess without successfully joining the zigbee coordinator.

      Is there a zigbee message sequence do I need to send back to the end device in order to join successfully?

      Delete
    3. There probably is, but I don't know what it is. Go prowl around the zigbee docs to see if you get a hint what should be done.

      Delete
    4. This comment has been removed by the author.

      Delete
  18. Hey Fatih,

    Have u found a Solution for your problem? I have the same issue with my Door Sensor :(

    ReplyDelete
    Replies
    1. Hi Patrick,

      Yes I have finally managed them to work. First be sure to update your Xbee Controller firmware! And then execute the commissioning process of ZigBee IAS Zone. You can find the exact steps from the ZigBee HA public documentation.

      Best Luck.


      Delete
    2. Hi Fatih,

      Thank you for your Response! I've downloaded the documentation but i didn't find this Steps. Can u say me the number of the page or posting me the Pairing steps? That would be nice.

      Delete
  19. This is the best ZigBee site yet.

    After days and days I can finally control SOMETHING with code. This is fantastic.

    For non-python people, you can get the xbee library with "pip install xbee".

    Next is receiving unsolicited data from a device. I'll keep reading here and experimenting.

    Wow! This is great!

    ReplyDelete
  20. Your blog is good for me because I needed information about home automation system which you have been stored in your blog. Thank you so much for the message. Home Automation Company in Sutherland Shire

    ReplyDelete