Tuesday, November 18, 2014

Raspberry Pi, CherryPy, Process Communication, Part 2

Last post I talked about experimenting with a different method of process communication: using HTTP between the processes.  This will allow me to move some of my work to another machine, and increase my ability to monitor the various items.

I got into this mess because of the plethora of different devices and protocols I'm messing with.  Each set of devices has a different protocol, some invented by me, others invented by industry.  If you have two different ZigBee devices that can't talk to each other, you need something to translate in the middle.  This isn't too hard, but it gets complex when you're chasing a bug around and can't poke the pieces to see what happens.  So, I started with my Wemo devices, and added a minimal webserver to the code.  This allows me to talk to and check the status of the switches using a browser as well as a separate process that collects the data.

It turned out to be relatively easy, but I (again) had to learn a new jargon.  I'm finding over and over that the key to doing anything in the home automation realm is the language.  When you look at a library or implementation, be prepared for a few hours with google to understand what they're saying.

CherryPy turned out to be pretty slick.  It works differently than the big servers like Apache, but it can be used pretty easily.  I started off with the CherryPy tutorial and just started adding things I needed until I had a skeleton.  That's described in the last post.  Then I took the skeleton in one window and the existing code in another and started the cut-and-paste process of combining them.  It turned out pretty well.  I have several web-like interfaces to the Wemo code that I can interact with, and so far, it's holding up pretty well.  For example, I have a human interface that is really easy to use; I just type in 192.168.0.205:51001 as a URL and I get back:

Current Wemo Light Switch Status

Front Porch is Off  
Outside Garage Lights are Off  
Cactus Spot Lights are Off  
West Patio Lights are Off  

The status is immediate, meaning I actually send off a UPNP request and get the state.  The buttons work and will operate the lights.  It's basically a tiny web server to control the lights without all the hassle of bringing up Apache and a ton of other things.  Yes, it could be extended to use pictures and really cool looking interfaces, but that wasn't what I was going for, this is a nice side effect.

The machine interface is different.  If I type in 192.168.0.205:51001/status, I get:

[{"cactusspot": "Off"}, {"frontporch": "Off"}, {"patio": "Off"}, {"outsidegarage": "Off"}]

Just the bare essentials: the names of the switches and their states right now.  Notice that I have to type in less characters for the human interaction than I do for the machine interaction?  Yep, that's on purpose, I'm lazy.  If you want to see how lazy I am, I have to type in:

192.168.0.205:51001/pCommand?command=OutsideLightsOn

To turn on a single light with the process interface, and it doesn't return anything because it updates the SQLite3 database I use instead.  Sure, it's a painful thing to do, but who cares?  I don't have to type it, it's constructed by code.  To illustrate how a web page can be constructed, here is the code for the human readable page I put up:

    @cherrypy.expose
    def index(self):
        status = "<strong>Current Wemo Light Switch Status</strong><br /><br />"
        status += "Front Porch is " + get("frontporch") + "&nbsp;&nbsp;"
        status += '<a ><button>Toggle</button></a>'
        status += "<br />"
        status += "Outside Garage Lights are " + get("outsidegarage") + "&nbsp;&nbsp;"
        status += '<a ><button>Toggle</button></a>'
        status += "<br />"
        status += "Cactus Spot Lights are " + get("cactusspot") + "&nbsp;&nbsp;"
        status += '<a "><button>Toggle</button></a>'
        status += "<br />"
        status += "West Patio Lights are " + get("patio") + "&nbsp;&nbsp;"
        status += '<a "><button>Toggle</button></a>'
        status += "<br />"
        return status

The entire page is constructed as a string and just sent back to my browser.  I used the simplest buttons and web techniques I could to make it as easy as possible.  Still, it works and doesn't look bad.  Notice that when you click on a button, it goes for the URL ./wemocommand ?  That's the code that will toggle a particular light.  I also use URL parameters (that's what the '?' is for) because they turned out to be easy to parse out.  I could have made this smaller by using a list, and I may some day, but this is much easier to understand for folk that are trying to copy the idea.  It also makes it simple to code for acting on the incoming HTML request:

    @cherrypy.expose
    def wemocommand(self, whichone):
        # first change the light state
        toggle(whichone)
        # now reload the index page to tell the user
        raise cherrypy.InternalRedirect('/index')

Yes, that's all there was to it.  I already had the toggle() routine to support the old interface and I simply redisplay the first page because it will go get the new state of the switches.  At this point in my experiment, I was beginning to really like CherryPy.

Just to round out this a bit, here is the code for the JSON string return I do for the process interface:

    @cherrypy.expose
    @cherrypy.tools.json_out() # This allows a dictionary input to go out as JSON
    def status(self):
        status = []
        for item in lightSwitches:
            status.append({item["name"]:get(item["name"])})
        return status

Notice that CherryPy already has a JSON translator built in?  Is that cool or what?  All I had to do was get the lights state in a list and then return it.  It makes the code a little hard to understand, but it was so simple to put together.

If you want to see the entire module, it's under the branch wemoHTML in my GitHub repository.  If you haven't figured out how to get to that yet (took me some time), here's a link <link> that should get you directly to it.  The module is wemocontrol.py in the house directory.

I still haven't adapted the other pieces necessary to fully implement the idea.  I have to change the web code (php) that handles commands and also the presentation page for my control web page.  I don't expect to have too much trouble with it, but I'll probably have to learn the meaning of some new words.

Saturday, November 15, 2014

Taking Another Look at Process Communication - CherryPy

I've been using SysV interprocess communication between the various processes that control and monitor the devices around the house, and it's been working fine.  But, at some point I'm going to want to separate the processes into different machines.  A Pi for the lights, a Pi for the devices I hope to have somewhere else, and a pet project of mine, a Pi to control the functions of the pool.  SysV doesn't talk between machines.  I can use TCP for this, but I don't want to invent a protocol on top of it, I want something I can just pick up and use.  How about HTTP?

I could bring up a web server on each machine and hook it into the processes that are involved, but that would take a bunch of research.  So I got to looking around and there were two nice solutions, Tornado <link> and CherryPy <link>.  They're both nice, but CherryPy is simpler to use, so I chose it as my first experiment.  Besides, it's cool to be running CherryPy on a Raspberry Pi, ... right?

It came up on the first try using their tutorial and worked.  That looks promising, but what about doing something else besides just waiting for an HTTP request?  How to have several of them running on the same machine talking between processes?  How do I keep track of the various addresses?

In my usual form, I just started writing code to see what would happen.  CherryPy uses a publish, subscribe philosophy, and provides some basic published items.  I subscribed to their 'main' and used that to run my tiny timer, and suddenly, it could do something besides just wait for HTML to come in.  So, I have my timers, the ability to periodically check device status and probably do some other stuff.  Now would be a good time to expand the experiment.

The idea is to be able to communicate with each control or monitor process to see what is going on with a browser from my recliner as well as a running process that accumulates everything for presentation.  Hopefully, this will make it much easier to tell what is going on when something quits working properly.  I hope to use JSON as the data transfer protocol and then we'll be able to read it as well as use it easily in code. Eventually, I want to be able to separate the processes between Raspberry Pi's and have them keep working.  My very own tiny data center setting on a shelf over there somewhere.

But, I'm a bit sick of changing things, breaking them, and not having a clue what I did to break them.  Enter GitHub <link>.  I've put my current Raspberry Pi source on GitHub so I can track changes and back out things that just didn't work well.  I can also create a fork for massive experiments, load them on another Pi and play until it works.  A side effect is that you folk can grab the source without having to copy and paste from a window.  There have been many problems with python and its indentation rules, this should fix that.  Also, if you find a bug, you can fix it and then I can grab it as well.

The latest test of my move to CherryPy is in the file cherrytest.py in the other-stuff directory in my GitHub repository <link>.  The house directory is where I put the code I'm running right now on my Pi to control the house.  When I start testing using CherryPi to communicate, I'll fork off a new version and play there until I can tell if it is going to work out.

My very own little sandbox to play in.

Monday, November 10, 2014

Wemo Light Switches and Intermittent Failures ... Found it!

I have four Wemo light switches operating around the house on outside lights.  You know the ones.  They're the switches that you remember after you get in bed and have the covers pulled up.  In my case, I just grab a phone and turn the darn things off from the bed ... except ...

Every once in a while they don't respond.  This has driven me nuts ever since I got the switches and I'm not alone.  There are other folk out there that have been annoyed by this behavior.  After I simplified my code running the switches <link>, and the problem was still there; I set out to see what the heck was going on.  At first the code would hang waiting for a response from the switches, so I put a timeout on the socket read.

When the inevitable timeout came, I at least had a clue; the switch just quit responding.  So, I caught the timeout and printed the URL that was involved.  A few days later, it timed out again and I discovered that the IP address was ok, but the port number the Wemo was listening to had changed.  There's simply no reason for the port number to change ... unless the switch had done it intentionally.  Naw, no one would do that.

Then, the power failed to my house and everything rebooted.  Fine, I'd wait until it happened again.

One evening, one of the switches failed.  I got an email telling me the switch had gone off line.  Yes, an email.  One of the great things about working on a Pi is that it can do things like this.  I had put code in to send me an email when one of the switches got into this state.  I checked the logs (another great thing) and sure enough, the port number had changed.  A little while later, another switch failed, then the other two as well.  The period of working was roughly three days, and then, the switches would change the port number they responded to.

I don't have definitive proof, but I firmly believe they reboot the switches every three days.  Over time I could probably accumulate evidence to support this, but I don't want evidence; I want the darn switches to work.

What I had done in the code was to set a timeout on the socket receive from a polling request to the switch.  When it timed out, it would send me mail and then my process would exit.  Since the process is controlled by upstart (google it), it would start right back up and rediscover the switches.  Everything should be fine then because the new port number would be recorded and control would be set up correctly.  I actually rebooted the control process to just start over.  Not very elegant, but it worked.  When the four switches went south, the process restarted each time and got control back.

Ha, take that Belkin.

Of course there's a downside.  During the time the process is restarting, there could be a command to the switches missed.  It takes about 30 seconds to discover the switches and get all set up so the window is small, but it's there.  I think I can live with that.  I removed the code that sends mail to keep the changes I make to Miranda to an absolute minimum and have the changes running right now.  After a couple of weeks, I'll know how well it worked out.

It's very important to me to avoid relying on an external service to control something around my house, so things like If This Then That, are not an option.  Sure, I like the Wemo app for checking the lights from town, but I don't want to rely on it.  The Internet is just too flaky, companies change policies, and people like to muck with other people's stuff.  I want control inside my own house.

I'm getting closer now.

Sunday, November 9, 2014

OK, Now I Have a Question I Need Help With

I ran across a forum discussion that got me to thinking <link>.  I was researching (again) measuring power with some non intrusive method when I ran into a typical snarky set of responses to a person showing a possible solution.  The author of the post showed this picture of a clamp current meter and some wiring:
Granted, it looks like it won't measure anything since both lines go through the current clamp, but I'm here to tell you this works.

Yep, if the direction is reversed using a loop in the wire it will measure total current through the two lines.  I actually tried it with a current clamp and some 14 gauge wires to a 220V light bulb in an Edison socket.  As further proof, here's a picture of the wiring inside my smart meter:
Look closely, you can zoom if you need to, but notice how one of the power legs goes through the current transformer in the opposite direction?  Here's another picture of the wires removed from the meter:
One current transformer and two wires to measure the total current being used by my house.  I asked the guy from the power company why they didn't just put the current transformer around the neutral and he said people could avoid part of the bill by referencing to ground avoiding the neutral path.  That actually makes sense in a very unsafe fashion.  There's a number of things you can do with current transformers that I hadn't thought of until now.  You can run both wires through in the same direction and tell if there is current to ground; that's how the GFCI you have in the bathroom works.  You can loop a wire around the transformer and double the voltage out to measure smaller currents.  Heck, there's a bunch of configurations that can help us measure things if we learn how they work.

My question is, how the heck does this work?  I know it does, I've done it, and the meter manufacturer has also.  I just can't get my head around the physics of it.

This also points out how folk on the web jump on someone who is correct and belittle them.  I guess we've all been there some time or other.  The author of the picture showed a heck of a lot more restraint than I would have in the same circumstances.

Saturday, November 8, 2014

Adding a USB Stick to the Raspberry Pi

If you beat the heck out of your SD card on a Raspberry Pi by having code that constantly updates a database, you're eventually going to start having problems with the card.  These little SD cards are tough little critters and (some of them) work really well, but they have a finite life.

I started noticing increased problems with database locks, Occasional corrupted data, the kind of thing that goes along with the SD card starting to fail.  It wasn't a really bad problem, but I don't want to have to deal with a bad problem, so I got one of these:


I picked this particular one after reading about a thousand reviews of various USB sticks and took the plunge for this particular device.  The idea is that I wanted one that wouldn't die in a couple of days, would serve other purposes if I decided to do something else on the Pi, and had enough capacity to be relevant for a long time.  16GB should last me a while on the Pi, and if it became photo storage later, it would handle a lot of them.

I let it set on the shelf for quite a while before I finally decided to use it as the main storage on my Pi.  It's Saturday, I really don't want to dig on the drainage trench I've been working on for two weeks, and the neighbors blocked my driveway with a backhoe, so it's time to do this.

When I went looking on the web for instructions to do this, there were literally thousands of them out there.  Most would work, but I don't want to do most of it on the PC and then stick the device on the Pi and hope.  I was amazed at how many Mac instructions there are; I thought Mac users were all artists that couldn't understand real computers (heh).  But I found a set of simple instructions created by pauly that fit the bill nicely <link>.

Since I'm a coward, I backed up the SD card first, then rebooted the Pi and started down the instructions.  Every single step worked exactly as pauly described, and in about an hour I rebooted using the SD card for the boot partition and the USB card for the main file system.  I didn't do anything about expanding swap space yet, nor did I mess with removing the old file system on the SD card.  Those may become useful at some point, but I don't need to mess with it now.

The Pi booted faster, but that's not very important since I don't boot it very often.  I noticed a slight decrease in load average, and time will tell if I see a decrease in the number of problems I've been having.  It's only been up for a couple of hours so I can't speak from experience yet.

I do need to look at the mount point I used for the usb stick though.  I used the default /dev/sda for it, and if I plug in another USB stick, I may have a problem with the default name.  There's the by-id device to look at and I should use it so the name is constant.  But, for now, I'll just get some experience with the setup and see if it holds together.  That's mostly why I have a backup and also kept the file system on the SD card.

This particular Pi has two XBees on it.  As I mentioned in previous posts, I had to do this because the default Digi network and Lowe's Iris devices don't play well together.  This was the third device I added to the USB buss; so I have a powered hub in place to take care of the fact that my USB buss probably uses more power than the computer that runs it.  I'm currently using one of these:
It has seven ports; I hope I don't need all of them, and came with a power supply that can handle that many devices.  I've been bitten by devices like this that just can't cut it when I plugged in a few power hungry peripherals.  It's really annoying to have to hunt down a power supply when you want to try something.  This little guy has worked well.

So, here's a possibility for you folk that are nearing the failure point of the SD card.  Most people that play with the Pi's don't run them 24x7 for months on end with a database and web server beating on things, so they'll never get to this point.

But, some of us do.

Thursday, October 30, 2014

ZIgBee Protocol, XBee, but this time an Arduino !

A couple of days ago I posted how to get a switch using the ZigBee protocol running with an XBee <link>; I'm pretty proud of that, but there's lots of folk out there with an Arduino already.  What about them?  Since I love those little board, I ported the code over to it as well.  See, the protocol is huge and extensive, but once you know how to operate a device that uses the protocol, you don't need much code to keep doing it.  That makes the little Arduino a cool device for controlling a ZigBee switch.

This code is the rough equivalent of the code I posted previously for the Raspberry Pi, same selections, except I left out the one that interrogates the switch, and let it report when the switch joins with the Arduino.  So, go look at the explanation I've already done to see how it works and how to use it.  Here's the Arduino Sketch:

/**
This is an implementation of Zigbee device communication using an XBee
and a Centralite Smart Switch 4256050-ZHAC

dave  (www.desert-home.com)
*/
 
// This code will handle both a Uno and a Mega2560 by careful use of
// the defines.  I tried it on both of them, and the only problem is that
// SoftwareSerial sometimes loses characters because the input buffer
// is too small.  If you have this problem, see the SoftwareSerial 
// documentation to see how to change it.
#include <XBee.h>
//#include <SoftwareSerial.h>
#include <Time.h>
#include <TimeAlarms.h>

// create reusable objects for messages we expect to handle
// using them over and over saves memory instead of sucking it off the
// stack every time we need to send or receive a message by creating
// a new object
XBee xbee = XBee();
XBeeResponse response = XBeeResponse();
ZBExpRxResponse rx = ZBExpRxResponse();
ZBExpCommand tx;
XBeeAddress64 Broadcast = XBeeAddress64(0x00000000, 0x0000ffff);

// Define the hardware serial port for the XBee (mega board)
#define ssRX 2
#define ssTX 3
// Or define NewSoftSerial TX/RX pins
// Connect Arduino pin 2 to Tx and 3 to Rx of the XBee
// I know this sounds backwards, but remember that output
// from the Arduino is input to the Xbee
//SoftwareSerial nss(ssRX, ssTX);

XBeeAddress64 switchLongAddress;
uint16_t switchShortAddress;
uint16_t payload[50];
uint16_t myFrameId=1;

void setup() {  
  // start serial
  Serial.begin(9600);
  // and the software serial port
  //nss.begin(9600);
  // Or the hardware serial port
  Serial1.begin(9600);
  // now that they are started, hook the XBee into 
  // whichever one you chose
  //xbee.setSerial(nss);
  xbee.setSerial(Serial1);
  setTime(0,0,0,1,1,14);  // just so alarms work well, I don't really need the time.
  Serial.println("started");
}

boolean firstTime = true;

void loop() {
  // Since this test code doesn't have the switch address, I'll
  // send a message to get the routes to the devices on the network
  // All devices are supposed to respond to this, and even the
  // XBee we're hooked to will respond for us automatically
  // The second message in will be the switch we want to work
  // with.  Thus, giving us the address we need to do things with
  if (firstTime){
    Serial.println(F("Wait while I locate the device"));
  // First broadcast a route record request so when the switch responds
  // I can get the addresses out of it
    Serial.println(F("Sending Route Record Request"));
    uint8_t rrrPayload[] = {0x12,0x01};
    tx = ZBExpCommand(Broadcast, //This will be broadcast to all devices
      0xfffe,
      0,    //src endpoint
      0,    //dest endpoint
      0x0032,    //cluster ID
      0x0000, //profile ID
      0,    //broadcast radius
      0x00,    //option
      rrrPayload, //payload
      sizeof(rrrPayload),    //payload length
      0x00);   // frame ID
    xbee.send(tx);
  firstTime = false;
  }

  // First, go check the XBee, This is non-blocking, so 
  // if nothing is there, it will just return.  This also allows
  // any message to come in at any time so thing can happen 
  // automatically.
  handleXbee();
  // After checking the XBee for data, look at the serial port
  // This is non blocking also.
  handleSerial();
  // Now, update the timer and do it all over again.
  // This code tries to not wait for anything.  It keeps it
  // from hanging up unexpectedly.  This way we can implement a 
  // watchdog timer to take care of the occasional problem.
  Alarm.delay(0); // Just for the alarm routines
}

void handleSerial(){
  if (Serial.available() > 0) {
    char incomingByte;
    
    incomingByte = Serial.read();
    // Originally, I had a routine to send messages, but it tended to hide 
    // the way the messages were constructed from new folk.  I changed it
    // back to a verbose construction of each message sent to control the
    // switch so people could more easily understand what they needed to do
    if (isdigit(incomingByte)){
      Serial.print("Selection: ");
      int selection = atoi(&incomingByte);
      Serial.print(selection, DEC);
      switch(selection){
        case 0: { // switch off
          Serial.println(F(" Turn switch off"));
          // In these outgoing messages I set the transaction sequence
          // number to 0xaa so it could be easily seen if I was dumping
          // messages as they went out.
          uint8_t offPayload[] = {0x11,0xaa,0x00};
          tx = ZBExpCommand(switchLongAddress,
            switchShortAddress,
            0,    //src endpoint
            1,    //dest endpoint
            0x0006,    //cluster ID
            0x0104, //profile ID
            0,    //broadcast radius
            0x00,    //option
            offPayload, //payload
            sizeof(offPayload),    //payload length
            0x00);   // frame ID
            xbee.send(tx);
          break;
        }
        case 1: { // switch on
          Serial.println(F(" Turn switch on"));
          uint8_t onPayload[] = {0x11,0xaa,0x01};
          tx = ZBExpCommand(switchLongAddress,
            switchShortAddress,
            0,    //src endpoint
            1,    //dest endpoint
            0x0006,    //cluster ID
            0x0104, //profile ID
            0,    //broadcast radius
            0x00,    //option
            onPayload, //payload
            sizeof(onPayload),    //payload length
            0x00);   // frame ID
          xbee.send(tx);
          break;
        }
        case 2: { // switch toggle
          Serial.println(F(" Toggle switch"));
          uint8_t togglePayload[] = {0x11,0xaa,0x02};
          tx = ZBExpCommand(switchLongAddress,
            switchShortAddress,
            0,    //src endpoint
            1,    //dest endpoint
            0x0006,    //cluster ID
            0x0104, //profile ID
            0,    //broadcast radius
            0x00,    //option
            togglePayload, //payload
            sizeof(togglePayload),    //payload length
            0x00);   // frame ID
          xbee.send(tx);
          break;
        }
        case 3: {
          Serial.println(F(" Dim"));
          uint8_t dimPayload[] = {0x11,0xaa,0x00,25,0x32,0x00};
          tx = ZBExpCommand(switchLongAddress,
            switchShortAddress,
            0,    //src endpoint
            1,    //dest endpoint
            0x0008,    //cluster ID
            0x0104, //profile ID
            0,    //broadcast radius
            0x00,    //option
            dimPayload, //payload
            sizeof(dimPayload),    //payload length
            0x00);   // frame ID
          xbee.send(tx);
          break;
        }
        case 4: {
          Serial.println(F(" Bright"));
          uint8_t brightPayload[] = {0x11,0xaa,0x00,255,0x32,0x00};
          tx = ZBExpCommand(switchLongAddress,
            switchShortAddress,
            0,    //src endpoint
            1,    //dest endpoint
            0x0008,    //cluster ID
            0x0104, //profile ID
            0,    //broadcast radius
            0x00,    //option
            brightPayload, //payload
            sizeof(brightPayload),    //payload length
            0x00);   // frame ID
          xbee.send(tx);
          break;
        }
        case 5: {
          Serial.println(F(" Get State of Light "));
          uint8_t ssPayload[] = {0x00,0xaa,0x00,0x00,0x00};
          tx = ZBExpCommand(switchLongAddress,
            switchShortAddress,
            0,    //src endpoint
            1,    //dest endpoint
            0x0006,    //cluster ID
            0x0104, //profile ID
            0,    //broadcast radius
            0x00,    //option
            ssPayload, //payload
            sizeof(ssPayload),    //payload length
            0x00);   // frame ID
          xbee.send(tx);
          break;
        }
        
        default:     
          Serial.println(F(" Try again"));
          break;
      }
      // Now a short delay combined with a character read
      // to empty the input buffer.  The IDE developers removed
      // the input flush that used to work for this.
      while(Serial.available() > 0){
        char t = Serial.read();
        delay(25);
      }
    }
  }
}
void handleXbee(){
  // doing the read without a timer makes it non-blocking, so
  // you can do other stuff in loop() as well.  Things like
  // looking at the console for something to turn the switch on
  // or off 
  xbee.readPacket();
  // the read above will set the available up to 
  // work when you check it.
  if (xbee.getResponse().isAvailable()) {
    // got something
    //Serial.println();
    //Serial.print("Frame Type is ");
    // Andrew called the XBee frame type ApiId, it's the first byte
    // of the frame specific data in the packet.
    int frameType = xbee.getResponse().getApiId();
    //Serial.println(frameType, HEX);
    //
    // All ZigBee device interaction is handled by the two XBee message type
    // ZB_EXPLICIT_RX_RESPONSE (ZigBee Explicit Rx Indicator Type 91)
    // ZB_EXPLICIT_TX_REQUEST (Explicit Addressing ZigBee Command Frame Type 11)
    // This test code only uses these and the Transmit Status message
    //
    if (frameType == ZB_EXPLICIT_RX_RESPONSE) {
      // now that you know it's a Zigbee receive packet
      // fill in the values
      xbee.getResponse().getZBExpRxResponse(rx);
      int senderProfileId = rx.getProfileId();
      // For this code, I decided to switch based on the profile ID.
      // The interaction is based on profile 0, the general one and
      // profile 0x0104, the Home Automation profile
      //Serial.print(F(" Profile ID: "));
      //Serial.print(senderProfileId, HEX);
      
      // get the 64 bit address out of the incoming packet so you know 
      // which device it came from
      //Serial.print(" from: ");
      XBeeAddress64 senderLongAddress = rx.getRemoteAddress64();
      //print32Bits(senderLongAddress.getMsb());
      //Serial.print(" ");
      //print32Bits(senderLongAddress.getLsb());
      
      // this is how to get the sender's
      // 16 bit address and show it
      uint16_t senderShortAddress = rx.getRemoteAddress16();
      //Serial.print(" (");
      //print16Bits(senderShortAddress);
      //Serial.println(")");
      
      // for right now, since I'm only working with one switch
      // save the addresses globally for the entire test module
      switchLongAddress = rx.getRemoteAddress64();
      switchShortAddress = rx.getRemoteAddress16();

      uint8_t* frameData = rx.getFrameData();
      // We're working with a message specifically designed for the
      // ZigBee protocol, see the XBee documentation to get the layout
      // of the message.
      //
      // I have the message and it's from a ZigBee device
      // so I have to deal with things like cluster ID, Profile ID
      // and the other strangely named fields that these devices use
      // for information and control
      //
      // I grab the cluster id out of the message to make the code
      // below simpler.
      //Serial.print(F(" Cluster ID: "));
      uint16_t clusterId = (rx.getClusterId());
      //print16Bits(clusterId);
      //
      // Note that cluster IDs have different uses under different profiles
      //  First I'll deal with the general profile.
      if (senderProfileId == 0x0000){  // This is the general profile
        if (clusterId == 0x00){
          //Serial.println(F(" Basic Cluster"));
          pass();
        }
        else if (clusterId == 0x0006){ // Match Descriptor
          //Serial.println(F(" Match Descriptor"));
          /*************************************/
          // I don't actually need this message, but it comes in as soon as
          // a device is plugged in.  I answer it with a messsage that says I
          // have an input cluster 0x19, since that's what it's looking for.
          // Ignoring this message doesn't seem to hurt anything either.
          uint8_t mdPayload[] = {0xAA,0x00,0x00,0x00,0x01,0x19};
          mdPayload[2] = switchShortAddress & 0x00ff;
          mdPayload[3] = switchShortAddress >> 8;
          ZBExpCommand tx = ZBExpCommand(switchLongAddress,
            switchShortAddress,
            0,    //src endpoint
            0,    //dest endpoint
            0x8006,    //cluster ID
            0x0000, //profile ID
            0,    //broadcast radius
            0x00,    //option
            mdPayload, //payload
            sizeof(mdPayload),    //payload length
            0x00);   // frame ID
          xbee.send(tx);
          // if you unplug a device, and then plug it back in, it loses the 
          // configuration for reporting on/off changes.  So, send the configuration
          // to get the switch working the way I want it to after the match
          // descriptor message.  
          Serial.println (F("sending cluster command Configure Reporting "));
          uint8_t crPayload[] = {0x00,0xaa,0x06,0x00,0x00,0x00,0x10,0x00,0x00,0x00,0x40,0x00,0x00};
          tx = ZBExpCommand(switchLongAddress,
            switchShortAddress,
            0,    //src endpoint
            1,    //dest endpoint
            0x0006,    //cluster ID
            0x0104, //profile ID
            0,    //broadcast radius
            0x00,    //option
            crPayload, //payload
            sizeof(crPayload),    //payload length
            0x00);   // frame ID
          xbee.send(tx);
          
       }
        else if (clusterId == 0x0013){  //device announce message
          // any time a new device joins a network, it's supposed to send this
          // message to tell everyone its there.  Once you get this message,
          // you can interogate the new device to find out what it is, and 
          // what it can do.
          Serial.println(F(" Device Announce Message"));
          switchLongAddress = rx.getRemoteAddress64();
          switchShortAddress = rx.getRemoteAddress16();
          // Ok we saw the switch, now just for fun, get it to tell us
          // what profile it is using and some other stuff.
          // We'll send an Acttive Endpoint Request to do this
          Serial.println (F("sending Active Endpoint Request "));
          // The active endpoint request needs the short address of the device
          // in the payload.  Remember, it needs to be little endian (backwards)
          // The first byte in the payload is simply a number to identify the message
          // the response will have the same number in it.
          uint8_t aePayload[] = {0xAA,0x00,0x00};
          aePayload[1] = switchShortAddress & 0x00ff;
          aePayload[2] = switchShortAddress >> 8;
          ZBExpCommand tx = ZBExpCommand(switchLongAddress,
            switchShortAddress,
            0,    //src endpoint
            0,    //dest endpoint
            0x0005,    //cluster ID
            0x0000, //profile ID
            0,    //broadcast radius
            0x00,    //option
            aePayload, //payload
            sizeof(aePayload),    //payload length
            0xaa);   // frame ID
          xbee.send(tx);
        }
        else if (clusterId == 0x8004){
          Serial.println(F(" Simple Descriptor Response "));
          // Since I've been through this switch a few times, I already know
          // what to expect out of it.  This response is how you get the actual 
          // clusters that it has code for, and the profile ID that it supports.
          // Since this is a light switch, it will support profile 0x104 and have
          // clusters that support things like on/off and reporting.
          // The items of interest are in the rf_data payload, and this is one way
          // to get them out.
          unsigned char *data = rx.getRFData();  // first get a pointer to the data
          Serial.print(F(" Transaction ID: "));
          print16Bits(data[0]); // remember the number that starts the payload?
          Serial.println();
          Serial.print(F(" Endpoint Reported: "));
          print8Bits(data[5]);
          Serial.println();
          Serial.print(F(" Profile ID: "));
          print8Bits(data[7]);  // Profile ID is 2 bytes long little endian (backwards)
          print8Bits(data[6]);
          Serial.println();
          Serial.print(F(" Device ID: "));
          print8Bits(data[9]);  // Device ID is 2 bytes long little endian (backwards)
          print8Bits(data[8]);
          Serial.println();
          Serial.print(F(" Device Version: "));
          print8Bits(data[10]);  // Device ID is 1 byte long
          Serial.println();
          Serial.print(F(" Number of input clusters: "));
          print8Bits(data[11]);  // Input cluster count
          Serial.print(F(", Clusters: "));
          Serial.println();
          for (int i = 0; i < data[11]; i++){
            Serial.print(F("    "));
             print8Bits(data[i*2+13]); // some more of that little endian crap
             print8Bits(data[i*2+12]);
             Serial.println();
          }
          int outidx = 11 + 1 + 2*data[11];
          Serial.print(F(" Number of output clusters: "));
          print8Bits(data[outidx]);  // Input cluster count
          Serial.print(F(", Clusters: "));
          Serial.println();
          for (int i = 0; i < data[outidx]; i++){
            Serial.print(F("    "));
             print8Bits(data[i*2 + outidx + 2]); // some more of that little endian crap
             print8Bits(data[i*2 + outidx + 1]);
             Serial.println();
          }
          Serial.println (F("sending cluster command Configure Reporting "));
          // OK, for illustration purposes, this is enough to actually do something
          // First though, let's set up the switch so that it reports when it has 
          // changed states in the on/off cluster (cluster 0006).  This will require we
          // send a message to the on/off cluster with the "Configure Reporting" command
          // (0x06) with a bunch of parameters to specify things.
          uint8_t crPayload[] = {0x00,0xaa,0x06,0x00,0x00,0x00,0x10,0x00,0x00,0x00,0x40,0x00,0x00};
          ZBExpCommand tx = ZBExpCommand(switchLongAddress,
            switchShortAddress,
            0,    //src endpoint
            1,    //dest endpoint
            0x0006,    //cluster ID
            0x0104, //profile ID
            0,    //broadcast radius
            0x00,    //option
            crPayload, //payload
            sizeof(crPayload),    //payload length
            0x00);   // frame ID
          xbee.send(tx);

        } 
        else if (clusterId == 0x8005){
          Serial.println(F(" Active Endpoints Response"));
          // This message tells us which endpoint to use
          // when controlling the switch.  Since this is only a switch,
          // it will give us back one endpoint.  I should really have a loop
          // in here to handle multiple endpoints, but ...
          Serial.print(F("  Active Endpoint Count reported: "));
          Serial.println(rx.getRFData()[4]);
          Serial.print(F("  Active Endpoint: "));
          Serial.println(rx.getRFData()[5]);
          // Now we know that it has an endpoint, but we don't know what profile
          // the endpoint is under.  So, we send a Simple Descriptor Request to get
          // that.
          Serial.println (F("sending Simple Descriptor Request "));
          // The request needs the short address of the device
          // in the payload.  Remember, it needs to be little endian (backwards)
          // The first byte in the payload is simply a number to identify the message
          // the response will have the same number in it.  The last number is the
          // endpoint we got back in the Active Endpoint Response.  
          // Also note that we're still dealing with profile 0 here, we haven't gotten
          // to the device we actually want to play with yet.
          uint8_t sdPayload[] = {0xAA,0x00,0x00,01};
          sdPayload[1] = switchShortAddress & 0x00ff;
          sdPayload[2] = switchShortAddress >> 8;
          sdPayload[3] = rx.getRFData()[5];
          ZBExpCommand tx = ZBExpCommand(switchLongAddress,
            switchShortAddress,
            0,    //src endpoint
            0,    //dest endpoint
            0x0004,    //cluster ID
            0x0000, //profile ID
            0,    //broadcast radius
            0x00,    //option
            sdPayload, //payload
            sizeof(sdPayload),    //payload length
            0xaa);   // frame ID
          xbee.send(tx);
        }
        else if (clusterId == 0x8032){
          Serial.print(" Response from: ");
          print16Bits(senderShortAddress);
          Serial.println();
          if(switchShortAddress != 0x0000){
            Serial.print(F("Got switch address "));
            Serial.println(F("Ready"));
          }
        }
        else{
          Serial.print(F(" Haven't implemented this cluster yet: "));
          Serial.println(clusterId,HEX);
        }
      }
      else if(senderProfileId == 0x0104){   // This is the Home Automation profile
        // Since these are all ZCL (ZigBee Cluster Library) messages, I'll suck out
        // the cluster command, and payload so they can be used easily.
        //Serial.println();
        //Serial.print(" RF Data Received: ");
        //for(int i=0; i < rx.getRFDataLength(); i++){
            //print8Bits(rx.getRFData()[i]);
            //Serial.print(' ');
        //}
        //Serial.println();
        if (clusterId == 0x0000){
          //Serial.print(F(" Basic Cluster"));
          pass();
        }
        else if (clusterId == 0x0006){ // Switch on/off 
          // Serial.println(F(" Switch on/off"));
          // with the Centralite switch, we don't have to respond
          // A message to this cluster tells us that the switch changed state
          // However, if the response hasn't been configured, it will give back 
          // default response (cluster command 0b)
          // so let's dig in and see what's going on.
          //
          // The first two bytes of the rfdata are the ZCL header, the rest of
          // the data is a three field indicator of the attribute that changed
          // two bytes of attribute identifier, a byte of datatype, and some bytes
          // of the new value of the attribute.  Since this particular attribute is a 
          // boolean (on or off), there will only be one byte.  So
          if(rx.getRFData()[2] == 0x0b){ // default response (usually means error)
            Serial.println(F("  Default Response: "));
            Serial.print(F("  Command: "));
            print8Bits(rx.getRFData()[3]);  
            Serial.println();
            Serial.print(F("  Status: "));
            print8Bits(rx.getRFData()[4]);  
            Serial.println();
          }
          else if (rx.getRFData()[2] == 0x0a || rx.getRFData()[2] == 0x01){
             // This is what we really want to know
            Serial.print(F("Light "));
            // The very last byte is the status
            if (rx.getRFData()[rx.getRFDataLength()-1] == 0x01){
              Serial.println(F("On"));
            }
            else{
              Serial.println(F("Off"));
            }
          }
          else{  // for now, the ones above were the only clusters I needed.
            //Serial.println(F("  I don't know what this is"));
            pass();
          }
        }
        else if (clusterId == 0x0008){ // This is the switch level cluster
          // right now I don't do anything with it, but it's where
          // the switch lets you know about level changes
        }
        else{
          Serial.print(F(" Haven't implemented this cluster yet: "));
          Serial.println(clusterId,HEX);
        }
      }
    }
    else {
      if (frameType == 0xa1){
        //Serial.println(F(" Route Record Request"));
        pass();
      }
      else if (frameType == ZB_TX_STATUS_RESPONSE){
        //Serial.print(F(" Transmit Status Response"));
        pass();
      }
      else{
        Serial.print(F("Got frame type: "));
        Serial.print(frameType, HEX);
        Serial.println(F(" I didn't implement this frame type for this experiment"));
      }
    }
  }
  else if (xbee.getResponse().isError()) {
    // some kind of error happened, I put the stars in so
    // it could easily be found
    Serial.print("************************************* error code:");
    Serial.println(xbee.getResponse().getErrorCode(),DEC);
  }
  else {
    // If you get here it only means you haven't collected enough bytes from
    // the XBee to compose a packet.
  }
}

/*-------------------------------------------------------*/
// null routine to avoid some syntax errors when debugging
void pass(){
    return;
}
// these routines are just to print the data with
// leading zeros and allow formatting such that it 
// will be easy to read.
void print32Bits(uint32_t dw){
  print16Bits(dw >> 16);
  print16Bits(dw & 0xFFFF);
}

void print16Bits(uint16_t w){
  print8Bits(w >> 8);
  print8Bits(w & 0x00FF);
}
  
void print8Bits(byte c){
  uint8_t nibble = (c >> 4);
  if (nibble <= 9)
    Serial.write(nibble + 0x30);
  else
    Serial.write(nibble + 0x37);
        
  nibble = (uint8_t) (c & 0x0F);
  if (nibble <= 9)
    Serial.write(nibble + 0x30);
  else
    Serial.write(nibble + 0x37);
}

Remember, on an Arduino the XBee API mode must be set to 2.

Have fun with it.

Tuesday, October 28, 2014

OK, Back to the ZigBee protocol and XBees ... AGAIN

I managed to hack into the Iris Smart Switch from Lowe's and they've been working fine, but there's always been this nagging little annoyance bothering me.  The Alertme Switch that Lowe's sells is NOT ZigBee compliant no matter what they may tell you.  In hacking at it I pointed out a few things that were not according to spec (yes, I've actually read that massive spec related to home automation), and I've been wondering what it would be like to work with an actual ZigBee device.

A reader set me up with one of these:


This is a Centralite 4256050-ZHAC and has an impressive set of capabilities.  They claim that it will work with any controller that is compliant with the ZigBee HA (Home Automation) specification.  That sounds like a challenge to me.  If it is compliant, I should be able to figure out how to work it using an XBee; so away I went.

After almost six DAYS of poking messages at the switch, I was only a tiny bit further along than I was when I started.  This silly thing simply wouldn't join with the XBee so I could see anything it did.  Then I stumbled across a note that said it had a special key.  Key?  It needs a key?  OK, I can try this.  It started working.  I was able to send something and get an answer back; sure the answer was an error, but it was an answer.  Thus began my exploration of the ZigBee protocol in earnest; I was going to make this switch work.

Once again, this isn't one of those posts where I tried for weeks and finally gave up because there just wasn't enough information out there, the machine was too slow, or someone kept something secret; I made it work and will give you the code and XBee configuration to follow in my footsteps down below.  But first I want to talk about the ZigBee protocol and its relationship to XBees a bit.

First, this is an incredibly complex protocol and not for the faint of heart.  Just learning some of the jargon is daunting, much less trying to put it to use.  Sure, there are libraries out there, but have you looked at the prices of those things?  I simply can't afford to mess with them at that price.  Also, the libraries are as hard to understand as the protocol, AND it has the overhead of the library that has to be learned also.  I kept wondering if the XBee somehow could help with this.  Turns out the XBee really can do ZigBee, there just aren't may people that have tried.  Actually, I couldn't find anyone besides me that actually had.

There are lots of pictures and explanations out there about the ideas behind ZigBee, and some of them are even correct, but it was still hard for me to understand.  Let me give you some basics.  The protocol has profiles, these are collections of specs and suggestions for the operation of a system.  Things like Home Automation, Electric Lights (that Hue thingie), Smart Energy, and a device can support one or more of these things.  I'm interested in Home Automation, they call it HA, and that's where I concentrated.  Within this they separate data that you read or change and call them attributes.  These attributes are put within clusters.  Don't get confused, this isn't really that tough.

Within the HA profile, there are some defined clusters and they have numbers that identify them.  Let's take cluster 0x0006, the on-off cluster.  This will have an attribute, the state of the device, and it is numbered 0x0000 and has a datatype of boolean; it tells you if the switch is on or off.  To read this attribute you send a command to the cluster asking for it and the cluster returns the number identifier, datatype and value of the attribute.  See, clusters have commands to operate on them and attributes that you can use.

To tell if the switch is on, send a cluster command 0x00 (read attribute) to cluster 0x0006 (on/off) and the device will send back a command 0x01 (read attribute response) to cluster 0x0006 with the attribute identifier, datatype, value.  Cool.

In the message you send, you also specify the endpoint you want the reply to be sent to and the endpoint you are sending to.  What's an endpoint?  An endpoint is simply a collection of clusters.  On the centralite switch, it has endpoints 0, the general one, and 1, specific to this device.  The general endpoint is where stuff that you want to deal with of a general nature goes and endpoint 1 is where you send stuff that deals with turning the light on and off.

Thus, you have to worry about profiles, endpoints, clusters, clusters commands, and attributes.  Actually it's not that bad, it's just hard to ferret out of the thousands of pages of documentation.  But, you ask, how does the XBee help me?  The XBee eliminates about half of the documentation from being necessary for us to mess with.  It handles all the interactions to set up a network, keep track of device routing, radio initialization, that stuff.  It also gives us a simpler (not simple) message format to use so we don't have to worry about the six or seven layers of protocol, we work totally at the application level and just let it do the rest.  Heck, it even handles the encryption for us.

Combine an XBee to handle the low level link stuff and our own code to handle the application, and you have a reasonable way of controlling these switches that are on the market.  Let me show you the command above in python:

zb.send('tx_explicit',
 dest_addr_long = switchLongAddr,
 dest_addr = switchShortAddr,
 src_endpoint = '\x00',
 dest_endpoint = '\x01',
 cluster = '\x00\x06', # cluster I want to deal with
 profile = '\x01\x04', # home automation profile
 data = '\x00'+'\xaa'+'\x00'+'\x00'+'\x00'
)

There's the long address, it's 32 bits long and doesn't change, ever.  The short address, it's 16 bits long and changes every time the switch joins with the controller; yes, even after a power failure.  The source endpoint.  This is zero because I didn't want to deal with more than one in my code; all the responses come back to endpoint zero.  The destination endpoint which is one on this switch. The cluster id of six as I mentioned above.  The profile 0x0104 which is the number for the HA protocol. And, some data.  The data is one byte of control bits, a byte transaction sequence number that I set to 0xaa so it would be easy recognize, the cluster command 0x00, and the attribute id of 0x0000.  The reason it is shown as ascii characters is a characteristic of the python XBee library implementation.

This may be confusing at first, but trust me, it actually makes sense once you get into it a bit.

This message will send a response to profile id 0x104, endpoint 00, cluster 0x0006, with a payload of 0x00, 0x00, 0x10, 01 if the light is on.  The first two bytes are the attribute id, the next byte is the datatype (0x10 means boolean) and the last byte is 1, meaning the switch is closed.

Are you getting an idea of how this works?  Now, I can hear you asking, "How the heck do I find out these values?"  They're documented in the Cluster Specification document, and there are messages that will itemize the endpoints and clusters within them that the device supports.  So, you send a message to the device to get the endpoints, it tells you what they are, then for each endpoint you ask what the attributes are and it responds.  You look at this stuff, see what you need and use it.

Actually makes sense in a deranged computer scientist sort of way.  But, let's talk about the setup for an XBee specifically to support the Home Automation profile.  That's what I wanted, to be able to turn this switch on and off.  First, it's different from the setup used on the Iris switch so don't think about that, take this as new.

Software Zigbee API Coordinator
Zigbee Stack Profile  (ZS) 2
Encryption Enable  (EE) 1
Encryption Options (EO) 0
Encryption Key  (KY) 5a6967426565416c6c69616e63653039
Network Encryption Key (NK) 0
API Enable (AP) 1
API Output Mode (AO) 3

Yes, you have to use the key.  That part took me the first week of messing with this to find out.  Of course, now that I know what to look for, it would take me about a minute to get it, but that's how we learn.  The difference in the encryption setup is what prevents this switch and the Iris switch from working with the same controller.  You can't have it both ways at once.  If anyone thinks of a way around this, let me know.

Once you have the XBee setup like this you can take the Centralite switch, press the button and hold it, then plug it in the wall.  When the led turns on, let go of the switch and it will join with the XBee automatically.  Yes, that's all there is to joining.  The two devices take care of it themselves and all you have to do is discover the device and start using it.  This is very different from the Alertme devices where we have to mess around with special secret commands to get it to work.  This device actually complies with the specification.

In the code below, I send a message asking for route information and grab the switch's address out of the response.  Then, I send it a command to set up reporting for the light and just wait for someone to tell the code what to do with the light. The commands are:

0 - Turn the switch off
1 - Turn the switch on
2 - Toggle the switch
3 - Dim the switch
4 - Brighten the switch
5 - Tell me the status of the switch
6 - Send messages and print responses about the switch.

Yes, the switch is capable of dimming a light.  The last command goes through a portion of the Zigbee discovery process to find out which endpoints are supported and what clusters and attributes are in them.  It's voluminous, but it's the first time I was actually able to see what the various buzz words actually represented.  This is the kind of thing I did to conquer the way the switch works.

#! /usr/bin/python

'''
This is an examination of a REAL ZigBee device.  The CentraLite 4256050-ZHAC

It has an impressive array of capabilities that I don't delve into in depth in
this examination, but it responds properly to the various ZigBee commands and holds
the clusters necessary to control a switch.

Nice little device
'''

# This is the super secret home automation key that is needed to 
# implement the HA profile.
# KY parameter on XBee = 5a6967426565416c6c69616e63653039
# Have fun

from xbee import ZigBee 
import logging
import datetime
import time
import serial
import sys, traceback
import shlex
from struct import *
'''
Before we get started there's a piece of this that drove me nuts.  Each message to a 
Zigbee cluster has a transaction sequence number and a header.  The transaction sequence
number isn't talked about at all in the Zigbee documentation (that I could find) and 
the header byte is drawn  backwards to everything I've ever dealt with.  So, I redrew 
the header byte so I could understand and use it:

7 6 5 4 3 2 1 0
      X          Disable Default Response 1 = don't return default message
        X        Direction 1 = server to client, 0 = client to server
          X      Manufacturer Specific 
              X  Frame Type 1 = cluster specific, 0 = entire profile
         
So, to send a cluster command, set bit zero.  If you want to be sure you get a reply, clearthe default response.  I haven't needed the manufacturer specific bit yet.
'''
switchLongAddr = '12'
switchShortAddr = '12'

'''
 This routine will print the data received so you can follow along if necessary
'''
def printData(data):
 for d in data:
  print d, ' : ',
  for e in data[d]:
   print "{0:02x}".format(ord(e)),
  if (d =='id'):
   print "({})".format(data[d]),
  print

def getAttributes(data, thisOne):
 ''' OK, now that I've listed the clusters, I'm going to see about 
 getting the attributes for one of them by sending a Discover
 attributes command.  This is not a ZDO command, it's a ZCL command.
 ZDO = ZigBee device object - the actual device
 ZCL = Zigbee cluster - the collection of routines to control it.
  
  frame control bits = 0b00 (this means a BINARY 00)
  manufacturer specific bit = 0, for normal, or one for manufacturer
  So, the frame control will be 0000000
  discover attributes command identifier = 0x0c
  
  then a zero to indicate the first attribute to be returned
  and a 0x0f to indicate the maximum number of attributes to 
  return.
 '''
 print "Sending Discover Attributes, Cluster:", repr(thisOne)
 zb.send('tx_explicit',
  dest_addr_long = data['source_addr_long'],
  dest_addr = data['source_addr'],
  src_endpoint = '\x00',
  dest_endpoint = '\x01',
  cluster = thisOne, # cluster I want to know about
  profile = '\x01\x04', # home automation profile
  # means: frame control 0, sequence number 0xaa, command 0c,
  # start at 0x0000 for a length of 0x0f
  data = '\x00' + '\xaa' + '\x0c'+ '\x00' + '\x00'+ '\x0f'
  )

# this is a call back function.  When a message
# comes in this function will get the data
def messageReceived(data):
 global switchLongAddr
 global switchShortAddr
 
 try:
  #print 'gotta packet',
  #printData(data)  # uncomment this to see the data returned
  
  # Since this is a test program, it will only support one switch
  # go get the long and short address out of the incoming packet
  # for more than one switch, this won't work
  switchLongAddr = data['source_addr_long']
  switchShortAddr = data['source_addr']
  
  if (data['id'] == 'rx_explicit'):
   #print "RF Explicit"
   #printData(data)
   clusterId = (ord(data['cluster'][0])*256) + ord(data['cluster'][1])
   print 'Cluster ID:', hex(clusterId),
   print "profile id:", repr(data['profile']),
   if (data['profile']=='\x01\x04'): # Home Automation Profile
    # This has to be handled differently than the general profile
    # each response if from a cluster that is actually doing something
    # so there are attributes and commands to think about.
    #
    # Since this is a ZCL message; which actually means this message is 
    # is supposed to use the ZigBee cluster library to actually do something
    # like turn on a light or check to see if it's on, the command way down
    # in the rf_data is important.  So, the commands may be repeated in
    # each cluster and do slightly different things
    #
    # I'm going to grab the cluster command out of the rf_data first so 
    # I don't have to code it into each cluster
    #print "take this apart"
    #print repr(data['rf_data'])
    if (data['rf_data'][0] == '\x08'): # was it successful?
     #should have a bit check to see if manufacturer data is here
     cCommand = data['rf_data'][2]
     print "Cluster command: ", hex(ord(cCommand))
    else:
     print "Cluster command failed"
     return
    # grab the payload data to make it easier to work with
    payload = data['rf_data'][3:] #from index 3 on is the payload for the command
    datatypes={'\x00':'no data',
       '\x10':'boolean',
       '\x18':'8 bit bitmap',
       '\x20':'unsigned 8 bit integer',
       '\x21':'unsigned 24 bit integer',
       '\x30':'8 bit enumeration',
       '\x42':'character string'}
    #print "Raw payload:",repr(payload)
    # handle these first commands in a general way
    if (cCommand == '\x0d'): # Discover Attributes
     # This tells you all the attributes for a particular cluster
     # and their datatypes
     print "Discover attributes response"
     if (payload[0] == '\x01'):
      print "All attributes returned"
     else:
      print "Didn't get all the attributes on one try"
     i = 1
     if (len(payload) == 1): # no actual attributes returned
      print "No attributes"
      return
     while (i < len(payload)-1):
      print "    Attribute = ", hex(ord(payload[i+1])) , hex(ord(payload[i])),
      try:
       print datatypes[payload[i+2]]
       i += 3
      except:
       print "I don't have an entry for datatype:", hex(ord(payload[i+2]))
       return
       
    if (clusterId == 0x0000): # Under HA this is the 'Basic' Cluster
     pass
    elif (clusterId == 0x0003): # 'identify' should make it flash a light or something 
     pass
    elif (clusterId == 0x0004): # 'Groups'
     pass
    elif (clusterId == 0x0005): # 'Scenes'  
     pass
    elif (clusterId == 0x0006): # 'On/Off' this is for switching or checking on and off  
     #print "inside cluster 6"
     if cCommand in ['\x0a','\x01']:
      # The very last byte tells me if the light is on.
      if (payload[-1] == '\x00'):
       print "Light is OFF"
      else:
       print "Light is ON"
     pass
    elif (clusterId == 0x0008): # 'Level'  
     pass
    else:
     print("Haven't implemented this yet")
   elif (data['profile']=='\x00\x00'): # The General Profile
    if (clusterId == 0x0000):
     print ("Network (16-bit) Address Request")
     #printData(data)
    elif (clusterId == 0x0008):
     # I couldn't find a definition for this 
     print("This was probably sent to the wrong profile")
    elif (clusterId == 0x0004):
     # Simple Descriptor Request, 
     print("Simple Descriptor Request")
     print("I don't respond to this")
     #printData(data)
    elif (clusterId == 0x0013):
     # This is the device announce message.
     print 'Device Announce Message'
     #printData(data)
     # This is a newly found device, so I'm going to tell it to 
     # report changes to the switch.  There are better ways of
     # doing this, but this is a test and demonstration
     print "sending 'configure reporting'"
     zb.send('tx_explicit',
      dest_addr_long = switchLongAddr,
      dest_addr = switchShortAddr,
      src_endpoint = '\x00',
      dest_endpoint = '\x01',
      cluster = '\x00\x06', # cluster I want to deal with
      profile = '\x01\x04', # home automation profile
      data = '\x00' + '\xaa' + '\x06' + '\x00' + '\x00' + '\x00' + '\x10' + '\x00' + '\x00' + '\x00' + '\x40' + '\x00' + '\x00'
     )
    elif (clusterId == 0x8000):
     print("Network (16-bit) Address Response")
     #printData(data)
    elif (clusterId == 0x8032):
     print "Route Record Response"
    elif (clusterId == 0x8038):
     print("Management Network Update Request");
    elif (clusterId == 0x8005):
     # this is the Active Endpoint Response This message tells you
     # what the device can do
     print 'Active Endpoint Response'
     printData(data)
     if (ord(data['rf_data'][1]) == 0): # this means success
      print "Active Endpoint reported back is: {0:02x}".format(ord(data['rf_data'][5]))
     print("Now trying simple descriptor request on endpoint 01")
     zb.send('tx_explicit',
      dest_addr_long = data['source_addr_long'],
      dest_addr = data['source_addr'],
      src_endpoint = '\x00',
      dest_endpoint = '\x00', # This has to go to endpoint 0 !
      cluster = '\x00\x04', #simple descriptor request'
      profile = '\x00\x00',
      data = '\x13' + data['source_addr'][1] + data['source_addr'][0] + '\x01'
     )
    elif (clusterId == 0x8004):
     print "simple descriptor response"
     try:
      clustersFound = []
      r = data['rf_data']
      if (ord(r[1]) == 0): # means success
       #take apart the simple descriptor returned
       endpoint, profileId, deviceId, version, inCount = \
        unpack('<BHHBB',r[5:12])
       print "    endpoint reported is: {0:02x}".format(endpoint)
       print "    profile id:  {0:04x}".format(profileId)
       print "    device id: {0:04x}".format(deviceId)
       print "    device version: {0:02x}".format(version)
       print "    input cluster count: {0:02x}".format(inCount)
       position = 12
       # input cluster list (16 bit words)
       for x in range (0,inCount):
        thisOne, = unpack("<H",r[position : position+2])
        clustersFound.append(r[position+1] + r[position])
        position += 2
        print "        input cluster {0:04x}".format(thisOne)
       outCount, = unpack("<B",r[position])
       position += 1
       print "    output cluster count: {0:02x}".format(outCount)
       #output cluster list (16 bit words)
       for x in range (0,outCount):
        thisOne, = unpack("<H",r[position : position+2])
        clustersFound.append(r[position+1] + r[position])
        position += 2
        print "        output cluster {0:04x}".format(thisOne)
       clustersFound.append('\x0b\x04')
       print "added special cluster"
       print "Completed Cluster List"
     except:
      print "error parsing Simple Descriptor"
      printData(data)
     print repr(clustersFound)
     for c in clustersFound:
      getAttributes(data, c) # Now, go get the attribute list for the cluster
    elif (clusterId == 0x0006):
     #print "Match Descriptor Request"
     # Match Descriptor Request
     #printData(data)
     pass
    else:
     print ("Unimplemented Cluster ID", hex(clusterId))
     print
   else:
    print ("Unimplemented Profile ID")
  elif(data['id'] == 'route_record_indicator'):
   #print("Route Record Indicator")
   pass
  else:
   print("some other type of packet")
   print(data)
 except:
  print "I didn't expect this error:", sys.exc_info()[0]
  traceback.print_exc()
  
if __name__ == "__main__":
 #------------ XBee Stuff -------------------------
 # this is the /dev/serial/by-id device for the USB card that holds the XBee
 ZIGBEEPORT = "/dev/serial/by-id/usb-FTDI_FT232R_USB_UART_A600eDiR-if00-port0"
 ZIGBEEBAUD_RATE = 9600
 # Open serial port for use by the XBee
 ser = serial.Serial(ZIGBEEPORT, ZIGBEEBAUD_RATE)


 # The XBee addresses I'm dealing with
 BROADCAST = '\x00\x00\x00\x00\x00\x00\xff\xff'
 UNKNOWN = '\xff\xfe' # This is the 'I don't know' 16 bit address

 #-------------------------------------------------
 logging.basicConfig()

  
 # Create XBee library API object, which spawns a new thread
 zb = ZigBee(ser, callback=messageReceived)
 print "started at ", time.strftime("%A, %B, %d at %H:%M:%S")
 notYet = True;
 firstTime = True;
 while True:
  try:
   if (firstTime):
    print("Wait while I locate the device")
    time.sleep(1)
    # First send a route record request so when the switch responds
    # I can get the addresses out of it
    print "Broadcasting route record request "
    zb.send('tx_explicit',
     dest_addr_long = BROADCAST,
     dest_addr = UNKNOWN,
     src_endpoint = '\x00',
     dest_endpoint = '\x00',
     cluster = '\x00\x32',
     profile = '\x00\x00',
     data = '\x12'+'\x01'
    )
    # if the device is already properly joined, ten seconds should be
    # enough time for it to have responded. So, configure it to
    # report that light has changed state.
    # If it hasn't joined, this will be ignored.
    time.sleep(5)
    print "sending 'configure reporting'"
    zb.send('tx_explicit',
     dest_addr_long = switchLongAddr,
     dest_addr = switchShortAddr,
     src_endpoint = '\x00',
     dest_endpoint = '\x01',
     cluster = '\x00\x06', # cluster I want to deal with
     profile = '\x01\x04', # home automation profile
     data = '\x00' + '\xaa' + '\x06' + '\x00' + '\x00' + '\x00' + '\x10' + '\x00' + '\x00' + '\x00' + '\x40' + '\x00' + '\x00'
    )
    firstTime = False
   print "Enter a number from 0 through 8 to send a command"
   str1 = raw_input("")
   # Turn Switch Off
   if(str1[0] == '0'):
    print 'Turn switch off'
    zb.send('tx_explicit',
     dest_addr_long = switchLongAddr,
     dest_addr = switchShortAddr,
     src_endpoint = '\x00',
     dest_endpoint = '\x01',
     cluster = '\x00\x06', # cluster I want to deal with
     profile = '\x01\x04', # home automation profile
     data = '\x01' + '\x01' + '\x00'
    )
   # Turn Switch On
   if(str1[0] == '1'):
    print 'Turn switch on'
    zb.send('tx_explicit',
     dest_addr_long = switchLongAddr,
     dest_addr = switchShortAddr,
     src_endpoint = '\x00',
     dest_endpoint = '\x01',
     cluster = '\x00\x06', # cluster I want to deal with
     profile = '\x01\x04', # home automation profile
     data = '\x01' + '\x01' + '\x01'
    )
   # Toggle Switch
   elif (str1[0] == '2'):
    zb.send('tx_explicit',
     dest_addr_long = switchLongAddr,
     dest_addr = switchShortAddr,
     src_endpoint = '\x00',
     dest_endpoint = '\x01',
     cluster = '\x00\x06', # cluster I want to deal with
     profile = '\x01\x04', # home automation profile
     data = '\x01' + '\x01' + '\x02'
    )
   # This will dim it to 20/256 over 5 seconds
   elif (str1[0] == '3'):
    print 'Dim it'
    zb.send('tx_explicit',
     dest_addr_long = switchLongAddr,
     dest_addr = switchShortAddr,
     src_endpoint = '\x00',
     dest_endpoint = '\x01',
     cluster = '\x00\x08', # cluster I want to deal with
     profile = '\x01\x04', # home automation profile
     data = '\x01'+'\xaa'+'\x00'+'\x25'+'\x32'+'\x00'
    )
   # This will brighten it up to 100% over 5 seconds
   elif (str1[0] == '4'):
    print 'Bright'
    zb.send('tx_explicit',
     dest_addr_long = switchLongAddr,
     dest_addr = switchShortAddr,
     src_endpoint = '\x00',
     dest_endpoint = '\x01',
     cluster = '\x00\x08', # cluster I want to deal with
     profile = '\x01\x04', # home automation profile
     data = '\x01'+'\xaa'+'\x00'+'\xff'+'\x32'+'\x00'
    )
   elif (str1[0] == '5'):
    print 'Report Switch Status'
    zb.send('tx_explicit',
     dest_addr_long = switchLongAddr,
     dest_addr = switchShortAddr,
     src_endpoint = '\x00',
     dest_endpoint = '\x01',
     cluster = '\x00\x06', # cluster I want to deal with
     profile = '\x01\x04', # home automation profile
     data = '\x00'+'\xaa'+'\x00'+'\x00'+'\x00'
    )
   elif (str1[0] == '6'):
    print 'Get Report from Switch'
    zb.send('tx_explicit',
     dest_addr_long = switchLongAddr,
     dest_addr = switchShortAddr,
     src_endpoint = '\x00',
     dest_endpoint = '\x00',
     cluster = '\x00\x05', # cluster I want to deal with
     profile = '\x00\x00', # home automation profile
     data = switchShortAddr[1]+switchShortAddr[0]
    )
  except IndexError:
   print "empty line, try again"
  except KeyboardInterrupt:
   print "Keyboard interrupt"
   break
  except NameError as e:
   print "NameError:",
   print e.message.split("'")[1]
   traceback.print_exc(file=sys.stdout)
  except:
   print "Unexpected error:", sys.exc_info()[0]
   traceback.print_exc(file=sys.stdout)
   break
   
  sys.stdout.flush() # if you're running non interactive, do this

 print ("After the while loop")
 # halt() must be called before closing the serial
 # port in order to ensure proper thread shutdown
 zb.halt()
 ser.close()

No, it isn't pretty, but it has comments.  It should be easy for folk to read and try out, and is the first example of a direct interface to a ZigBee compliant device I've ever seen.  This should take some of the mystery out of the protocol and the controllers that use it.  This code could be expanded to work one of the thermostats, receive from a panic button, or even one of those simple alarm switches.  The XBee does all the really hard stuff and saves us from worrying about it.

Have fun