Monday, September 2, 2013

Raspberry Pi and XBee Asynchronous Operation

In my last post <link> I whined a bit about using the Python XBee library on the Pi; it's complicated, but I made it work.  This is not actually too hard to understand, but it is more complicated that just checking to see if something is out there and then using it.

What I did was use the XBee library's asynchronous call, which creates another thread, to receive the message, then put the message on a queue.  In the original thread, I check the queue and pull off the message, dismantle it and use it.  Actually, this isn't a terrible way to do it, just very different from what I've done before.

Then, I realized that waiting on the queue to have something in it was silly.  I simply checked the queue to see if something was there and if not continued.  If there is something there, I call a routine to handle it.  To test it, I added the Python scheduler to the code and set up a timer so that, every 30 seconds, I send a status request message out to my network, and see if the answer comes back.  This is a feature of my network, not a general thing.  What I did was enable the current Arduino house controller to respond to a very simple message by sending the status of a few devices as a broadcast.  This allows me to have devices that send the query and look at the response to see what's going on.  It also helps by being a source of messages that I could look for.

The code got pretty complex, but I tried to comment the heck out of it so you can see what is going on:

The Raspberry Pi Script
#! /usr/bin/python
# This is an example of asyncronous receive
# What it actually does is fork off a new process
# to do the XBee receive.  This way, the main
# code can go do somthing else and hand waiting
# for the XBee messages to come in to another
# process.

from xbee import ZigBee
from apscheduler.scheduler import Scheduler
import time
import serial
import Queue

# on the Raspberry Pi the serial port is ttyAMA0
PORT = '/dev/ttyAMA0'
BAUD_RATE = 9600

# The XBee addresses I'm dealing with
BROADCAST = '\x00\x00\x00\x00\x00\x00\xff\xff'
UNKNOWN = '\xff\xfe' # This is the 'I don't know' 16 bit address

packets = Queue.Queue()

# Open serial port
ser = serial.Serial(PORT, BAUD_RATE)

# this is a call back function.  When a message
# comes in this function will get the data
def message_received(data):
        packets.put(data, block=False)
        print 'gotta packet'

def sendPacket(where, what):
        # I'm only going to send the absolute minimum.
                dest_addr_long = where,
                # I always use the 'unknown' value for this
                # it's too much trouble to keep track of two
                # addresses for the device
                dest_addr = UNKNOWN,
                data = what)

# In my house network sending a '?\r' (question mark, carriage
# return) causes the controller to send a packet with some status
# information in it as a broadcast.  As a test, I'll send it and
# the receive above should catch the response.
def sendQueryPacket():
        # I'm broadcasting this message only
        # because it makes it easier for a monitoring
        # XBee to see the packet.  This is a test
        # module, remember?
        print 'sending query packet'
        sendPacket(BROADCAST, '?\r')

# OK, another thread has caught the packet from the XBee network,
# put it on a queue, this process has taken it off the queue and
# passed it to this routine, now we can take it apart and see
# what is going on ... whew!
def handlePacket(data):
        print 'In handlePacket: ',
        print data['id'],
        if data['id'] == 'tx_status':
                print data['deliver_status'].encode('hex')
        elif data['id'] == 'rx':
                print data['rf_data']
                print 'Unimplemented frame type'

# Create XBee library API object, which spawns a new thread
zb = ZigBee(ser, callback=message_received)

sendsched = Scheduler()

# every 30 seconds send a house query packet to the XBee network
sendsched.add_interval_job(sendQueryPacket, seconds=30)

# Do other stuff in the main thread
while True:
                if packets.qsize() > 0:
                        # got a packet from recv thread
                        # See, the receive thread gets them
                        # puts them on a queue and here is
                        # where I pick them off to use
                        newPacket = packets.get_nowait()
                        # now go dismantle the packet
                        # and use it.
        except KeyboardInterrupt:

# halt() must be called before closing the serial
# port in order to ensure proper thread shutdown

Yes, this also means I can both send and receive XBee messages.  So, I have a scheduled XBee message that can get the response, a separate thread to receive XBee messages and not load down the main thread, the ability to get the data out of the message and use it.  Now, I need to combine that with a web server that can display the results.

One thing that needs to be understood about the Python XBee library:  it only returns good packets.  If there is a collision and the checksum doesn't work, you don't get the message.  If the message is fragmented by noise, you don't get the message.  If anything goes wrong, you don't get the message.  That makes debugging a network problem almost impossible since you can't see anything when things go bad.  So, keep an arduino around to look at stuff or build a sniffer like I did <link> to follow the traffic.  XCTU can help, but remember, if the message is sent to a specific address, it's invisible to the output of a different XBee and XBees don't have a promiscuous mode like Ethernet chips.

I'm going to look at web servers now.

Edit: It took exactly 5 minutes to get a web server running.  I'm starting to like this board.


  1. Thanks a is really useful which i want to do similar with xbee, arduino and R.pi. make ask few questions later. You made my day!

  2. Dave,
    I'm working on adding more to my two way communication vi the Python zigbee library. I'm a python newb and getting stuff to work by trial and error at this point.

    One problem I'm having is I can't seem to get more than a single byte to pass through the TX send message. My call on the python side is:
    xbee.send('tx', data="abasdf;kljads;flkjasdf;lkj", dest_addr_long='\x00\x13\xa2\x00\x40\xad\xbd\x76', dest_addr='\xff\xfe', frame_id='\x00')

    Basically I'm just wanting to send a larger packet of data. However, on the Arduino side, I have:
    uint8_t* tempData = rx.getData();
    nss.println(tempData[0]); //Prints 'a'
    nss.println(tempData[1]); //Prints 'b'
    nss.println(sizeof(tempData)); //Prints '2'

    The TX message is documented as follows:
    [{'name':'id', 'len':1, 'default':b'\x10'},
    {'name':'frame_id', 'len':1, 'default':b'\x01'},
    {'name':'dest_addr_long', 'len':8, 'default':None},
    {'name':'dest_addr', 'len':2, 'default':None},
    {'name':'broadcast_radius','len':1, 'default':b'\x00'},
    {'name':'options', 'len':1, 'default':b'\x00'},
    {'name':'data', 'len':None, 'default':None}],

    What I'm thinking is if you don't over-ride the length, the API simply checks for some data and sets the length to 1 byte, but I could be way off.

    Do you see what I'm not doing correctly here?

    1. It's tough working from code fragments, but I understand your frustration. What I suggest is to grab the code I posted to take apart an XBee frame on the arduino from and see what you get with it. Not that I'm perfect, but it's always fun to use something that works as a starting point. You may have to make some changes to fit your uses and give you less or more debug, but it's a place to start.

      Give it a try and feel free to ask questions about it.

    2. So you think the problem is in the arduino code? From all the debug messages I put in, I figured it had to be an issue with the python server code. Do you have any examples of using the python library to send more than one byte of data? Are you doing anything different than this to set the size of the string you send?

      xbee.send('tx', data="This is my message", dest_addr_long='\x00\x13\xa2\x00\x40\xad\xbd\x76', dest_addr='\xff\xfe', frame_id='\x00')

    3. well Dave, I am shocked, but putting in some of your arduino sketch results in printing my entire string. I guess I just don't understand the Python well enough at this point to determine what the size I'm prining actually is. I really appreciate the tips; I'd be stuck on this one for a while!

    4. Jeff, help me here, did you get past the problem you were having? Regarding python, it's totally different from c++ that you are using on the arduino. Python's string handling capability is amazing, and has led me to love the way you can do things with strings in it. Basically though, with python you don't worry as much about the length of things, they just sort of happens.

      I have several examples of my python code, take a look at my XBee page, it's the tab up top. I have tons of XBee code on this blog.

    5. Yup, I got past it. Through all of the debugging I was doing, it led me to believe that the python xbee code was only sending the first 2 bytes of data. I put in print statements in the construct command method, etc.

      Turned out, you were correct, and my data handling on the arduino side was the culprit. My code was only seeing 2 bytes of data, when the full data string was there all along. I pulled some snippets from the sketch you linked and was able to print it all out. Now I can start putting in my own protocol finally!

      Have you done anything new with your Android app? I actually got GCM (Google Cloud Messaging) working (there is a Python package for it). It is what apps like Gmail and such use to send a "tickle" alert to your mobile device to notify it to query for data. You can also send small amounts of data with it as well. The way I am using it is, if my home security is armed and a door is opened, the server will note that and immediately send a notification to all of my devices. Pretty neat!

    6. Re: GCM: no, I'd never even heard of it. But, you can bet I'll be looking at it now ... thanks

      Have fun.

  3. When you receive message at the coordinator (from a router or end device), is there a way to know from which router/end device the message came from?

  4. Yes there is, you just look at the source address returned. You have to be in API mode to do this. Another way is to simply put the name of the sender in the message and then check it at the coordinator.

    I use the name technique myself because it is way easier to use when debugging problems because I can just read it.

    1. Thanks, I realized afterwards that you can use data['source_addr'] if it is a 'rx' or 'rx_long_addr' mode. So whoever is sending also needs to be in the API mode right?

    2. Nope, the sending end can be in any mode. All packets are sent with the source address, but you have to be in API mode to read it.

  5. I am using XBee S2C, both coordinator and router in API mode (AP=2) mode. I am running your code in my Mac on the coordinator XBee. On the router (at XCTU) I see that it is receiving '?' that is sent from the coordinator and the coordinator is receiving the 'tx_status' message. All good.

    Now on the router side, I create a TX frame and send it. I am sure the communication is happening (the RSSI red led on the xbee shield glows w/ TX/RX led). But I do not see anything on the coordinator side which is running your python code.

    What am I missing?

    1. You lost me a little bit. You're running my code on a MAC ?? How?

    2. Yes, just run the python code on a terminal, connect the XBee on the USB port and your Mac becomes the coordinator (our router or ED) Xbee

    3. OK, I'm still missing something here. Do you have two Arduinos? One working as a router and the other as a coordinator with the coordinator hooked into a terminal program on the Mac?

      I'm missing how you run Arduino code on a terminal, or a Mac. If you have the setup described above, then I'm starting to understand.

    4. I don't have any Arduino.

      Coordinator - XBee connected to a Mac.I run your code in a python terminal here.

      Router - XBee connected to another Mac. I open XCTU here and see if it is getting any message. Yes, it gets your '?' message. Now if I send a data (either in API more or in AT mode), that data reach to coordinator (TX/RX led blinks) but as the output of your python code, I don't see any message.

    5. Got it, you have two Macs one with a router and one with a coordinator and the coordinator is running python.

      Now that I understand, you are probably not constructing the packet correctly. Use ONLY API mode, and I use API 2 because it is compatible with other things I do, but it doesn't really matter. Use the tool in XCTU to construct a packet and send it to the coordinator and see what you get.

      You can get into the source of the XBee library in python and actually put in print statements when it receives data to see what's coming in. I did that at first when I was working through some problems I created.

    6. Yes, I am at AP=2 in both the coordinator and router. And yes I did construct the packet in XCTU and send it to coordinator. I see the transaction is happening (TX/RX blinks) but as I mentioned nothing shown in the output of the python code.

      Which files in XBee python library you think I should look into?

    7. If memory serves me, the read is in, but it's been a long time since I had trouble getting it to work. Take a look and see if I'm right.

      Also, have you tried just doing a read from your port while sending from the router? You should be able to set up a simple read and just watch for characters coming in from the XBee. I did it with 'cat' to be sure I had the right port and such. There could be something wrong just reading the port.

    8. yea, did simple serial read and it worked earlier.

  6. Can an XBee in API mode receive data through

    Following your example, I wrote this simple data send code in API mode that runs in the coordinator ( The router at AT mode receives the data using below code

    while True:
    incoming = ser.readline().strip()
    if incoming != 'A':
    print '%s' % incoming
    ser.write('%s\n' % ack)

    Now I need the coordinator also receive data, so I do this in the coordinator code - after sending, it also reads the serial and prints data

    while True:
    sendData(WHERE, dataString)
    incoming = ser.readline().strip() #added to receive serial data at coordinator
    print '%s' % incoming #print serial data
    except KeyboardInterrupt:

    But as soon as I add those two lines, the communication stops. Is this because coordinator is in API mode and can't receive in serial? And I have to write the receive code for API mode (which I couldn't make work yet)?

    Coordinator API=2, running the code in python @ Mac
    Router AT mode, running below code @ Mac

    1. First, look at my explanation of what APi mode really is:

      Now, if your XBee is in API mode you must send the data in a carefully constructed packet or it will just get rejected by the XBee. If the XBee is in AT mode, it will try to send whatever you give it by enclosing it into a surrounding packet that the other end may not understand.

      So, what it looks like is that you've got some kind of problem that defies an easy explanation. I suggest you go back to AT mode at both ends on the XBee and send a simple text line of something like "Hello this is the router." and work up from there. Get the path working first so you don't have to worry about connections and such; just treat the XBee as a piece of wire and read and write to and from it.

      Once you have that working and confirm you have some sort of path between them, move to API mode on the coordinator to see if it can receive the message as part of the data.

    2. Resolved. Your xbee API mode link solved the problem. I was setting AP=2 while it should be AP=1 in xbee python library.

      Yes tried the both side AT mode before reaching you blog. That works fine.

      Now back to API mode. I want to retrieve the source address of the packet. I am sending data in this format

      zb.send('tx', dest_addr_long = address, dest_addr = UNKNOWN, data = datatosend)

      So on the receiving side, it is captured as a 'rx' type packet. It shows the data but doesn't print the source address. probably because source_addr of 'rx' type is not the 64-bit long address? How can I modify the above zb.send line so that the receiver receives 'rx_long_addr' type packet (think it will show the 64 bit source address)? Below code

      if data['id'] == 'tx_status':
      print data['deliver_status'].encode('hex')
      elif data['id'] == 'rx':
      print data['source_addr']
      print data['rf_data']
      elif data['id'] == 'rx_long_addr':
      print data['source_addr']
      print data['rf_data']

      Below output

      In handlePacket: rx


    3. Good, that problem is gone. All packets should have the long source address in them. Try just printing the entire set of values returned by the XBee library to see what you get.

      print data

      and see what happens. For me, I get a list of all the items being returned and the names of the items so I know how to spell them when I want to get them out of the value set returned. The data will be printed as ascii characters, so you may get what looks like trash, but it will give you an indication. Once you have that you can use something on the order of:

      for key, value in data.iteritems():
      if key in ["rf_data","id"]:
      packet += key + " "
      packet += value + ", "
      packet += key + " "
      packet += "".join("%02x " % ord(b) for b in data[key]) + ", "

      To make it pretty. I use the code above to munge the data into a mqtt message for something else to process or me to use in debugging.

    4. Thanks that helped. did print data and saw that the address is actually source_addr_long. Also had to do print repr(data['source_addr_long']) to print the address correctly in the terminal as it has escape sequence in it.