Raspberry Pi and SQLite3 and a Web Server

Remember when I thought it would be easy to get something out of an SQLite3 data base and put it on a screen with a web server?  I don't care how old I get, I'm still way too naive.  This turned out to be a royal pain in the bottom.  It started out when I went looking for a language to do it in.  Since I haven't ever used php, I decided to try it out.  It took me almost three hours of research to figure out what I needed to do to get it to work with the web server, and then another couple of hours to get php to work with SQLite3.  Yes, there were examples on the web, most of them were incoherent, and most of the rest of them were wrong.

What happens is that all the authors assume you know something about what your doing.  Well, that wasn't the case.  They also assume you know about permissions, file locations and such things for ALL the variations of unix out there.  That wasn't the case either.  They try desperately to give enough detail, but too often, leave something crucial out that kills the effort.  To add another level of complication, the unix variants change pretty rapidly, so if you hit a blog post over a few months old, be wary of it; it may not work anymore.  If you don't give up though, you'll find some lonely little post in an obscure forum that has it all reduced down to a simple one line command that actually works.

So, after I got enough stuff installed to actually write code, I had to learn enough php to do something.  I put together a tiny bit of code to open the database and immediately had my first failure.  Did you know that a php module absolutely has to have a file name that ends with .php or it won't work?  I do ... now.  After chasing down that problem, I started adding lines and fixing misunderstandings and stepped through about 8 hours of failure until I actually got up a web page with data I had taken from my thermostats in python, put in a database using sql, retrieved in php, and put up on a web page in html.

It may not look like much, but I'm sure proud of it:

The code to get this out of the data base I described in the previous post looks like this:

The php Module

<?php
# This is about the minimal debugging
# I could find that was easy to use
ini_set('display_errors', 'On');
error_reporting(E_ALL|E_STRICT);

# This is the database open call, it returns a
# database object that has to be use going forward
$db = new SQLite3('/home/pi/database/desert-home');

# you do a query statment that return a strange
# SQLite3Result object that you have to use
# in the fetch statement below
$result = ($db->query('SELECT * FROM thermostats;'));
#print_r($result); # I wanted to see what it actually was

# The fetch call will return a boolean False if it hits
# the end, so why not use it in a while loop?
#
# The fetchArray() call can return an 'associated' array
# that actually means give you back an array of ordered
# pairs with name, value.  This is cool because it means
# I can access the various values by name. Each call to
# fetchArray return one row from the thermostats table
while ($res = $result->fetchArray(SQLITE3_ASSOC)){
        #var_dump($res);
        print ("<strong>" . $res["location"] ." thermostat,</strong><br />");
        print ("Currently: " . $res["status"] . " <br \>");
        print ("Temperature: " . $res["temp-reading"] . " <br \>");
        print ("Settings are: <br \>");
        print ("Mode: " . $res["s-mode"] . " <br \>");
        print ("Temperature: " . $res["s-temp"] . " <br \>");
        print ("Fan: " . $res["s-fan"] . " <br \>");
        print ("<br>");
}
$db->close(); # I opened it, I should close it
?>

Php is an odd language, but it has some real strengths.  At some point I may post about the ones that I noticed and fell in love with, but there are several thousand articles out there that describe its strengths and weaknesses so I'd just be adding chaff if I went into it too much.

So, now I have one device being read, recorded, and monitored.  There are a lot more devices, and I still have to think about controlling them.

I'm starting to understand why more people aren't doing things like this around their own home.

Raspberry Pi and Sqlite3

If you got here through a search engine looking for a tutorial on Sqlite3 on the Raspberry Pi, that's not what this is.  I've been setting up a new Raspberry Pi to control my house and decided to experiment with a data base to store current data in so multiple processes can get at it.  After reading way too much information on various data base managers, I decided to test Sqlite3.  It seems like a reasonable choice for a really small system since it doesn't require a manager process and isn't split across multiple controllers.  Since the interface language is SQL, whatever I wind up doing can be ported to a larger data base tool when I need to.

So, after installing Sqlite3 on my Pi, I spent about two hours cobbling together enough code to try it out.  I chose to monitor my thermostats <link>, since they are ethernet enabled and respond directly to requests, then to store their current state in the data base.  The idea is that I can put up a web page that shows the status of the two thermostats and eventually add controls to change the settings.  Once again, there was a ton of information on the web about how to use Sqlite3, and things went pretty smoothly.

I like this little data base manager.  There's not a huge set up, actually, there wasn't much set up at all.  Just install it, and use it.  As usual, here's the code I came up with:

The python Script

import sys
import time
from apscheduler.scheduler import Scheduler
import urllib2
import sqlite3
import logging


logging.basicConfig()

def openSite(Url):
        try:
                webHandle = urllib2.urlopen(Url)
        except urllib2.HTTPError, e:
                errorDesc = BaseHTTPServer.BaseHTTPRequestHandler.responses[e.code][0]
                print "Cannot retrieve URL: " + str(e.code) + ": " + errorDesc
                sys.exit(1);
        except urllib2.URLError, e:
                print "cannot retrieve URL: " + e.reason[1]
        except:
                print "Cannot retrieve URL: Unknown error"
                sys.exit (1)
        return webHandle

def getThermoStatus(whichOne):
        website = openSite("HTTP://" + whichOne[0] + "/status")
        # now read the status that came back from it
        websiteHtml = website.read()
        # After getting the status from the little web server on
        # the arduino thermostat, strip off the trailing cr,lf
        # and separate the values into a list that can
        # be used to tell what is going on
        return  websiteHtml.rstrip().split(",")

def ThermostatStatus():
        # The scheduler will run this as a separate thread
        # so I have to open and close the database within
        # this routine

        print(time.strftime("%A, %B %d at %H:%M:%S"))
        # open the database and set up the cursor (I don't have a
        # clue why a cursor is needed)
        dbconn = sqlite3.connect('/home/pi/database/desert-home')
        c = dbconn.cursor()
        for whichOne in ['North', 'South']:
                c.execute("select address from thermostats "
                        "where location=?; ", (whichOne,))
                thermoIp = c.fetchone()
                status = getThermoStatus(thermoIp)
                print whichOne + " reports: " + str(status)
                c.execute("update thermostats set 'temp-reading' = ?, "
                        "status = ?, "
                        "'s-temp' = ?, "
                        "'s-mode' = ?, "
                        "'s-fan' = ?, "
                        "peak = ?,"
                        "utime = ?"
                        "where location = ?;",
                        (status[0],status[1],
                        status[2],status[3],
                        status[4],status[5],
                        time.strftime("%A, %B %d at %H:%M:%S"),
                        whichOne))
                dbconn.commit()

        print
        dbconn.close()

# I like things that are scheduled
# that way I don't have to worry about them
# being called, because they take care of themselves
sched = Scheduler()
sched.start()

# schedule reading the thermostats for every minute
sched.add_interval_job(ThermostatStatus, minutes=1)

# This is a priming read to show immediate results
ThermostatStatus()
# all the real work is done by the scheduler
# so the main code can just hang
while 1:
        try:
                time.sleep(1)
        except KeyboardInterrupt:
                break

# I want to look into using atexit for this
sched.shutdown(wait=False)

This works really well to interrogate each thermostat and put the results into a data base.  While I was working on it, I decided to get a little bit fancy and actually put the location and IP addresses of the two thermostats in the data base, retrieving and using them in the communication process.  Now that I know how, I'll have to think about doing the same thing when I adapt this to talk to my little XBee devices.

Once again, I set up the scheduler to cause things to happen.  This led to an interesting discovery, the scheduler starts a new thread to do the work.  I noticed this when Sqlite3 refused a connection because I opened the data base in one thread and then tried to use it in another one.  Rather than being an inconvenience, this is actually great.  I managed to prove that I can pass data from independent processes through the data base.  This will make the eventual loading of data by a web server much easier.

Since that was easy, I decided to put the actual time I last talked to a particular device in the data base as well.  Later, I can look at that time to see if the devices are having a problem.  Knowing a device is doing well is a constant concern when you have a bunch of them that are working independently around the house.

Now, I need to decide if I should work on a web page to display the status of the thermostats or start combining the various pieces of code I've constructed together to monitor the house.  But right this second it started to sprinkle outside.  Since rain is so rare here, me and the dog are going for a walk.

Raspberry Pi and Xively Part 3

I ran the python script I posted previously overnight <link>.  Here's graphs I picked up from Xively:

The little downward spikes in the temperature chart are an artifact of the way I measure outside temperature <link>.  I have techniques to correct this, but I didn't bother for this experiment.  The power chart is real and corresponded to my regular stuff that is running live.

It was interesting working out how to run the script in background with the terminal disassociated, but like everything else, it turned out to be easy once I figured it out.  Seems the python interpreter doesn't output text when you disassociate the terminal so you can't tell it's working.  That is unless you discover the secret parameter '-u' that makes it flush the output.  The command turned out to be:

nohup python -u scriptname.py > logfile &

Then you can 'tail -f logfile' to see what's happening.  None of this is unusual for *nix systems, it's just annoying to have to do a web search every 2.7 seconds to find out something that isn't obvious.  I'm sure it will get better over time ... maybe.

So, I have a way of grabbing and logging it to the cloud now and I'm testing ideas for temporary storage of the various data for web presentation.  Feels like some progress.

Raspberry Pi and Xively Part 2

Yesterday I was annoyed at Xively after spending hours trying to get their library to work with the limited documentation, well if I had written this earlier today, I would have been totally livid.  I overcame my problem with updating more than one datastream (Xively term) at a time, but it certainly wasnt because their documentation was clear.  Far from it, I found an example of updating a couple of items in a download of their library, BUT IT HAD COMPILE ERRORS.

We've all seen this.  Examples that need libraries they don't mention, code fragments that don't make sense, full blown examples that only illustrate a trivial case, and the ultimate insult, examples that don't compile.  I chased one of those for about an hour before I found the solution.

I did get a python script to work updating more than one datastream to a brand new feed I created on Xively.  Yes, you have to learn a new set of terms to use this stuff.  Armed with this tiny bit of success, I put in the code to gather XBee packets and update some global variables, then push them up to Xively and ran it for an hour or so to watch what happened.  It worked pretty well.  I don't have code in it to gather all the data I want, or a way to store it such that my new web server can present it, but I've got a nice start.

Let's talk about some of the things I've discovered.  The python scheduler works really well and does all the stuff I want.  I can set a routine to run any time I want, even to the point of six months from now at noon.  It's not quite as good at tiny periods, but I don't need that right now.  The Xively library is actually pretty extensive, but the documentation is terrible.  They don't even have a list of classes that are available, they rely on samples that suck instead.  The XBee library doesn't work the way a person coming from an Arduino experience expects it to, it forces you to use threads which makes passing data around harder than expected.  To make up for this, there is a cool library that can queue things up for you so another thread can grab them.  This little queue is really nice and could be used in a lot of different ways.

The HUGE advantage is the raspberry's handling of the internet.  It just works.  No long delays while the ethernet chip makes up its mind to work, you have tons of connections to play with, processing the returned data is a snap since python has an enormous string handling library.  There's so much that can be done there it's amazing.

Here's what I have so far:

The Python Script

#! /usr/bin/python
# This is an example of asyncronous receive
# What it actually does is fork off a new process
# to do the XBee receive.  This way, the main
# code can go do somthing else and hand waiting
# for the XBee messages to come in to another
# process.

from xbee import ZigBee
from apscheduler.scheduler import Scheduler
import logging
import datetime
import time
import serial
import Queue
import xively

#-------------------------------------------------
# on the Raspberry Pi the serial port is ttyAMA0
XBEEPORT = '/dev/ttyAMA0'
XBEEBAUD_RATE = 9600

# The XBee addresses I'm dealing with
BROADCAST = '\x00\x00\x00\x00\x00\x00\xff\xff'
UNKNOWN = '\xff\xfe' # This is the 'I don't know' 16 bit address

# The Xively feed id and API key that is needed
FEED_ID = 'putsomethinghere'
API_KEY = 'and here to'

# Global items that I want to keep track of
CurrentPower = 0
DayMaxPower = 0
DayMinPower = 50000
CurrentOutTemp = 0
DayOutMaxTemp = -50
DayOutMinTemp = 200

#-------------------------------------------------
logging.basicConfig()

#------------ XBee Stuff ------------------------
packets = Queue.Queue() # When I get a packet, I put it on here

# Open serial port for use by the XBee
ser = serial.Serial(XBEEPORT, XBEEBAUD_RATE)

# this is a call back function.  When a message
# comes in this function will get the data
def message_received(data):
        packets.put(data, block=False)
        #print 'gotta packet'

def sendPacket(where, what):
        # I'm only going to send the absolute minimum.
        zb.send('tx',
                dest_addr_long = where,
                # I always use the 'unknown' value for this
                # it's too much trouble to keep track of two
                # addresses for the device
                dest_addr = UNKNOWN,
                data = what)

# In my house network sending a '?\r' (question mark, carriage
# return) causes the controller to send a packet with some status
# information in it as a broadcast.  As a test, I'll send it and
# the receive above should catch the response.
def sendQueryPacket():
        # I'm broadcasting this message only
        # because it makes it easier for a monitoring
        # XBee to see the packet.  This is a test
        # module, remember?
        #print 'sending query packet'
        sendPacket(BROADCAST, '?\r')

# OK, another thread has caught the packet from the XBee network,
# put it on a queue, this process has taken it off the queue and
# passed it to this routine, now we can take it apart and see
# what is going on ... whew!
def handlePacket(data):
        global CurrentPower, DayMaxPower, DayMinPower
        global CurrentOutTemp, DayOutMaxTemp, DayOutMinTemp

        #print data # for debugging so you can see things
        # this packet is returned every time you do a transmit
        # (can be configure out), to tell you that the XBee
        # actually send the darn thing
        if data['id'] == 'tx_status':
                if ord(data['deliver_status']) != 0:
                        print 'Transmit error = ',
                        print data['deliver_status'].encode('hex')
        # The receive packet is the workhorse, all the good stuff
        # happens with this packet.
        elif data['id'] == 'rx':
                rxList = data['rf_data'].split(',')
                if rxList[0] == 'Status':
                        # remember, it's sent as a string by the XBees
                        tmp = int(rxList[1]) # index 1 is current power
                        if tmp > 0:  # Things can happen to cause this
                                # and I don't want to record a zero
                                CurrentPower = tmp
                                DayMaxPower = max(DayMaxPower,tmp)
                                DayMinPower = min(DayMinPower,tmp)
                                tmp = int(rxList[3]) # index 3 is outside temp
                                CurrentOutTemp = tmp
                                DayOutMaxTemp = max(DayOutMaxTemp, tmp)
                                DayOutMinTemp = min(DayOutMinTemp, tmp)
        else:
                print 'Unimplemented XBee frame type'

#-------------------------------------------------

# This little status routine gets run by scheduler
# every 15 seconds
def printHouseData():
        print('Power Data: Current %s, Min %s, Max %s'
                %(CurrentPower, DayMinPower, DayMaxPower))
        print('Outside Temp: Current %s, Min %s, Max %s'
                %(CurrentOutTemp, DayOutMinTemp, DayOutMaxTemp))
        print

# This is where the update to Xively happens
def updateXively():
        print("Updating Xively with value: %s and %s"%(CurrentPower, CurrentOutT
emp))
        print
        now = datetime.datetime.utcnow()
        feed.datastreams = [
                xively.Datastream(id='outside_temp', current_value=CurrentOutTem
p, at=now),
                xively.Datastream(id='power_usage', current_value=CurrentPower,
at=now)
                ]
        feed.update()

#------------------Stuff I schedule to happen -----
sendsched = Scheduler()
sendsched.start()

# every 30 seconds send a house query packet to the XBee network
sendsched.add_interval_job(sendQueryPacket, seconds=30)
# every 15 seconds print the most current power info
sendsched.add_interval_job(printHouseData, seconds=15)
# every minute update the data store on Xively
sendsched.add_interval_job(updateXively, seconds=60)

# Create XBee library API object, which spawns a new thread
zb = ZigBee(ser, callback=message_received)

# Initialize api client
api = xively.XivelyAPIClient(API_KEY)
# and get my feed
feed = api.feeds.get(FEED_ID)

#Do other stuff in the main thread
while True:
        try:
                time.sleep(0.1)
                if packets.qsize() > 0:
                        # got a packet from recv thread
                        # See, the receive thread gets them
                        # puts them on a queue and here is
                        # where I pick them off to use
                        newPacket = packets.get_nowait()
                        # now go dismantle the packet
                        # and use it.
                        handlePacket(newPacket)
        except KeyboardInterrupt:
                break

# halt() must be called before closing the serial
# port in order to ensure proper thread shutdown
zb.halt()
ser.close()

This gives the following output on the raspberry:

Console Output

pi@deserthome:~/src$ python powertoxively.py
Power Data: Current 0, Min 50000, Max 0
Outside Temp: Current 0, Min 200, Max -50

Power Data: Current 639, Min 639, Max 639
Outside Temp: Current 109, Min 109, Max 109

Power Data: Current 639, Min 639, Max 639
Outside Temp: Current 109, Min 109, Max 109

Power Data: Current 636, Min 636, Max 639
Outside Temp: Current 109, Min 109, Max 109

 Updating Xively with value: 636 and 109

Power Data: Current 636, Min 636, Max 639
Outside Temp: Current 109, Min 109, Max 109

Power Data: Current 638, Min 636, Max 639
Outside Temp: Current 109, Min 109, Max 109

Power Data: Current 638, Min 636, Max 639
Outside Temp: Current 109, Min 109, Max 109

Not a huge bunch of impressive stuff, but it illustrates the point.  I put as many comments in there as I could, both to help people understand and to nudge my own memory when I come back to this after doing something else for a while.

What I do is create a thread to catch XBee packets and queue them up to be handled.  In the main thread, I grab the packets off the queue and take them apart, saving a couple of important items in global variables.  I have an event scheduled to print the value of the global variables every 15 seconds and another event scheduled to run every minute and send updates to Xively.  Yes, this is an odd way of doing it, but it's what I already do on my current house controller.  I've found that scheduling things to happen is a much simpler way of handling tasks than anything else I've tried.

There's no internet handling in this module at all.  I will do that next.  As I mentioned before, I have two thermostats that are hooked to my local lan that can take commands and respond; I'll put the code in to query them every so often and save the results.  Since the internet handling in python is so robust, that shouldn't be a problem at all.

The big problem is deciding how to store the house data in such a way that the web server I have running on the Pi can get at it.  Everyone uses a database, but I'm not sure a big hunk of code like that is reasonable for a task like this.  Gotta think about it and experiment a bit before I go that route.

Perseverance, or maybe bull-headedness, has gotten me this far and I truly hope other people that are thinking about doing something like this stumble across this site.  It just might save them some of the headaches I've had.

Part Three of this is here <link>

Raspberry Pi and Xively Part 1

Frankly, I've been disappointed in Xively since it changed from the old Cosm.  Remember, before that it was Pachube.  Once upon a time it was a group of ambitious programmers that were working on a dream, now it's something else; I'm not sure what.  However, I haven't given up on them.  They still haven't completed things like their graphing API that everyone wants to use, but they have expanded their default graphs so I can select the period I want to see.  So, maybe there's still some interest among their developers.  However, I did ask them to come up with a way to port the legacy feeds to their new development system, and they put me in contact with a developer that was supposed to work on it.  This was over a month ago and I haven't heard from him again in weeks.  The single most annoying thing was their removal of the forum they had.  Now, if you have a question you're supposed to go to StackOverflow, and those folks can be rude and condescending.  Sure, I'm not the sharpest tool in the shed, but there are people on there that are just plain jerks.  I absolutely hate asking questions there.

Anyway, I started playing with Python and their site.  Since I might as well try their new interface and various capabilities at the same time, I followed their example and used their library.  The first thing I ran into is that their python library is considered preview so I had to use the '--pre' option to load the darn thing.  Next, they want me to use a virtual environment to develop, since I don't want to learn anything unnecessary at this point, I just ignored that part.  Using the example shown at http://gnublade.github.io/xively-python/tutorials/raspberrypi.html, I put together a module and tried it.  It worked.

However, do not under any circumstances name your source module the same as the library you're trying out.  I named my source module 'xively.py' and it took me a couple of hours to figure out what the heck the problem was.  When I changed it to 'testxively.py', I could take out the hundred or so debug statements that I had scattered all over the place and get on with trying it out.

Right now, I'm trying to combine gathering some data from my XBee network and incorporate it with the Xively example to create a module that can update Xively with real data.  In my pursuit, I had to learn about logging with Python; seems the scheduler module uses system logging if it encounters an error.  The Xively example only updates one data item, so that means I have to learn how to update a whole series of them (ain't no example for that).

So, maybe you understand why I labeled this entry as 'Part 1', I'm not sure how many parts there will be, but this is getting really confusing.

Part 2 of this is here <link>

Raspberry Pi and XBee Asynchronous Operation

In my last post <link> I whined a bit about using the Python XBee library on the Pi; it's complicated, but I made it work.  This is not actually too hard to understand, but it is more complicated that just checking to see if something is out there and then using it.

What I did was use the XBee library's asynchronous call, which creates another thread, to receive the message, then put the message on a queue.  In the original thread, I check the queue and pull off the message, dismantle it and use it.  Actually, this isn't a terrible way to do it, just very different from what I've done before.

Then, I realized that waiting on the queue to have something in it was silly.  I simply checked the queue to see if something was there and if not continued.  If there is something there, I call a routine to handle it.  To test it, I added the Python scheduler to the code and set up a timer so that, every 30 seconds, I send a status request message out to my network, and see if the answer comes back.  This is a feature of my network, not a general thing.  What I did was enable the current Arduino house controller to respond to a very simple message by sending the status of a few devices as a broadcast.  This allows me to have devices that send the query and look at the response to see what's going on.  It also helps by being a source of messages that I could look for.

The code got pretty complex, but I tried to comment the heck out of it so you can see what is going on:

The Raspberry Pi Script

#! /usr/bin/python
# This is an example of asyncronous receive
# What it actually does is fork off a new process
# to do the XBee receive.  This way, the main
# code can go do somthing else and hand waiting
# for the XBee messages to come in to another
# process.

from xbee import ZigBee
from apscheduler.scheduler import Scheduler
import time
import serial
import Queue

# on the Raspberry Pi the serial port is ttyAMA0
PORT = '/dev/ttyAMA0'
BAUD_RATE = 9600

# The XBee addresses I'm dealing with
BROADCAST = '\x00\x00\x00\x00\x00\x00\xff\xff'
UNKNOWN = '\xff\xfe' # This is the 'I don't know' 16 bit address


packets = Queue.Queue()

# Open serial port
ser = serial.Serial(PORT, BAUD_RATE)

# this is a call back function.  When a message
# comes in this function will get the data
def message_received(data):
        packets.put(data, block=False)
        print 'gotta packet'

def sendPacket(where, what):
        # I'm only going to send the absolute minimum.
        zb.send('tx',
                dest_addr_long = where,
                # I always use the 'unknown' value for this
                # it's too much trouble to keep track of two
                # addresses for the device
                dest_addr = UNKNOWN,
                data = what)

# In my house network sending a '?\r' (question mark, carriage
# return) causes the controller to send a packet with some status
# information in it as a broadcast.  As a test, I'll send it and
# the receive above should catch the response.
def sendQueryPacket():
        # I'm broadcasting this message only
        # because it makes it easier for a monitoring
        # XBee to see the packet.  This is a test
        # module, remember?
        print 'sending query packet'
        sendPacket(BROADCAST, '?\r')

# OK, another thread has caught the packet from the XBee network,
# put it on a queue, this process has taken it off the queue and
# passed it to this routine, now we can take it apart and see
# what is going on ... whew!
def handlePacket(data):
        print 'In handlePacket: ',
        print data['id'],
        if data['id'] == 'tx_status':
                print data['deliver_status'].encode('hex')
        elif data['id'] == 'rx':
                print data['rf_data']
        else:
                print 'Unimplemented frame type'


# Create XBee library API object, which spawns a new thread
zb = ZigBee(ser, callback=message_received)

sendsched = Scheduler()
sendsched.start()

# every 30 seconds send a house query packet to the XBee network
sendsched.add_interval_job(sendQueryPacket, seconds=30)

# Do other stuff in the main thread
while True:
        try:
                time.sleep(0.1)
                if packets.qsize() > 0:
                        # got a packet from recv thread
                        # See, the receive thread gets them
                        # puts them on a queue and here is
                        # where I pick them off to use
                        newPacket = packets.get_nowait()
                        # now go dismantle the packet
                        # and use it.
                        handlePacket(newPacket)
        except KeyboardInterrupt:
                break

# halt() must be called before closing the serial
# port in order to ensure proper thread shutdown
zb.halt()
ser.close()

Yes, this also means I can both send and receive XBee messages.  So, I have a scheduled XBee message that can get the response, a separate thread to receive XBee messages and not load down the main thread, the ability to get the data out of the message and use it.  Now, I need to combine that with a web server that can display the results.

One thing that needs to be understood about the Python XBee library:  it only returns good packets.  If there is a collision and the checksum doesn't work, you don't get the message.  If the message is fragmented by noise, you don't get the message.  If anything goes wrong, you don't get the message.  That makes debugging a network problem almost impossible since you can't see anything when things go bad.  So, keep an arduino around to look at stuff or build a sniffer like I did <link> to follow the traffic.  XCTU can help, but remember, if the message is sent to a specific address, it's invisible to the output of a different XBee and XBees don't have a promiscuous mode like Ethernet chips.

I'm going to look at web servers now.

Edit: It took exactly 5 minutes to get a web server running.  I'm starting to like this board.

Raspberry Pi and XBee

The continuation of this post is here <link>.

So, since I have the little Pi working with the internet, how about getting an XBee hooked to it and start receiving data from my home network?  Well, it isn't hard to hook an XBee up to a Pi; the little device uses 3.3 volts and has an output pin, so I hooked one up.  Four wires later, I had the XBee attached, powered and ready to go.  Of course, the serial port (the only one on the Pi) is already used for a console, but there's (again) a thousand web sites out there that show how to change the init files to allow the serial port to be used directly, so I used one of the examples and freed the serial port for use.

A quick aside here.  In looking around the web, I haven't found a single instance where someone successfully hooked a Pi up to an XBee and did something real with it.  This didn't bode well for me, I want this thing to monitor a network of a dozen XBees and keep the devices they're attached to under control, not print pretty messages on the screen.  It actually looks like a lot of people buy the Pi because it's cool and don't do much besides bring up a little web server on it.

But, the first thing is to see how I can read data from the XBee on the Raspberry Pi.  So, prowling around I found an XBee library for Python and it seems to support everything I need as well as some things I might want to use someday.  I installed it on the Pi and wrote a little test program to see what happens:

The Python Script

#!/usr/bin/python
import serial
from xbee import ZigBee

serial_port = serial.Serial('/dev/ttyAMA0', 9600)

zb = ZigBee(serial_port)

while True:
    try:
        data = zb.wait_read_frame() #Get data for later use
        #print data # for debugging only
        print data['rf_data']

    except KeyboardInterrupt:
        break

serial_port.close()

This actually wasn't as simple as it looks.  This tiny little piece of code took almost all day to get to work and gave me a lot of trouble.  Sure, it looks easy, but that's after failing over and over again.  Most of my problems came from not having a clear example to work from and a horrible dearth of documentation.  As usual, libraries provided by community efforts have minimal documentation, but this was an example of even less than that.  There was also problems with not having any description of the proper XBee settings.  For example, the author wrote that the XBee had to be in API mode, so following the Andrew Rapp library example, I set it to API mode 2.  I got nothing.  That led me to look at the code for the library and it turns out this library uses API mode 1 by default.  Switching to API mode 1 allowed me to actually see data.  And, there were a number of problems like that.

Another one that drove me nuts for an hour or so was that he (the author) sees the XBees as either Series 1 which uses different packets or Series 2 which is ZigBee based (mostly), but requires you to use special classes for each of them  So, if you import like this:

from xbee import XBee

I won't work at all using a series 2 XBee running the various ZigBee software.  It will work with the older software.  That's why the example above has:

from xbee import ZigBee

The author didn't understand that the series 2 devices can run multiple styles and versions of the protocols.

So, I can read XBee packets and pick the data out of them to do something with, but now along comes another problem.  All he supplies is a blocking read.  That's about useless for any project that needs to be able to catch packets and do something else at the same time.  He does have an asyncronous read in the code, but it uses threads to set aside a process that handles the XBee interaction, but since threads can't share data, it would require interprocess communication or queues to pass the data around.

Something like that takes all the usefullness out of an incredibly simple device like the XBee.  If you have to have multiple processes, queues, locks, and such to read a message, no one is going to bother.

I'm still looking at it, but this particular path doesn't show much promise so far.