A New Web Interface for Desert-Home

Remotely monitoring and controlling my house has been part of the technical work on the house ever since I started. Like every other homeowner in the world I wonder if I closed the garage door ten miles away from the house. Monitoring the power usage and closing the garage doors was my main motivation for getting into this.

I came up with a web page that let me control things and see what was going on a few years ago, and have never been totally satisfied with it <link>. It worked and told me what I wanted, but it looks too small on a phone and is kinda clunky.

Here it is on my phone:

Later, I discovered the MIT Android App Inventor and created an app using it. This tool was (almost) exactly what I needed to get a native Android app on my phone<link>. The problem with it came up when the app got big. The way they work it is with a single work space that you place graphic elements into creating an application. When it gets big, you can't see enough of it to move things around and connect the dots. While I used that app, I looked around for other solutions.

This actually led me to create a native Android application that could talk to the house. The Android app was pretty cool:

Normally, the 'Wait' would be replaced with the current temp, and the various gauges would be showing data, but things changed (as usual) and I didn't update it. Obviously, I was really into green and brown at the time, plus I still liked the brass gauges. This app was fun because each of the device panels would slide in from the side and slide back off when I wanted to get rid of them. That left the display of only the items I was interested in at that moment. People were properly impressed; I was proud.

The problem is that Android is a pain to program to. The development environment is HUGE and takes a significant hunk of the storage on my laptop. Testing it means putting it on my phone, testing a bit, fixing it, putting it back on the phone, over and over again. The methods work, but they get really monotonous. There's an emulator in the SDK, but I couldn't get it to work reliably or fast enough to actually do things. Plus, at that time, the emulator couldn't talk over the network which means large portions of the code couldn't work there.

I'm not knocking the Android SDK, they did a fantastic job of putting it together and working out a way to develop apps, but it's a huge mess for an individual trying to do a simple job like this. I was taking on the jobs of a team of developers instead of just coming up with something to control my house.

I went looking again. Since I had come up with three methods already, I knew more about what I wanted to have:

A web interface of some kind that would work nicely on the phone as well as my laptop. I was perfectly willing to accept some compromises because of the difference in screen dimensions and mouse vs. finger input methods, but I wanted it to be mostly seamless between the two.

One set of code for both of them. It is too cumbersome for an individual bringing up a control system to keep track of code for different viewing devices. I could never remember to make corresponding changes to each piece, and one of them, or the other, always fell behind.

I want to be able to put a tablet on the wall, walk up to it, change something around the house and not think about what device I'm using to make the change. So, a phone, a tablet or two, the laptop, my friends computer, etc. Kind of daunting when you think about it; that's why it's been years of messing around.

But, I recently bought a Samsung Galaxy Note 7. Yes, the one that is exploding and catching fire and isn't allowed to be turned on when you're flying. That one. I love it. Sure it's big, but that means I can actually see things on it, and the fast charging is absolutely great. I can charge it from totally dead to full in less than two hours. No wonder they tend to blow up. Anyway, the new phone prompted me to take another look at something to control the house with.

Prowling around the web, I found several interfaces that looked promising, but they all had those esoteric marketing descriptions that have a lot of words, but very little to tell you how to actually use it. Some really good authoring systems that could fit the bill, but there were flexibility concerns with them. While I was looking, I kept thinking about how things that some corporation supplies change whenever they change a manager, so I decided to use some JavaScript library that I could, if needed, copy to my own machine in total and avoid the dependency on them keeping their stuff online and active. I do this with Arduino libraries, why not be prepared to do it with other stuff as well?

Enter JQuery Mobile. It's a couple of libraries that sit on top of the JQuery we all know and love (hate), and has been around for a few years. It's not maintained very much anymore since the lead developer has moved on, but it does what I want and handles all the crap of resizing various things when it displays on a different size screen. It even supports 'themes'; that's when you want to change the look of the presentation with different colors and such. They put in animation for popups and sliders, so I don't have to learn about that, I can just use them.

Sure, it may not keep up with the latest trends since it hasn't had a substantive change in quite a while, but that only means I don't have to keep changing things because it did. No, it doesn't support older browsers, but I don't use them; I keep my browser up to date. I can also copy the libraries to my Pi and keep what I have for a very long time.

It took me a couple of weeks to put the code together, and it looks like a whole lot of stuff already out there on the web, but it works with the way I do things. I still have my cool gauges to show various readings, and the charts that I use to tell me when something is starting to have problems. I managed to leverage a whole lot of the code I used for the original web interface to save me time learning how to do that all over again. So, I have my gauges, charts and control buttons. I even messed around a bit with a custom display that looks pretty cool for the thermostats. Let me take you on a quick tour; first a picture of what the main screen looks like on a laptop:

Boring right? I only show two gauges on the initial screen because that's really all I'm interested in most of the time. The temperature and the power usage right this minute. The other items are buttons to separate pages in the application that I can go to for more information. Looking at the thermostats:

Yes, they have color and are too far apart on the screen. The resolution of the laptop does this. I could easily expand them to be larger, but this works for now and adjusts fine on the phone or tablet. This is what it looks like on the phone in portrait:

This is the way it would look on a tablet mounted to the wall where I walk all the time. Tells me what I want to know in a glance. Similarly, in landscape:

I personally think this looks nice as a two thermostat control panel. Tells me everything I want to know at a glance and a little interaction can make changes. More about the way to change things in a moment, first, look at the presentation of the first page in landscape:

The two gauges are clear and easy to read. Notice I finally got tired of the brass gauges and went to pastels and chrome. I'll get tired of this too, but for now it suits me. In portrait:

The two gauges are still clear, but now they're vertical; I got this display for free with the JQuery Mobile library. I can scroll down the screen to see the other options by just brushing the screen:

In case you're wondering, the footer is a marque that slides from the left across the screen; that's why it looks different on each picture. Now, back to the thermostats and how to change them. When I tap on the little gear on the bottom right of the thermostat it slides in a popup that holds the controls:

Here I can change whatever I need to and then tap the 'Set' button to send the changes off to the thermostat. Slick huh? Yes, the house runs warm during the day; I live in the desert remember? Also, I don't like high electricity bills.

Yes, I have the charts on there. For example, the house power and outside temperature are displayed by the 'House Chart' button:

This is landscape mode, the portrait mode looks too crowded since all the data is compressed to fit on the screen. This is touch zoomable, and I can zoom in on any part of it. This comes free with HighCharts, the charting tool I've used for the last couple of years.

I do the same thing for the fridge, freezer and freezer in the garage:

That's the house freezer with its stats for the last 24 hours, and when I tap on it:

The chart for its activity during the last 24 hours slides in and displays. I even enabled the points on the chart for examination. I can keep really good tabs on the appliances I've implemented so far. I really want to include other things over time in a similar fashion.

The pool looks like this:

This tells me the pool motor is running on high, the waterfall is on, the light is on and the fountain is on as well. Yes, I turned them all on for this screenshot. The motor is black, green, red for off, low, high. I tried to animate it, but is looked silly. Each item is a button that can control that accessory, for example:

Tapping on the 'Off' button will actually turn the fountain off. There are other things I want to add like the temperature of the water and actual speed of the motor, but I haven't gotten to them yet. 

No, I didn't forget the weather station:

Each of the parameters I measure has its own gauge and shows the last reading I took. In portrait mode they are a series of gauges arranged vertically. When I have enough data to chart it accumulated, I'll add graphs for them. I'm not sure what's the right way to do that yet. Do I need 24 hours or the last week? I'll get to that someday.

The Statistics button on the bottom shows this:

I'm going to add stuff like rainfall this year, last rainfall, highest historic wind speed; you know, the stuff that impresses people that don't realize how easy a database query is to code. (Don't look at the low temp above, that's not a bug, it's a feature)

The last thing I'll bother you with is the control page for the lights. I currently only control four of them here. Here's the display as it normally appears:

All the lights are off. I do this control differently, each light is a button that can be toggled, so tapping the patio light will turn it on. So if I tap a couple of them the page will look like this:

Kind of adds a little color to the display. These are controlled by wemo light switches that I described breaking into through several posts.

Like I've said annoyingly, this is one source for all devices that I may want to use to talk to it. I kind of hated giving up on the other attempts, especially the Android native code, but it's already out of date and several versions behind in the Android system it runs on. It's unlikely the web will change that fast and the latest thrust is to use web applications anyway.

Those of you clever enough to figure out the URL are welcome to take a look, but remember, be kind; it's running on a little Raspberry Pi. You can't actually control anything unless you figure out the super-secret high security password, and if you do, I'll just change it.

Now it's time to see about exchanging that phone ...

Android, Samsung Phone, My very own app, and a Raspberry Pi Home Automation Server

I did it; I finally took the plunge and created an app for Android that runs on my phone and controls the house.  I didn't want to go through the learning curve necessary to conquer the Android Development Environment, so I just relied on the web interface for all this time.  The problem is that I got a new phone and it was too great a temptation for me.  I had to do it.

I lucked out and ran across AppInventor 2 from MIT (it was once owned by Google) and it looked pretty cool, so I started trying it out.  The very first tutorial was enough to get me wound up.  Over several days, I experimented and coaxed an app out of it and then went totally nuts.  It's still a work in progress like all my projects, but it is SO COOL!

Here's the screen you see when you first tap on it

The first three buttons: Power, Temp, and Clock open the SteelSeries gauges I've already created and put them up so I can watch them.  The clock is bound to the time at my house so I can see exactly what time it is there wherever I am.  The data is supplied by my Raspberry Pi in JSON format so there is very little traffic to load it down.  All the formatting and such is done exclusively on the phone saving my little Pi for doing the monitoring tasks.  Those buttons were a pain.  Do you know how hard it is to find cool buttons out there?  That is without having to spend days editing them to get them to look OK.  Here's what it looks like with the power gauge showing:

Yes, this is the exact same display code I use for the web interface.  Appinventor 2 has provisions for a little browser that can handle javascript code.  I can't do the fancy graphs yet, because they are Flash based.  I'll have to look into that in the future.  Just for fun, here's the display when I put up the temperature gauge:

I stack the various controls and gauges one on top of the other so I can scroll between them if needed.  For example, Here's the screen when I have both thermostats showing:

I color the buttons yellow to show inactive and green for active which means they should be showing on the screen.  You can see how the two A/C unit selectors are green and below the controls and status of each of them.  One of them is idle and the other is recirculating the air; that's what the green fan is showing.  Yes, the fan turns; figuring out how to make it rotate was painful since the tool doesn't support animated GIFs yet.  The black fan doesn't turn silly, it means it isn't running.  There's also red and blue fans to show heating or cooling.  The tool has a 'picker' screen built into it, so to change the mode from Off to something else I pop up this display to choose from:

There's something similar for the fan also.  I got fancier on the temperature and put up a slider and set buttons:

When you slide it towards the red, the temperature setting goes up and similarly the other direction, then touch set and you have a new temperature setting.  When you touch set, the temperature controls go away and it leaves you with just the thermostat readings displayed.  This gives me darn good control over the thermostats and a nice flashy display.  I put pictures on the lights to show their state:

Those lights were a total pain in the butt.  I had to construct them from pictures that were waaay too big and didn't match the off representation.  Took a few hours using paint.net to get them right.  To turn off the lights, just touch the icon itself:

Notice how I managed to find a nice clear bulb and use it for both states.  No, I can't dim the bulbs, nor do I want to.  The garage doors are still pretty primitive:

But, they get the job done.  I want to fancy that display up some over time.  Similarly, the pool controls are still in the beginning stages:

Lots of refinements are needed here.  All of it works though, I can turn on the pool light anywhere I have data service on the phone.  Of course, no rural place would be complete without knowing the state of the septic tank:

When septic gets too full, the OK icon goes red.  I do the same thing when the acid pump goes low.  I show this to people and they invariably click on the septic tank:

I love their reaction when they have to select "I'm a Dummy" from the picker.  There's a couple of Easter Egg messages like that in other places as well.  Hey, gotta have fun with it too ... right?

Needless to say, I spent a heck of a lot of time just playing with the controls and watching things turn on and off around the house.  I actually had the front lights cycling in a kind of rhythm for a while, much to the amusement of the neighbors.  I guess the new will wear off in a few weeks, but then I'll just make something else and hook it into the system.  I'll also be looking for cute icons to include in various places in the app; it would be nice to have a cartoon garage door to show when it is open.  I'm thinking about using my old cell phone as a house control.  Take the battery that expanded out <link> and hook it to power.  Since everything will work except the actual phone, I should be able to turn it into a house controller that sets by the bed, or maybe mount it on the wall.  It would give me great pleasure to repurpose that device so it would actually get some use ... finally.

I can't wait to show this off.  So, people, what's holding you up from building something like this as well?  No, I won't port this to the iPhone.

Belkin's Wemo Light Switch Part 1

This is an ongoing project (aren't they all) so I'm calling this part one.

I recently (couple three weeks ago) bought a Wemo Light Switch (link).  I was fascinated by the fact that it runs embedded Linux and could be hacked a bit to do some things I want.  They cost about $50 (US), so they aren't the cheapest devices out there, but have you looked at light switches that can be operated remotely lately?  Fifty bucks isn't that bad, some of the other ones cost a whole lot more and many come in at around the same price.  So, it's not as bad as it sounds.

However, if you scan the web, you'll find a lot of articles that are obviously trying to get hits because all they say is what Belkin already says on their site.  There are also articles that recount how they work; something that you can get from any of the suppliers out there.  But, hidden among the chaff are a few really interesting tidbits.  First, is that the device came out quite a while ago, but only to work with the iphone.  Then they came out with a beta version of an Android app for people to try.  The reviews of the device are mixed.  Some love it and it does exactly what they want, other hate the device and Belkin because the device doesn't work and Belkin support doesn't respond worth a darn.

So, with people being totally disgusted with the device, a marginal (if even that) Android app, and terrible support, I bought one.  Nothing like jumping head first into the shallow end of the pool.

The device came in and I rushed right in to install it to replace one of those X10 switches that are so darn unreliable.  I carefully read the almost non-existent instructions, downloaded the Android app, and sure enough, it didn't work.  The set up is like some other devices, the switch has a Wifi chip in it and it is initialized to set up an ad-hoc network so you can initialize it to your in-house wireless network.  So, you take your phone, connect to the switch's ad-hoc usecured network and configure the chip to talk to your own wireless.  My problem was that the Wemo Android app wouldn't talk to the switch; my phone would, but the app couldn't see it.

I searched the web, Belkin's support site ... nothing.  I did find a number of folk that had hacked into it using some pretty clever techniques, but nothing that I could use.  Finally, I picked up the phone and called Belkin support.  I got the usual (for technical support everywhere) scripted responses and a promise to get a call back from a higher level technician.  That NEVER happened.  So, I thought, "Who has an Android phone near me?"  As you all know by now, I live in the sticks, so I had to do some searching, but I found a neighbor with a phone that would help.  I loaded the Wemo Android app on their phone and tried it again.  It worked!!

Yes, a different phone worked.  The neighbor's phone was running a slightly older version of Android and was of course, a different brand.  So, I set up the switch and tried my phone again since the switch was now talking to my Wifi network.  No luck, the app running on my phone couldn't see the switch.  Fine, there's more than one way to talk over ethernet.  Once again out to the web to see what I could find.

As luck would have it, I found the perfect solution for me, and probably a lot of other people out there.  This guy named Ian McCracken had already messed with the switches and developed a set of tools and API for the Wemo switch.  Bless Him.  He put together a nice python library that supports discovery and use of the switches and documented a ton of the intricacies involved.  Basically, this guy saved my butt on this little switch.  His web page where he describes it is here <link> and his library is here <link>.  Now just to be fair, another great guy: Isaac Kelly did some discovery work that made it pretty clear how to manipulate the Wemo switch.  Isaac's stuff is here <link> and he used a tool called Miranda created by /dev/ttys0 <link>.  But, Miranda had to be patched to support the Wemo switch because it uses a little known technique; of course, Isaac Kelly suggested the patch.  Miranda is here <link> and the patch is described here <link>.

So, with a patched Miranda, and a cool python library, I got the switch to work in my house.  Whew!  I still couldn't operate the switch with my phone, but I could turn it on and off with my Raspberry Pi.  Slick.

This all happened while I was waiting for the return call from Belkin.  That call never happened.  So, I decided to make a butthead out of myself and annoy the heck out of Belkin.  How can a normal person like me annoy a big company?  I went to their support site for the device <link> and nothing there could help me.  Next I went to getsatisfaction.com and found a Wemo forum there.  I posted a couple of times about the problem and how their response took forever and didn't accomplish anything <link>.  When that didn't work, I got a brainstorm.

I looked on Twitter and among all the crap was a Belkin marketing user.  Cool.  Right there in their marketing hype I posted about how I couldn't get any help for a product that they made that didn't work.  That got an immediate response.  I tweeted the forum entry I put up and they got on the stick and sent me mail telling me that they were referring the problem to someone to handle.  After a couple more of these, I actually got a phone call and a contact point at Belkin.

This nice lady tried with me over the phone to get the cell phone to talk to the Wemo, no luck.  She sent me some debugging instructions and I tried them to gather some information to help them diagnose the problem.  Suddenly, my phone started working.  Yep, you guessed it something changed and the Android app started working with the Wemo switch.  The sequence of events that got my phone to work with the switch is too long to go into in this post, but I will post it if anyone is interested.

So, I can turn on a light with a piece of code running on my Raspberry Pi, or over the internet using my cell phone.  I plan on extending this to allow my web interface to operate the switch as well, since that will only be a tiny bit of additional code.  It also opens up the possibility of adding more Wemo devices for things like appliances and other lights.  Belkin is also introducing a device that will control and monitor the power usage of an appliance; bet you know what I'll be ordering soon.

The lesson to be learned here is Legion. First, don't give up too easily.  Second, hit them where they will notice it: their online marketing. Third, expect a fix from the oddest circumstances.  Fourth, there are lots of other folk out there that have the same kind of problem, look for them.  Last, there are technical folk out there that can help with a lot of things; you just have to find them.

Raspberry Pi Home Automation Process Architecture

Don't you dare make fun of the title of this post.  Yes, it's a bit presumptuous, but I couldn't think of anything else.  After a lot of experimenting and trying to overcome various problems, I finally settled on an architecture of the various devices and controls that I think can work over time.

Let's talk about the XBee network I have as an example.  I have a network of XBees that monitor and control things around the house and this has to hook into the Pi.  So, in an earlier post, I discuss how easy to hook into the Pi it was.  Now, that I've used up the only serial port on the Pi, I ran into a problem.  Only one process can reliably hook to the Pi's serial port and communicate both ways.  If you have two processes talking to one port, bad things happen like overlaying data, seeing a character come in and not being able to get it because some other process got to it first; that kind of thing.  This isn't like an arduino where there is only one piece of code that needs this stuff, there's a number of them. So, what to do?  Functionally, the entire process interaction looks like this:

The House Monitor box is a process that runs all the time and constantly monitors the XBee network as well as periodically interrogating the ethernet devices.  When data comes in, it updates the database with the current readings.  It is the sole interface to the XBee network.  To get commands from the internet to do something like open a garage door, the web page sends a jquery() PUT to the command handler who forwards it (translating on the way) to the House Monitor.  The house monitor forwards it along to the XBee network.  In due course, the device will receive the command, do whatever, and send back a status that is recorded in the database.

If I load one of the presentation pages, it will send a jquery() GET to the Data Provider code that will suck the data out of the database and hand it back to the page for display on a user's browser.  The web pages have a timer running on them that refreshes the data every so often so it can be used as a monitor of what's going on.  For example, the swimming pool does several things on its own, like turn the motor on high in the morning, drop it to low for a while, then finally turn it off entirely during the peak period.  Each of these things is caught by the House Monitor process and recorded as the current state in the data base.  So a presentation page will update and show that the motor did its thing.  If I open the garage door, the garage controller will report it and House Monitor will stick that in the data base too.  Basically, the database is the holder of all the current status of devices, including the current power usage.

Yes, this means there is a small delay between commanding something to happen and seeing the result of it on the display, but that's not important to me.  There's always a delay between sending a signal to a real device and seeing the result.  

Doing things this way, I can make an interface inside the house that talks directly to the House Monitor and it can control stuff with no delay (other that what's required to do whatever it is).  The idea is that the House Monitor process is the thing in actual control, other software and devices talk to it.

I separated the presentation from the data gathering so I can change an HTML page anytime I want to without worrying about how the data handling will have to change to support it.  This way I can mess with tables, pictures, colors and stuff without accidentally screwing up the data handling.  It also hides the database entirely away from the web, so certain hacking techniques can't harm it.  It also means that I can have multiple presentation pages without having to duplicate the way I grab stuff out of the data base.

I also got ALL the devices I update to the various cloud services talking to the Pi.  I have updated my account on Xively to the latest stuff they came out with and am recording the actual readings there every minute or so.  I may lose the old data, since they still haven't given me any way to move it into the new system.  Or, I may just write a python script to suck the data out and put it back into the new feeds.  It would only take a few million transactions to get it here and put it there.  He he.  

Code?  You want me to show you the Code?  Actually, there's nothing secret about it and I'm totally willing to share, but I'm going to hold off a few more days until I can remove the old controller from service and reuse the page it is described on.  

Physically, the Pi is naked, no box, no special lights on it, and hooked to a breadboarded XBee.  Nope, I haven't even started looking for something to put it in, or decided if I want it to have a display.  It would be nice to have a display that shows something, but there are so many possibilities that I'm kind of stuck deciding what to do.   The odd way they constructed the little board doesn't lend itself to any simple ideas.  It'll work out in time; I may just use a cardboard box.

Here's the latest web page presentation:

Yes, the gauges reflect the very latest readings from the sensors, the boxes show the real state of the various devices, and the buttons work.  It isn't very pretty, but I wasn't working on pretty.  Actually, I made all the boxes float so they display in different places depending on the resolution of the browser.  I can see all of them on my phone by just scrolling around.  This part will be constantly modified as I think of something to try out.  It's just a presentation page and I don't have to change anything else to make it look completely different.   Oh, the little diag button up there will expand into a scrolling screen of debug information; makes it nice to see some of the underlying communication when I need to.

Less than an amp of power and roughly fifty buck so far.  Go ahead, try and beat it.

Raspberry Pi, Getting Control Of The Devices

Still making progress on this thing. There's a little pressure to get it finished since I want it to take over the house control functions. I managed to get the buttons working to control the thermostats and next is my first XBee device, and it is going to be painful. I haven't found or experimented with a php library for controlling the XBees so back to searching the web for clues. Meanwhile, I was having a lot of trouble getting the various syntax correct and was also having some trouble with the command to the devices. I needed something on the web page to tell me what was actually going on that could maybe become a monitor of the network internals. I sneaked the code that I use to put up the code boxes on this blog and shoved it into the controller web page. It worked like a charm; I was even able find a way to force it to scroll to the bottom as new items came in. This could turn out to be a great way of monitoring actual conversations between devices later when the entire device set is hooked in. Here's an example of it in operation:

Recognize the blue box?  It's got a bunch of debug on it from me hooking in the Acid Pump, but that particular piece doesn't work yet since it's XBee controlled.  It's actually pretty easy to hide things in web pages, so I may hide it and put a button up to show it when needed.  There's actually no limit to what kinds of things I can do now that I have a real web server running to present the data.

I still haven't settled on the entire architecture.  I have a background process that collects data from around the house and shoves it into a database.  Then the web page interacts with a php file to suck the data back out.  All the buttons connect to another php file that translates the clicks into commands that are tailored for each particular device since they are all different from each other.  Yes, I screwed up and made them different.  But as I went along, I learned things and did them differently as each new device came online.  Actually, the separation from data display and command makes it easier to add in a device.  I construct a basic page, implement the commands, and go back and play with presentation.  Since presentation is a never ending task, it's nice to have it isolated.

Raspberry Pi, Now it's actually getting somewhere

So, when I started trying to put things together, it got so darn complex that I changed a ton of stuff to make it somewhat simpler, at least for me.  Here's the current display:

The buttons don't do anything except show an alert to prove they do something, I still have a number of devices that I have to include, and there's a ton of stuff to pretty it up and put in some presets I've grown used to.  I currently have presets for things like A/C off, summer night settings, winter night settings, that kind of thing.

But, isn't it cool ?

All the data is live and real, including the SteelSeries gauges up at the top.  Sorry, it isn't on the internet yet, I have to wait until I get it finished enough to turn off the old controller before I can put it online.  I broke the web interface into two components, one that presents the web page and the other that sucks things out of the data base and hands them off to the web interface.  The code to create a complete screen was just getting out of hand, so I split it apart.

This gave me the opportunity to try out tables and colors and things, as well as experiment with getting data out of the database to construct a JSON response.  This actually turned out to be relatively easy, but it drove me nuts thinking of variable names.  I got to mess with jquery() which turns out to be quite a workhorse and makes things even simpler.

I'm starting to get tired of reading tutorials on various things.  But the single most annoying thing is the difference between comment delimiters in the various languages.  Heck, you have to use multiple different kinds in a single web page ... silly.

I'll do a post on the overall architecture of the system soon, I want to conquer the buttons and build in the rest of the devices so I can replace the current controller.  With the way things have been going I'll probably change the architecture two or three times before it's done.

With the cost of tablets dropping, my ultimate plan is to have a couple around the house that are used to control things.  Y'know, one of them by the bed so I don't have to get up to change something.  The ultimate remote control.

Heck, I may hook the barbecue up to it someday.

Raspberry Pi, XBee, SQLite3, and a Web Page

It's been raining on and off, so I thought this would be a good chance (excuse) to experiment with reading XBee packets (in python), and saving them in a database.  It would be cool to also send the data out to Xively to be saved, and maybe display them on a web page.  So, I sat down and did it.

Granted, it was a pain switching between python, SQL, and php, but after a while, you kind of get used to it.  I took the XBee example I previously posted that does asynchronous reads of XBee network packets and modified it to save some of the data items to my SQLite3 database.  Then I modified the php web page from a couple of days ago to get the items out of the database and display them.  Things went much faster and easier since I didn't have to install anything, just add things here and there.

That's the way it usually goes; getting started is over 50% of the work.  Then the next 40% moves along pretty well with the last 10% taking forever.

Here's the updated XBee python code to catch stuff and save it in the database.  If you look closely, I started grabbing the packets from the Acid Pump and saving them.  That's my first attempt at grabbing a particular device and saving it.  The status packet I've been grabbing is forwarded by my controller <link> that I've been using for a couple of years and hope to replace.  I'll have to work on the packets being sent by my XBee thermostat <link> that is setting in a housing on a fence post outside next.

The python Script

#! /usr/bin/python
# This is the actual house Monitor Module
#
# I take the techniques tried in other modules and incorporate them
# to gather data around the house and save it in a data base.  The
# data base can be read for presentation in a web page and also
# forwarded to Xively for cloud storage and graphing.
#
# This particular version only reads the XBee network, it
# doesn't go to my internal lan to gather data, that's next
# on my list.
#
# For the XBee network, I fork off a new process
# to do the XBee receive.  This way, the main
# code can go do somthing else and hand waiting
# for the XBee messages to come in to another
# process.

from xbee import ZigBee
from apscheduler.scheduler import Scheduler
import logging
import datetime
import time
import serial
import Queue
import xively
import sqlite3


#-------------------------------------------------
# on the Raspberry Pi the serial port is ttyAMA0
XBEEPORT = '/dev/ttyAMA0'
XBEEBAUD_RATE = 9600

# The XBee addresses I'm dealing with
BROADCAST = '\x00\x00\x00\x00\x00\x00\xff\xff'
UNKNOWN = '\xff\xfe' # This is the 'I don't know' 16 bit address

# The Xively feed id and API key that is needed
FEED_ID = '1428598370'
API_KEY = 'put_your_xively_key_here'

# Global items that I want to keep track of
CurrentPower = 0
DayMaxPower = 0
DayMinPower = 50000
CurrentOutTemp = 0
DayOutMaxTemp = -50
DayOutMinTemp = 200

#-------------------------------------------------
logging.basicConfig()

#------------ XBee Stuff ------------------------
packets = Queue.Queue() # When I get a packet, I put it on here

# Open serial port for use by the XBee
ser = serial.Serial(XBEEPORT, XBEEBAUD_RATE)

# this is a call back function.  When a message
# comes in this function will get the data
def message_received(data):
        packets.put(data, block=False)
        #print 'gotta packet'

def sendPacket(where, what):
        # I'm only going to send the absolute minimum.
        zb.send('tx',
                dest_addr_long = where,
                # I always use the 'unknown' value for this
                # it's too much trouble to keep track of two
                # addresses for the device
                dest_addr = UNKNOWN,
                data = what)

# In my house network sending a '?\r' (question mark, carriage
# return) causes the controller to send a packet with some status
# information in it as a broadcast.  As a test, I'll send it and
# the receive above should catch the response.
def sendQueryPacket():
        # I'm broadcasting this message only
        # because it makes it easier for a monitoring
        # XBee to see the packet.  This allows me to monitor
        # some of the traffic with a regular XBee and not
        # load up the network unnecessarily.
        #print 'sending query packet'
        sendPacket(BROADCAST, '?\r')

# OK, another thread has caught the packet from the XBee network,
# put it on a queue, this process has taken it off the queue and
# passed it to this routine, now we can take it apart and see
# what is going on ... whew!
def handlePacket(data):
        global CurrentPower, DayMaxPower, DayMinPower
        global CurrentOutTemp, DayOutMaxTemp, DayOutMinTemp

        #print data # for debugging so you can see things
        # this packet is returned every time you do a transmit
        # (can be configured out), to tell you that the XBee
        # actually send the darn thing
        if data['id'] == 'tx_status':
                if ord(data['deliver_status']) != 0:
                        print 'Transmit error = ',
                        print data['deliver_status'].encode('hex')
        # The receive packet is the workhorse, all the good stuff
        # happens with this packet.
        elif data['id'] == 'rx':
                rxList = data['rf_data'].split(',')
                if rxList[0] == 'Status':
                        # remember, it's sent as a string by the XBees
                        tmp = int(rxList[1]) # index 1 is current power
                        if tmp > 0:  # Things can happen to cause this
                                # and I don't want to record a zero
                                CurrentPower = tmp
                                DayMaxPower = max(DayMaxPower,tmp)
                                DayMinPower = min(DayMinPower,tmp)
                                tmp = int(rxList[3]) # index 3 is outside temp
                                CurrentOutTemp = tmp
                                DayOutMaxTemp = max(DayOutMaxTemp, tmp)
                                DayOutMinTemp = min(DayOutMinTemp, tmp)
                                dbconn = sqlite3.connect('/home/pi/database/desert-home')
                                c = dbconn.cursor()
                                # do stuff
                                c.execute("update housestatus "
                                        "set curentpower = ?, "
                                        "daymaxpower = ?,"
                                        "dayminpower = ?,"
                                        "currentouttemp = ?,"
                                        "dayoutmaxtemp = ?,"
                                        "dayoutmintemp = ?,"
                                        "utime = ?;",
                                        (CurrentPower, DayMaxPower, DayMinPower,
                                        CurrentOutTemp, DayOutMaxTemp,
                                        DayOutMinTemp,
                                        time.strftime("%A, %B, %d at %H:%M:%S")))
                                dbconn.commit()
                                dbconn.close()


                elif rxList[0] == 'AcidPump':
                        # This is the Acid Pump Status packet
                        # it has 'AcidPump,time_t,status,level,#times_sent_message
                        # I only want to save status, level, and the last
                        # time it reported in to the database for now
                        dbconn = sqlite3.connect('/home/pi/database/desert-home')
                        c = dbconn.cursor()
                        c.execute("update acidpump set status = ?, "
                                "'level' = ?,"
                                "utime = ?;",
                                (rxList[2], rxList[3],
                                time.strftime("%A, %B, %d at %H:%M:%S")))
                        dbconn.commit()
                        dbconn.close()
                else:
                        #print ("can\'t handle " + rxList[0] + ' yet')
                        pass
        else:
                print ('Unimplemented XBee frame type' + data['id'])

#-------------------------------------------------

# This little status routine gets run by scheduler
# every 15 seconds
def printHouseData():
        print('Power Data: Current %s, Min %s, Max %s'
                %(CurrentPower, DayMinPower, DayMaxPower))
        print('Outside Temp: Current %s, Min %s, Max %s'
                %(CurrentOutTemp, DayOutMinTemp, DayOutMaxTemp))
        print

# This is where the update to Xively happens
def updateXively():
        print("Updating Xively with value: %s and %s"%(CurrentPower, CurrentOutTemp))
        print
        # Currently I have to use UTC for the time,
        # there's a bug somewhere in the library or
        # Xively.  It doesn't matter though because
        # it's easy to convert
        now = datetime.datetime.utcnow()
        # open the database
        dbconn = sqlite3.connect('/home/pi/database/desert-home')
        c = dbconn.cursor()
        # Yes, there are better ways to do the stuff below,
        # but I wanted to use a single statement to get it
        # from the data base an update the field going to
        # Xively.  It turns out that is is a rather odd
        # looking statement, but it works.
        feed.datastreams = [
                xively.Datastream(id='outside_temp',
                        current_value = c.execute(
                                "select currentouttemp from housestatus")
                                .fetchone(),
                        at=now),
                xively.Datastream(id='power_usage',
                        current_value = c.execute(
                                "select curentpower from housestatus")
                                .fetchone(),
                        at=now)
                ]
        dbconn.close() # close the data base
        feed.update()  # and update Xively with the latest

#------------------Stuff I schedule to happen -----
sendsched = Scheduler()
sendsched.start()

# every 30 seconds send a house query packet to the XBee network
sendsched.add_interval_job(sendQueryPacket, seconds=30)
# every 15 seconds print the most current power info
sendsched.add_interval_job(printHouseData, seconds=15)
# every minute update the data store on Xively
sendsched.add_interval_job(updateXively, seconds=60)

# Create XBee library API object, which spawns a new thread
zb = ZigBee(ser, callback=message_received)

# Initialize api client
api = xively.XivelyAPIClient(API_KEY)
# and get my feed
feed = api.feeds.get(FEED_ID)

#This is the main thread.  Since most of the real work is done by
# scheduled tasks, this code checks to see if packets have been
# captured and calls the packet decoder
while True:
        try:
                time.sleep(0.1)
                if packets.qsize() > 0:
                        # got a packet from recv thread
                        # See, the receive thread gets them
                        # puts them on a queue and here is
                        # where I pick them off to use
                        newPacket = packets.get_nowait()
                        # now go dismantle the packet
                        # and use it.
                        handlePacket(newPacket)
        except KeyboardInterrupt:
                break

# halt() must be called before closing the serial
# port in order to ensure proper thread shutdown
zb.halt()
ser.close()

This has my most recent discoveries regarding python and its interface to the SQLite3 database.  It took me a while to figure out how to grab a single item out of the data base, but it's illustrated above.  Of course I updated the web page I'm working on to show off the current power usage and outside temperature that is forwarded by my controller.  Here's the modified web page:

The PHP Web Page

# This is about the minimal debugging

# I could find that was easy to use
ini_set('display_errors', 'On');
error_reporting(E_ALL|E_STRICT);

# This is the database open call, it returns a
# database object that has to be use going forward
$db = new SQLite3('/home/pi/database/desert-home');
#
# I'm going to get the power usage and outside temperature
# from the database and display it.  I use the querySingle()
# call because I haven't messed with it yet.  I could have gotten
# the entire record and just displayed the parts I needed instead
$power = $db->querySingle("select curentpower from housestatus;");
$outtemp = $db->querySingle("select currentouttemp from housestatus;");
print ("<Strong>Current Power usage: ". $power . " Watts<br>");
print ("Current Outside Temperature: " . $outtemp . "&deg;F</strong><br><br>");

# you do a query statment that return a strange
# SQLite3Result object that you have to use
# in the fetch statement below
$result = ($db->query('SELECT * FROM thermostats;'));
#print_r($result); # I wanted to see what it actually was
#
# The fetch call will return a boolean False if it hits
# the end, so why not use it in a while loop?
#
# The fetchArray() call can return an 'associated' array
# that actually means give you back an array of ordered
# pairs with name, value.  This is cool because it means
# I can access the various values by name. Each call to
# fetchArray return one row from the thermostats table
while ($res = $result->fetchArray(SQLITE3_ASSOC)){
        #var_dump($res);
        print ("<strong>" . $res["location"] ." thermostat,</strong><br />");
        print ("Currently: " . $res["status"] . " <br \>");
        print ("Temperature: " . $res["temp-reading"] . "&deg; <br \>");
        print ("Settings are: <br \>");
        print ("Mode: " . $res["s-mode"] . " <br \>");
        print ("Temperature: " . $res["s-temp"] . "&deg; <br \>");
        print ("Fan: " . $res["s-fan"] . " <br \>");
        print ("<br>");
}
$db->close(); # I opened it, I should close it
?>

There's not a whole lot I can point out that isn't already in the comments above.  Here's the web page as the browser displays it:

Yep, I've discovered how to put up a degree symbol.  Now, I have two processes that run in background all the time updating the database with different items.  That's a bit silly, so I'm going to combine them into one piece of code that goes over my lan and talks to the web devices as well as monitoring the XBee network for updates.  That way I only have one thing to make sure is running all the time.

And yes, the temperature up there is correct.  The rainstorm has lowered the temperature enough that I have all the doors open that don't get rained into.  Time to air out the house.