Showing posts with label Weather. Show all posts
Showing posts with label Weather. Show all posts

Friday, October 25, 2019

Naturally, Blogger Changed Something

Of course blogger changed how the blog appears ! That led to a mess of my background image tileing and messing things up in the appearance. After some messing around I managed to get it appearing somewhat reasonable, but had to change the background picture. I've used the same one for years and years now, but so be it; I can update with the times ... sort of.

I kind of like the look, and will stay with it for while, or until google gets a bug up their butt and changes things again.

Yes, I'm still alive. I haven't been blogging because I got myself involved with local politics and have been up to my ears in that effort for many months. I do get in a technical project once in a while, but haven't had the time to write them up.

I'll put them on the blog at some point, since they may be interesting to some people out there. Here's a quick list off the top of my head of the things I've been into that haven't surfaced here yet.

1. A horrible problem with my XBee network. My end devices would leave the network and not rejoin. That meant that some of the critical ones that controlled the AIR CONDITIONING didn't update the thermostats that control the temperature. Hot house or cold house depending on where it quit working. The thermostats protected me against using power during the Peak period, so the power bill didn't get out of hand, however it was painful to deal with. I have it mostly fixed, there's still one strangeness that I have to work on to get everything back to working reliably though.

2. I still have the parts for monitoring my 220V devices piled up with an enclosure, and haven't done anything else with them. They stare at me mournfully each time I walk through the garage. I'll get to them as soon as I can. To see that project's beginnings look here <link>.

3. Stumbled across a really cool phone from the crank-to-call days. I fully plan to put this thing in service. I may have to update a few parts, but maybe I can hide them inside. There's a lot of these out there, so I'm not going to worry much about the value afterwards, but I do want to keep it as original as possible. Here's some pictures of the phone; notice it has a BRASS earpiece !


It's an Australian model, but the parts are easily found to update it. This phone was updated over the years that it was in service. The crank was disconnected and the handle lost (or taken by the phone guy that did the work) the earpiece and mouthpiece elements were changed and finally the dial was added. It is really fun to play around with. I hope I can get the ringer to work; that would be great.

Might want a switch though.

3. Stupid AcuRite 5n1 weather station channel A died. Now I have to set it up for channel C since I that's the only way it will work. Sigh, don't happen to know what the frequency is for channel C would you? If you want to look at this project, it's all under 'Weather' in the list on the right side top of the page.

4. The battery life of my XBee end device temperature sensors has been a little over 5 months consistently. That's not as much as I calculated, but it's still pretty good. I'm getting tired of buying AA cells though. They're not expensive, but when you have a half dozen devices that run on three of them each, it can get annoying replacing them. I always seem to be out of the darn things.

So, I'm looking at those Li-Ion cells that sell pretty cheaply on Amazon. They seem to last forever, have close to 4 VDC output when full and are really fast to recharge. I have a bunch that I use in flashlights around the house and one installed in a sensor being tested (for months, these things take time). I suspect I'll be switching over to them on all the battery devices; they're just great since you can always have one charged up for replacement and don't have to run to Walmart because the AC crapped out.

And Etc. See, I'm still at it, just don't have the time to gather the graphs, pictures, measurements and such that it takes to make a coherent blog post about these things.

But, like the Terminator, "I'll be back."



Wednesday, March 9, 2016

My Weather Station is Back Online

Yep, I got it working. About a week ago my weather station Raspberry Pi died, and predictably, I couldn't get it back up without major work <link>. It's all better now running on a Raspberry Pi model B. Yes, I know that isn't the latest of the Pi series, but I had it already, OK.

The Raspbian Jessie booted to a graphic user interface directly, not to a command line. This was a bit of a surprise because I usually bring up the Pi's I have headless so I don't have to hunt down enough equipment to bring them up on my TV. But, since I had the stuff handy from trying to get it working before I gave up and just reloaded, it came up and I configured it, then stopped the GUI. Loading the GUI makes the boot sequence much longer and I usually just use putty on a laptop to control these machines, I don't need it.

Contrary to many posts out there, the configuration screens worked just fine. I was able to assign a static IP address and finish the configuration easily. There was a little hunting through the various menus and icons to find things, but that's part and parcel of any first time installation.

But installing my stuff turned out to be somewhat harder than I expected because, as usual, every piece of supporting software had changed and I had to hunt down what I needed. I used the latest (as of the date on this post) version of libusb, it was 1.0.20, which was several versions later than the one that I would get from a normal apt-get on the Pi.

This version worked well and had a few 'gotchas's' in it that I had to stumble through to get it to work. There was enough documentation and forum postings out there that I found to help me through the process.

I also used the latest rtl-sdr. It was pretty easy as well, but also had a couple of things that I had to chase down. Then I loaded the version of rtl_433 that I had set aside so I wouldn't have to keep making my changes in that code. With the poking around I did in libusb and rtl-sdr, this piece was super simple.

I hooked them all together by starting things on the command line and it worked well. I was able to read the weatherhead RF signal and save it off to my database. The interface I have to move the data around for viewing worked just as it had before. The only thing left was to implement start up  and restart using systemd instead of upstart.

For those of you that wonder why I needed all this, let me describe what's going on a bit. The usb implementation on the Pi is kernal based and not easily modified by the user. That means I had to get libusb and install it. libusb is a user space interface to the usb capabilities on Linux. Then, I had to be able to read one of those sdr radios so I could find the RF signal from the AcuRite weatherhead. I posted about this earlier <link>. Now that I had the signal, decoding it requires rtl_433 which is a piece of software that was developed to decode the signals of many of the wireless devices available such as thermostats, temperature sensors, door bells, weatherheads, etc.

So, it's software piece piled on software piece piled on ... Sure, I could have incorporated all the things I needed into one piece of software and just ran it, but that's not what leveraging software is all about. You use the piece that does the job you need and add special tweaks specific to your implementation. That saves time and doesn't cost much if anything.

With it working and the only piece left being restarting the processes, I dived into systemd. First though, a tiny bit of background. Unix (and linux came from unix regardless of what people say) has to have something to get it going. This used to be the init process. Init would load the various pieces of software and handle how they interacted. Over time init was replaced with various other things that had more features or less bugs. As is rampant in unix and its derivatives, everyone hated something about each of them. Raspberry Pi uses an operating system called 'Raspbian' which is Debian modified to work specifically on the Pi. There are a ton of other operating systems and the choice of which one to actually use is totally up to the person that installs it. I chose to use Raspian because it was the one the folk that produced the Pi chose to supply. I figured it would have the most support and documentation of various forms over time.

The Raspian that I installed with my first Pi used Sys-V init. This system five init is really old and complex to use. I hate complex, so I replaced it with the clearly superior init process call 'upstart'. Upstart is cool and easy to use once you install it. However, the Debian folk decided that the init process 'systemd' was better and moved to it. Raspberry folk went with the trend and moved to it as well.

That was the decision point that caused me to have to rebuild the weather station Pi from scratch. Upstart wouldn't allow the Pi to boot up at all with the new Jessie level OS that you will get when you update your Pi. But, I did all that and now I faced the dreaded systemd configuration. I looked at about a hundred web sites that discussed how to use systemd to handle starting and restarting a process, and quite honestly, was about to give up because everything was too complex and peppered with 'try this' to give me enough confidence to dive in.

At this point I want to mention that following Linux questions on a forum is so predictable. Someone has a problem and asks for help with it. The first response is to ask for more information about how the system is set up, what they saw that wasn't mentioned in the question, and usually some test or other that can't be run because the problem won't let them. Then a response asking them what they are doing. Then a response that says they should be doing it another way because that way is so much better. Then several posts about how it should be done regardless of the original intent. Lastly, the thread ends with no resolution because the discussion has wandered off into never, never land.

I saw that over and over. I finally stumbled across a blog posting (love blog postings) that whittled all the crap down to something amazingly simple compared to what I had been clicking through for hours <link>. Neil cut all the crap out and just showed how to handle a simple case. I didn't have to dredge through long discussions of 'units' or 'forking' or relative merits of daemons; I could just take his example and use it. It worked first try for one of the processes and third try for the other.

Then I configured logrotate to handle the log files I use for debugging and I was done. Finally.

I still have to back up the SD card and store the image on my NAS, and I'll probably back up the other Pi's just to be somewhat safer (obviously you can't actually be 'safe'). I'll also update my github repo for the various things I changed along the way, but that won't assure me of no problems the next time, just make it somewhat easier to recover.

What will really make it easier the next time is what I learned this time.

Thursday, March 3, 2016

How Can I Possibly Be Annoyed at the Raspberry Pi

Yes, they're all over the place and my Google news feed keeps telling me about how wonderful they are, but I almost threw one across the room today. The story is that my weather station Pi lost its SD card; they do that sometimes due to the heavy usage the cards aren't designed for. I've had it happen before, and thought I was prepared this time. You guessed it, I wasn't.

Yes, I have backups of the system, and backups of the software; the problem is that I didn't expect the folk that control the Raspberry Pi software to close some doors on me. You see, after I decided to update and upgrade the OS on the Pi, it wouldn't boot. I like to stay reasonably up to date on my stuff, but often wait a while to get the latest OS and tool changes. With the other non-technical projects I've been working on, it's been a few months since I took the time to update the little machines. During this period Raspberry Pi went from Debian Wheezy to Jessie.

Besides the usual wondering who the heck thinks up these silly names, I thought now would be a good time to update my systems. That was my first mistake; my second was when I didn't read ALL about the changes in Jessie. You see, they decided to use 'systemd' as their default init process and I switched to upstart quite a while ago because it was better. There's a few hundred pages out there on what's best and why, but I don't really care about that; I just want something that works, and I can keep updated.

Fine, I'd switch to the default for the Debian Jessie, but you can't update over a Debian Wheezy that has upstart on it; that makes the machine unbootable. To add insult to injury, this all happened when I was trying to update Wheezy. They don't update Wheezy, they switched me over to Jessie and just left me hanging.

Literally. Yep, hanging on a console message with the machine unable to boot.

I know how to get out of this mess and catch my machines up to the latest, but I really hate having to do it. At least it was my weather station and not the control system for the house. So my plans are to update to Jessie, then load all the packages I use to run this stuff, and finally to update my startup scripts to use the init they want me to use. Probably about three or four days of fooling around for the weather station, then I get to do it all over again for the house controller.

I want to be up to date with the latest because, eventually, the Pi Zero will be back in stock and I really want to play with it. No, I won't worry about updating something that will become a control part inside something else; I'll just use it like some other chip. I'll have to be careful to isolate any code on one away from the OS and come up with ways of starting processes that don't involve the OS, essentially treating them like little black boxes. At $5, I won't much care if they fail and have to be replaced. Sort of takes some of the fun out of them though.

The very next iteration of my devices will NOT use the unix facilities to start and stop processes or to monitor what they do. I'm going back to basics and working out ways to handle it that don't rely on some programmers data politics.

Subject change. A few folk have expressed disappointment that I haven't been posting about my tractor rebuilding, dog door, trenching, culvert, and other projects for the last few months. When I told them that the projects weren't technical, they said, "So?" I didn't have an answer because I've written about rattlesnakes, dogs, shades on the patio, etc.

Sorry tech weenies, I'm going to go back to my old ways and talk about surviving in the desert with a bunch of technology included. It just may be hydraulics, water flow, pruning techniques and such as well as little tiny computers and hacking into strange devices. Heck, I may well post about the technical aspects of using snake shot to control the varmints that try to kill me, as well as my continuing war with woodpeckers.

But my Pi 3B is already on order.

Friday, June 19, 2015

MQTT, Early Lessons Learned, and a Concern

So, I'm stepping through code trying to work out the best way of switching to mqtt for handling data from various devices and commands to control them and it DIED.

Yep, I couldn't publish anything even though everything was connected. Without anything being published, I naturally couldn't update the weather data base, and everything got lost. Losing data is no big deal, I'll get more; it's not having a clue what went wrong that sucks. I went and looked at the logging for mqtt and enabled it to syslog so I could at least see something, and that's when it started working again. Probably just a coincidence, but I'm leaving the broker logging enabled for a while. During this process I found a nice tool for interacting with mqtt from my Windows laptop mqtt-spy, it is a java application and allows me to stuff things into mqtt and monitor any of the 'topics', but all it told me was that nothing was getting published.

I'll just wait for it to happen again and see if I can get more information.

Otherwise, I decided to organize the topics differently. This is the first of what will probably be a large number of decisions like this ... mostly wrong. Instead of having the weather devices under the topic Desert-Home/Weather, they're now under Desert-Home/Device. I did this because I realized that weather is a collection of readings, not a device. So, the AcuRite 5n1 and the barometer are now Desert-Home/Device/5n1 and Desert-Home/Device/Barometer.

Yes, a comment on my last post by Grant got me to thinking.

As I was moving and changing code, I thought of something else; wouldn't it be cool to create a log of all the activity of the various things in one place on one machine so I could prowl through the day to day operation easily? Thus was invented mqttlogger.py. This new process subscribes to two topics: Desert-Home/Log and Desert-Home/Attention and simply writes what it receives to stdout. I redirect stdout to a file in the init configuration file and ta da, a log of whatever activity I want in one place. The topic Desert-Home/Attention is for stuff like the batteries going dead in the weather head or one of my battery operated devices. I'll eventually hook email to the Attention topic so I can be notified more easily.

The possible plan is to create a topic called Weather that has levels for the various readings that matter. There would be Desert-Home/Weather/CurrentTemp, HighestDaily, CurrentBarometer, RainfallToday, etc. Then I could pick and choose from the various values to present when I get that far. Frankly though, that's a lot of work and it may get modified to a JSON string that represents the weather reading right now. We'll see when I get there.

The weather station software has been converted to use mqtt now and is working reasonably well with the caveat that it may stop at any time because of the problem I mentioned up top. I can monitor the various things using mqtt-spy to see each device and the items involved including the log I created.

I'll post the code for the logger soon, I want to get a little time on it before I go and embarrass myself, and I want to give it a couple of days before cramming it into GitHub.

If it wasn't for that nagging possibility of it failing again I'd be really pleased.

Thursday, June 18, 2015

MQTT Conversion, First Steps

Last post I talked about trying out mqtt and being impressed with the ease of using it. It really went well and I decided to convert my house monitoring system to using this tool. So I took the very first steps. This is going to be a complicated conversion because I already have a system running and I don't want to break it while I'm moving to something different, so I decided on a few things to get me started.

I'm going to divide the sensors, saving data, and presenting data into different things. For example, I'm starting with the weather station because it's the simplest, but still has multiple sensing devices. The entire house is published under the 'topic' Desert-Home, so the weather station will be Desert-home/Weather, and the two current sensors are the Acurite 5n1 and the barometer out on the fence. The interesting thing here is that they are radically different.

The 5n1 is detailed on about a hundred posts on this blog, but finally wound up being a sensor on the roof that transmits in the 933MHz range. I catch and decode the signal with an SDR (software defined radio) and some code I stole and heavily modified. The readings from this sensor will be published under Desert-Home/Weather/5n1.

The barometer is a device I built up myself and put in a Stevenson Screen out on a fence post in the yard. It has an XBee that sends a JSON string to a Pi. This will be published under Desert-Home/Weather/Barometer.

I can see each of these devices by simply subscribing to 'Desert-Home/Weather/#'. The pound sign means to get everything that comes in under 'Desert-Home/Weather' and that will include every sensor I come up with related to weather.

I also decided to use JSON strings as much as possible for a few reasons. Most important to me is the ease of reading a JSON string when you're monitoring the network to see why something is messing up. The JSON string has key:value pairs that tell you what it is and what the value is. This helps a lot when I'm looking for a problem. This will be painful since most of my sensors use another technique entirely and they will have to be converted. Another reason important to me is that I don't have to write parsers to get the data back into variables, There are libraries that will do that for me and all I have to do is use them. Maybe that will make up for the pain I go through converting the devices.

So, since the XBee network I have is on a different Pi than the weather station RF software, I get to try out a multi-machine solution as the very first thing I try. Step one is done; I brought up a mqtt server (mosquitto, see previous post) and added publish lines to the XBee receive code and the weather station decoding. It worked on the second try; I had a couple of syntax errors to fix.

Mosquitto caught the data and all I have to do to see it is use the tool mosquitto_sub to watch the data as it is generated. The command line:

 mosquitto_sub -h "192.168.0.205" -t "Desert-Home/Weather/#" -v

shows me what is coming in as it happens:

Desert-Home/Weather/Barometer {"Barometer":{"temperature":"114.4","pressure":"1011.4","utime":"1434639446"}}
Desert-Home/Weather/5n1 {"sensorId":{"SID":"92","t":"1434664650"},"channel":{"CH":"A","t":"1434664650"},"messageCaught":{"MC":"0","t":"1434664650"},"battLevel":{"BAT":"7","t":"1434664650"},"windSpeed":{"WS":"6.0","t":"1434664650"},"windDirection":{"WD":"W","t":"1434664650"},"temperature":{"T":"114.6","t":"1434664632"},"humidity":{"H":"9","t":"1434664632"},"rainCounter":{"RC":"445","t":"1434664650"}}
Desert-Home/Weather/5n1 {"sensorId":{"SID":"92","t":"1434664650"},"channel":{"CH":"A","t":"1434664650"},"messageCaught":{"MC":"0","t":"1434664650"},"battLevel":{"BAT":"7","t":"1434664650"},"windSpeed":{"WS":"6.0","t":"1434664650"},"windDirection":{"WD":"W","t":"1434664650"},"temperature":{"T":"114.6","t":"1434664632"},"humidity":{"H":"9","t":"1434664632"},"rainCounter":{"RC":"445","t":"1434664650"}}
Desert-Home/Weather/5n1 {"sensorId":{"SID":"92","t":"1434664650"},"channel":{"CH":"A","t":"1434664650"},"messageCaught":{"MC":"0","t":"1434664650"},"battLevel":{"BAT":"7","t":"1434664650"},"windSpeed":{"WS":"6.0","t":"1434664650"},"windDirection":{"WD":"W","t":"1434664650"},"temperature":{"T":"114.6","t":"1434664632"},"humidity":{"H":"9","t":"1434664632"},"rainCounter":{"RC":"445","t":"1434664650"}}

Yes, the outside temperature is REALLY 114.4, remember I live in Arizona and this is the middle of June, but it's a dry heat, the humidity reading is 9%. As if that matters when the temperature is this high ...

But, also notice that I have both strings coming to the subscription even though they are very different devices, on different Pi's. Heck, they could be half way around the world and it wouldn't matter, it would still work. I tested that with a conspirator just this morning.

Now to follow my plans, I have to catch this data in some other process and save it to my database. I'll get to that soon, maybe even today, but I'm going to hold off on changing the presentation to the web for later because I have many different presentations to consider when I get to that point. See, currently I take everything presented from my data bases. If I'm using my Android app to look at the house, the data is coming out of the data base. When I send a command to close the garage doors, the command is executed and the app doesn't update until it is recorded in the data base. I may change that to reflect the latest reading from the mqtt server. We'll see, and then I'll probably change it.

The code is not in GitHub, I'm going to hold off until I get enough in there to make a significant difference. But, the example from the last post is exactly what I did with the difference above. This is a lot simpler and more versatile than I expected.

Wednesday, June 17, 2015

OK, Fine, I'll take a look at MQTT

A number of my readers have mentioned mqtt as a great tool for the kind of thing I'm doing around the house. A couple of them have pointed out that what I'm doing could be done much easier using mqtt and even sent me web sites to look at.

Naturally, I resisted. That's sort of the way I am, but also, the various web sites out there make it look so complex that I just didn't want to try and deal with it. Another stinking process with an API and things I had to learn to use it ... gag!

Then Glenn sent me a link to an Arduino implementation of publishing data to an mqtt server <link> and things clicked into place. What happened was the Arduino was too small and slow for fancy implementations, it had to fit in a couple of K and run on a 16MHz machine, so it was as simple as it could be. That got me to thinking --- there's really no protocol, you build it yourself, There's no registration, special language, ... none of the stuff that drives me nuts. You just use it.

Fine, I've eaten my words before, so I gave it a shot. AWESOME set of tools. This thing is going to be the center piece of a major update to my house control system.

OK, Glen, Brant, others too many to itemize, go ahead ... scream, "I told you so, but you wouldn't listen." I deserve it.

The way I started was to install the mqtt client software on my weather station Pi. I put a few lines in the python code to import, initialize an object and then a single line to send the data to a public mqtt server. Yes, that's literally all there was to publishing data, one single line of actual code.

Then I used the one of the tools to see if it actually got there and I could read it back.

pi@deserthome:~/src/other-things$ mosquitto_sub -h "test.mosquitto.org" -t "Desert-Home/#" -v
Desert-Home/Weather/String {"sensorId":{"SID":"92","t":"1434572606"},"channel":{"CH":"A","t":"1434572606"},"messageCaught":{"MC":"0","t":"1434572606"},"battLevel":{"BAT":"7","t":"1434572606"},"windSpeed":{"WS":"4.0","t":"1434572606"},"windDirection":{"WD":"SSE","t":"1434572587"},"temperature":{"T":"111.7","t":"1434572606"},"humidity":{"H":"11","t":"1434572606"},"rainCounter":{"RC":"445","t":"1434572587"}}

Desert-Home/Weather/String {"sensorId":{"SID":"92","t":"1434572606"},"channel":{"CH":"A","t":"1434572606"},"messageCaught":{"MC":"0","t":"1434572606"},"battLevel":{"BAT":"7","t":"1434572606"},"windSpeed":{"WS":"4.0","t":"1434572606"},"windDirection":{"WD":"SSE","t":"1434572587"},"temperature":{"T":"111.7","t":"1434572606"},"humidity":{"H":"11","t":"1434572606"},"rainCounter":{"RC":"445","t":"1434572587"}}

Desert-Home/Weather/String {"sensorId":{"SID":"92","t":"1434572623"},"channel":{"CH":"A","t":"1434572623"},"messageCaught":{"MC":"0","t":"1434572623"},"battLevel":{"BAT":"7","t":"1434572623"},"windSpeed":{"WS":"2.5","t":"1434572623"},"windDirection":{"WD":"SSW","t":"1434572623"},"temperature":{"T":"111.7","t":"1434572606"},"humidity":{"H":"11","t":"1434572606"},"rainCounter":{"RC":"445","t":"1434572623"}}

If you've been following the weather station posts, these lines will look really familiar. The tool mosqitto_sub is a test subscriber that took the public server I used, 'test.mosquitto.org', and looked for the published data keyed under 'Desert-Home/Weather/String' and printed it to the console of the Pi. So, what was happening was that my weather station Pi was reading the data from a radio and sending it over the internet to a server. Then on a different Pi, I was reading the data back as it came in from the server I subscribed to. Three machines involved working over the internet and all I had to do was write one line of actual code to do it.

To try it out, install the mqtt client software for python and the command line tool I used above. No, I didn't misspell it, there really are two 't's in there.

 sudo apt-get install mosquitto mosquitto-clients
 sudo pip install paho-mqtt

This will actually install the mqtt broker and start it up, but you can stop it by the command:

sudo /etc/init.d/mosquitto stop

Then, in some code somewhere, put the lines:

import paho.mqtt.client as mqtt

mqttc = mqtt.Client()
mqttc.connect("test.mosquitto.org", 1883, 60)
mqttc.loop_start()

To create the object, initialize it and start it up. Then the single line somewhere you are logging or recording something like this:

mqttc.publish("Desert-Home/Weather/String",buff);

The string "Desert-Home/Weather/String" is called a 'topic'. This is a hierarchical set of keys that you construct for your own purposes. I chose "Desert-Home" for obvious reasons, then "Weather" because it's for the weather station, then "String" because I was publishing the JSON string produced by my code that reads the AcuRite 5n1 weather head. I may create another item "OutsideTemperature" to hold the latest outside reading from the fence post sensor in which case it would be:

 "Desert-Home/Weather/OutsideTemperature."

Then to get the data back as it is published to the server, use a line like:

mosquitto_sub -h "test.mosquitto.org" -t "Whatever/you/decided" -v

Slick isn't it. There's no mystery here, no special stuff you have to learn, it's all totally up to you how you use it. To make it more versatile, the '#' and '+' characters have special meaning that can help you debug and select things. Check the various sites out there for the usage of those symbols.

There's a ba-jillion web sites out there that talk about mqtt, so I'm not going to recap the light-weight, short transaction, fail safe blather that they cover ad nauseum, instead over the next few weeks, I'm going to convert my house monitor and control system to actually use it and try to show you how to follow my footsteps improving and changing it to suit your particular needs. I'm sure I'm going to mess it up and have to redo things a couple of hundred times, but this should, at least, be good for a laugh.

During this process I'm going to visit my sensor devices and update the code on them to my latest experience level and implement several changes to them as well. JSON strings as data output and input so I can get rid of parsers, that kind of stuff. I'll gateway the devices from the XBee network to messages that will be published to mqtt and caught by whatever needs to handle the data. I don't know what I'm going to do with the Iris devices yet, but something will come to me.

Stay tuned.

Sunday, April 12, 2015

Getting Back To The Weather Station

I know I've worn out my welcome about the AcuRite 5n1weather station, sorry. This time it's a bit different and I'm really, really open to input and ideas. I decided that the console that came with the weather sensor is too strange and quirky to deal with long term and set up a Raspberry Pi to act as my weather station. I used a radio and directly receive the transmissions from the weatherhead <link> instead of the USB interface I worked on forever <link>

I have a lot of work to go on it, but I'll update it over time as I figure out what I actually want. To date, I set up a little SQLite3 database and record each of the readings there on a periodic basis. Currently, I update each record either as it comes in, or every minute. The reason for the difference is that my barometer and outside temperature sensor is on my XBee network and handled by a different machine, so I only interrogate it on a minute basis.

Yes, the little Pi weather station is a separate machine and I have it talk to the Pi that controls my house to get the two readings.

Actually, let me elaborate on this a bit. Remember way back when I posted about bringing up an HTML interface for the various processes I run using CherryPy <link>? That little effort worked incredibly well. For most processes I have at least two interfaces, one that I can read and one that is formatted in JSON that a machine can read. So, to check on a particular process, I just type the IP address and a port number into the browser and the process responds so I can see whatever I set up. This interaction doesn't care what machine it's on or where the request came from. Using this I have the weather software on one Pi, then get the readings for barometric pressure and outdoor temperature from another one. It's so cool.

I may expand on this idea and have several Pi's running doing different things for me. I could put the XBee network on a machine by itself and have other things talk to it to get the data. Sure it raises the risk of failure because it increases the number of machines that can fail, but It also means I have all the power of a Pi to do a task and not have to worry about things bogging it down. I may move all the upload processes to a single machine so that stuff is separate as well.

But enough of that, I'm here to talk about the weather station. I keep data on a periodic basis for: Rain counter, Outside Temperature on the fence, Outside Temperature at the weatherhead, Barometric Pressure, Wind Speed, Wind Direction, and Humidity. I also keep a daily record, written at midnight that hold the Barometric pressure at that moment, the high and low temperature for the day, and the rain counter at that moment. I create this record once a day, so I can use it as a daily data point on a graph.

From this I can calculate the daily rainfall, and when I have enough data, the weekly, monthly, etc for my location. I have enough data to graph the temperature on a daily basis, and even use the midnight record to go longer term. That stuff, of course, will depend on gathering the data over time. Or, I could download the data from Xively and fill it in for the past if it becomes relevant.

This looks like it will eat up about a megabyte a week in space, and I have roughly 4 Gig available on the Pi, so I have some time to get an NAS online to hold the data long term.

But, as usual, I have no idea what the heck I'm doing. I don't have a clue what I may want next month, or what is the best way to gather the stuff that I don't know about yet. The station is running right now, but it isn't hooked into the web yet so I can't show it off. I'll get to that in a week or so.

I looked into loading the stuff up to Weather Underground, but that didn't excite me very much. I've been fighting cloud providers for a few years now and I'm not impressed too much anymore. With the terabytes of NAS storage available these days and the ease of bringing up a significant little computer, who needs the headaches of specialized protocols and changing terms of service?

So, you folk that have weather stations and such, what am I missing?

Oh, the code isn't on GitHub yet, I'm not sure how I want to arrange it there. I may set up another repository for this instead of adding it to what's already there. I need to think about this a while. But, if anyone wants to take a look, let me know and I'll move more quickly and make it available.

The next post on this project is here <link>.

Saturday, February 14, 2015

Acurite Console Barometer: OK, This is it.


Back in a previous post I tried to decode the barometer reading from the AcuRite console.  I just couldn't get it, but one of the readers, Play Kube, stepped up and nailed it for us. The post is here <link> and his findings are down in the comments.  He emailed them to me and I posted them for him. Then I added code to my stuff and recorded the readings for a few days and this is the correlation to my (very accurate) fence post barometer. I spent quite a bit of time making sure my barometer compared to the weather stations around me and it was dead on.  The decoded AcuRite readings follow mine really, really well.  Take a look:


Isn't that great?  There's a couple of caveats though: The console barometer is NOT temperature or altitude corrected.  That's because we don't have access to the temperature or altitude corrections that the chip manufacturer supplies inside the chip. These values can be read and the math applied to get the reading dead on, but we don't have them to work with.

The way I corrected the readings was to simply adjust the algorithm to compensate.  This means that if I move the console up or down by a significant amount (I move to sea level) or the temperature changes (put it in the sun) the readings will change.  However, in most of our houses the temperature stays pretty stable and it's unlikely we'll be moving soon.  If one of these does happen, just readjust the algorithm and you'll be all set for the next time.

Looking at the chart, the difference in the readings is less than a mbar and tracks exactly.  Here's the code I used to decode the reading:

                    float bar = 6.23 * (R2[23] << 8 | R2[24]) - 20402;
                    weatherData.barometer = bar / 100; // convert to mbar from pascals
                    weatherData.barometer += 81.1; //adjust for altitude

It's a simple linear equation like a line graph from high school algebra.  Take the last two bytes and form a single number from them, then use a slope of 6.23 and an offset of -20402.  This will get you the barometric pressure reading if you were at the same altitude and temperature as the person that figured it out. So, to adjust it to your location, just go look up the barometric pressure reading for your location at the same time and add the difference in.  That's what the 81.1 number above means, I just added it in as the last step. The changes will be in GitHub soon.

Don't think this is too simple a solution.  The manufacturer of the chip put a lot of work into getting the sensor to follow pressure changes, then they put the adjustments for temperature and fixed altitude in the chip so they can be read and applied.  There was a lot of work done to get it to work. The trick was to get the slope correct, then the offset was just the method of getting it to line up vertically.  The altitude offset is simply necessary to adjust for the specific location.

Slick, but remember, change altitude or temperature and you may have to revisit the algorithm and change the values. We owe this discovery to Play Kube for sending me the mail a few days ago.

I still haven't found the battery level from the weather head though.

Wednesday, February 4, 2015

Reading the AcuRite 5n1 Sensor Set, This time with RF

I've managed to read the RF signal coming from the AcuRite weather head.  It wasn't too hard to do since I had some significant help from other folk that have done the same thing.  The problem was only that there wasn't anyplace out there that I could just look at for an answer that was somewhat coherent to someone that wasn't conversant in the latest digital tuner techniques.  It turns out the answer is quite simple to do, once you work your way through the confusion.  I hope I can clear it up for folk because this is the last piece of the AcuRite puzzle that I think is worth pursuing. 

First, get one of these:


This the tuner chip that is used in digital TV that has been hooked to a USB interface and enclosed in a plastic case.  The chip is a RTL2832 and has a huge bandwidth; it can be used to receive AM, FM, SSB, etc all the way up past the 900MHz range that the AcuRite transmits on.  It's a fun device in and of itself and is the subject of blog posts all over the web. You can listen in to commercial flights, the cops parked at the corner watching your garage, even the doorbell transmitter in your neighbors house.  I didn't want it for boring things like this, I wanted to get the rainfall count out of my 5n1 sensor set without using the console.

So, I downloaded a great tool for doing this kind of thing called SDRsharp and installed it on my laptop.  After visiting several web sites that described how to use its features, and listening to the five or six FM stations I can receive out here, I went looking for the AcuRite sensor.  I found it.


That's it, the spike that is showing on the capture.  It took me a little while prowling around the spectrum to find it, but there it is, Oh ... wait ...


Notice in the bottom panel that there are three different signals that are on almost exactly the same frequency?  Yep, I'm picking up three devices up in the 433MHz area and any one of them could be what I'm looking for.  After a bunch of looking, it turns out the first one I found is the one I want, but how to I get the data out of it? Heck, how do I separate the one I want from the others?

This has already been solved for us under Linux by a tool called rtl_433.  But before we talk about that, use SDRsharp to zero in on the frequency you need to use.  There's a little drift between devices both at the radio and at the transmitter, getting the exact frequency reading will help you later.  Write it down somewhere.

OK, I'm getting too technical without explaining what is going on.  These little tuner chips were designed for digital TV and the description (stolen from here <link> is:
What is RTL-SDR?
RTL-SDR is a very cheap software defined radio that uses a DVB-T TV tuner dongle based on the RTL2832U chipset. With the combined efforts of Antti Palosaari, Eric Fry and Osmocom it was found that the signal I/Q data could be accessed directly, which allowed the DVB-T TV tuner to be converted into a wideband software defined radio via a new software driver.
Essentially, this means that a cheap $20 TV tuner USB dongle with the RTL2832U chip can be used as a computer based radio scanner. This sort of scanner capability would have cost hundreds or even thousands just a few years ago. The RTL-SDR is also often referred to as RTL2832U, DVB-T SDR, or the “$20 Software Defined Radio”.
There's an entire web site dedicated to these little devices at http://www.rtl-sdr.com/ and another at http://www.rtlsdr.com that describes many of the uses people have come up with, so I won't go into it much more deeply.

Next, there's a Linux library that supports this and a really cool tool you can run on your PC to play around. The tool is called SDRsharp and has a dedicated website <link>; just go there and select the download link to get it.  The screenshots above are taken from this tool

Now, before I get into describing how to use this on a Raspberry Pi, you should be aware that these little radio receivers pull a lot of power.  When you plug them into the Pi, it will suck so much power that the Pi will reset and have to boot back up.  If your power supply is not up to it, the radio won't work properly.  A powered hub will help isolate the Pi from the high current, but isn't absolutely necessary if you power supply is strong enough and you don't mind it rebooting when you plug the receiver in.

The Linux library is called  rtl_sdr and is at http://sdr.osmocom.org/trac/, but don't download it yet. Don't misunderstand, this is a nice tool, but not exactly what you want to read the AcuRite weather head. What you want is a tool called rtl_433 that was developed specifically to decode the various devices that operate in the same unlicensed frequency range.  Things like temperature sensors, doorbells, full blown weather stations, etc.

There are a lot of blog posts out there on how to install the rtl_433 tool, but none of them worked for me.  I had to actually follow the instructions as written in the readme file that came with it because the authors had recently added the general purpose library for SDR (software defined radio) when they found out it needed to be changed to help their tool work better. When I downloaded the zip file, I got it all.  It took me about three days to figure that out and get it to compile and work.  Then, when I went back later in preparation for this post, they had changed rtl_433 a bunch. I guess someone got a burst of energy and they updated everything in sight. They had gone back to installing rtl_sdr as a separate step and that would make things much harder.

I just created a new one under my login on GitHub.  I really didn't like the idea of things changing every time I wanted to do something and using it for my weather station would depend on two projects out there that can change at any time.  I tried to use the Git fork and other tools and got totally confused with the various methods of backing up in a respository; I guess my very own version isn't a bad solution.

There is a dependency on libusb though.  If you've already installed my USB code that previous posts described, you've got that requirement already.  If you haven't, you'll need to install libusb. Go to my post on this <link> and follow the part about installing libusb and libudev-dev.  This is a USB device (of course) and is very, very similar to the code I created to read the console device.

Now you're ready to work on rtl_433, so just go to github and grab it <link> by clicking on the 'Download ZIP' button on the lower right.  To build it though, they use a tool called 'cmake' that isn't on the Pi by default.  Fortunately, it can be installed easily

sudo apt-get install cmake

Now you can (finally) just follow the instructions in the readme that comes with the library to build it.  Don't do like I did and think I knew everything and install the rtl_sdr library and try to get it to link up properly with rtl_433; that is a path to almost terminal frustration.  Just build rtl_433 from my copy. You'll probably want to rename it once you get it unzipped to something easier to type.  Here's the commands I used to build mine:
cd Desert-Home-rtl_433-master
mkdir build
cd build
cmake ../
make
Now, you are almost ready to actually read the RF from the weatherstation, but (another one of those) the decoder included with rtl_433 is wrong. I don't know if the author of that particular piece had the same kind of problems I did originally, but he made some of the same mistakes I did.  So, I put together my own version that decodes my weatherhead and corresponds with the console. These changes are part of the zip file you downloaded; you don't have to do anything special.

The great thing about doing it this way is the weatherhead provides a checksum on the data.  That means no more undetected short packets or weird data because something in the AcuRite console is messed up. It also means you only get the 5 sensors included in the weatherhead: temperature, humidity, wind speed, wind direction, and rainfall count.  But, since I installed my own barometer <link>, and plan on building anything else I may need, that's good enough for me.

Just for completion's sake, here is sample output from my weatherhead:

pi@deserthome2:~/src/Desert-Home-rtl_433-master/build$ cd src
pi@deserthome2:~/src/Desert-Home-rtl_433-master/build/src$ rtl_433
Registering protocol[01] Rubicson Temperature Sensor
Registering protocol[02] Prologue Temperature Sensor
Registering protocol[03] Silvercrest Remote Control
Registering protocol[04] ELV EM 1000
Registering protocol[05] ELV WS 2000
Registering protocol[06] Waveman Switch Transmitter
Registering protocol[07] Steffen Switch Transmitter
Registering protocol[08] Acurite 5n1 Weather Station
Found 1 device(s):
  0:  Generic, RTL2832U, SN: 77771111153705700

Using device 0: Generic RTL2832U (e.g. hama nano)
Found Rafael Micro R820T tuner
Exact sample rate is: 250000.000414 Hz
Sample rate set to 250000.
Sample rate decimation set to 0. 250000->250000
Bit detection level set to 10000.
Tuner gain set to Auto.
Reading samples in async mode...
Tuned to 433920000 Hz.
Detected Acurite 5n1 sensor
wind speed: 2.0 mph, temp: 54.5F, humidity: 67%
Detected Acurite 5n1 sensor
wind speed: 1.5 mph, wind direction: NE, rain counter: 344,
Detected Acurite 5n1 sensor
wind speed: 1.5 mph, temp: 54.5F, humidity: 67%
Detected Acurite 5n1 sensor
wind speed: 1.5 mph, wind direction: NE, rain counter: 344,

Notice how the build process put the files in the directory rtl_433/build/src/ ?  I'm not responsible for that ... that was the way I got it.  Just move them wherever you want them to reside at run time.

Remember way above I told you to write down the frequency you found the weather head transmitting on?  This is where you may have to use it.  If it drifted far enough, you won't get a good signal from the transmitter up on the roof or out in the yard.  You can simply specify the frequency you want to listen to with the '-f' parameter like this:

rtl_433 -f 433.915e6

This means to set the receiver at 433.915 MHz.  I actually use this number for my weather head.

There's still some stuff to be done from my perspective, I want to put some code in to present the data as a JSON string so it can be used exactly like I did in the other version.  I also want to wait until I have both types of messages in hand before I output anything.  I'll be doing that over time and moving to exclusively use the RF from the weather head.  That will eliminate all the little irritations that the console entails. So, if you grab the code at some point down the line, there will be changes that reflect my specific use; if you want this particular version, the commit # in Github is 5c72e32 and you can get to it by clicking on 'Commits', then finding the number in the list and clicking on the browse next to it that looks like this <> . That will get you back through any changes to the level it was when I wrote this.

This will also make the Raspberry Pi a possible replacement for their expensive bridge device that is limited in what it can do and completely closed to us changing it.

But what will I do with the console?  It's actually an attractive device and I think I'll put it in the kitchen somewhere.  It will give me a nice conversation piece in the most often used room.

Have fun.

Edit Feb 13, 2015:  One of the readers, John, had a problem with attaching to the SDR device.  He chased it down to some code needing to be added in the USB open code.  He emailed it to me and I tried out his fix and it worked fine.  The problem relates back to removing the default hardware driver when starting up like I mentioned way back in a previous post <link>.  I tested it to be sure it would work for me as well and it was great.  I've already put the change into the github repository as commit number 86c70cf .  See, it does pay off to read all the posts after all. Feel free to grab it

Saturday, January 31, 2015

Let's Finish Off the Barometer Project.

In the last entry <link> I posted the code for my barometer; I took a simple program of a few lines and turned it into a monster that does everything imaginable.  Now, I should actually set it up to work and put it in service.  Yes, my XBee shield came in and worked fine first try, so I mounted the little breakout board on it and added four wires to connect the barometer chip to the Arduino pins and tested it.


Made a nice little package when I got it all put together, so I dismantled the Stephenson Screen on my fence and tie wrapped it inside.


I had the old temperature sensor in there for years with no problems, so I don't think I need to get any fancier than that.

One thing I did in the code that I didn't mention in the last post was how to correct the reading to adjust for sea level comparisons.  See, to make barometer reading useful around the world, weather folk adjust them for the altitude so that there is a common base for comparison.  There's some really good explanations for why and how to do this out there, so if you don't already understand this, do a quick search.

From our perspective, this is incredibly easy; just call a different routine to read the data:

float pressure = bmp.readSealevelPressure(altitude)/100.0;

This statement will get the pressure adjusted for 'altitude' and the division by 100 will convert it from Pascals to millibars.  If you want inches of mercury, just look up the conversion factor.  The Adafruit library has already got the code for the conversion, all you have to do is look up your altitude in Google Maps, or read it off a GPS at the location where you put the barometer.

I've already adjusted my house monitor code to use this device and if you look at my house display, the outside temp and barometric pressure are taken from the new set up. I was lucky enough to catch a small storm so I could compare it with the other stations in my area, and it was right on.  It's really great when a sensor does exactly what the datasheet says it will.

I really like this sensor; the temperature measurement is very stable as is the barometer.  It's a snap to read and only takes two signal wires for all of it.  Since there's room on the device for more sensors, I may think about luminance or something in the future.

All this playing with the AcuRite weatherstation has been fun, and now I'm going to look at conquering the RF signal it sends.  What I'm going for now is far, far away from what I started off to do, but isn't that how it goes?  I want to grab the RF directly and use that data along with any other sensors I need to add and bring up a separate Raspberry Pi to handle weather presentation. Remember, the code is already in GitHub for you to grab <link>.

Funny how these projects tend to take on a life of their own.

Wednesday, January 28, 2015

My Barometer, and Tips and Tricks With XBee Device Code.

In the last post <link> I put together a basic barometer to substitute for the silly one on the AcuRite console that came with my 5n1 weatherhead.  My XBee shield still hasn't come in, but I got a wild hair and decided to go ahead and put together the code while I waited.  As I was stealing pieces from various other projects I've done and combining it to implement this latest one, I realized there was a significant piece of code I haven't shared yet.

Early on I got tired of having my XBee set up wrong because I forgot something when I configured it and created a little piece of code to check the configuration to some degree at boot time on the Arduino.  By sending AT commands to the XBee, you can check things like software version, operating mode, etc.  Then if something was messed up, you can correct it easily without a long debugging session to discover some minor setting error.  I also use this to get the name of the device, and the address that it normally sends to.  Using this piece of code makes the entire device more configurable.  For example, you need to change to broadcast, instead of a specific address, to monitor the packets, just use XCTU to remotely change the DL, DH attributes and reset the Arduino so it reads the new address.  If you decide to change the device name, remotely change the NI attribute, and again, reboot the arduino.  I've been using this in one form or another now for a while to help work the kinks out of the latest device I'm working with.

Yes, you can change the configuration of the XBee that you stuck in the corner of the attic remotely,  This has been a capability of XCTU for a long time, and the latest version of XCTU makes it even easier.  To remotely reset an Arduino, I simply decode a command I send and then call location zero.

These two tiny tricks makes minor changes to the network really easy and can be done from my recliner.

The code that follows (below somewhere) is a simple thing with complete XBee interaction,  I have examples of sending formatted strings containing float data through the XBee, parsing a long unsigned integer out of an XBee message and using it, simulating the data portion of an XBee message for debugging, setting up a timer to send status periodically, reading some of the configuration out of the XBee to be sure it's OK before continuing the work.  That kind of thing.

Since the barometer code is so easy to understand, I thought that would be a perfect device to put all this stuff in and use as an illustration.  Go take a look at the barometer <link> then come back here and I'll step through some of the things that you can use in your own code to make the XBee a valuable device in do it yourself home automation.

First, just to get it out of the way, resetting the Arduino from software.  If you haven't hit it yet, you will, that time when you know a reset will cure a malfunction on the Arduino, but its up high, out in the yard, in the dark on the side of the house; somewhere hard or inconvenient to get to.  Wouldn't it be nice to just send a command to it and have it reset itself?  A software reset is easy to do.  If you need a hard reset (similar to pulling the plug) this won't do it; but those are not often needed.  Just put this line somewhere at the top of the code:

void(* resetFunc) (void) = 0; //declare reset function @ address 0

and when you need it, call it like this:

resetFunc();  //This will reset the board and start all over.

Yep, that's all there is to it.  I use this a lot for overcoming memory leaks in libraries that haven't been tested long enough, memory loss from using Streams in c++, that kind of problem.  It also helps overcome my own bugs from time to time.  As an example, in the code below I use it twice.  I have one software reset that is on a timer to reset the board every day around midnight.  This is to recover memory lost to whatever code does a malloc() and then doesn't remember to do a free() to release the memory back.  This is a very common problem (called memory leak) and has plagued tiny computers as long as they've existed.  When you want to run an Arduino 24/7, you have to code defensively.  The other instance is when I send a reset command over the XBee network to cause the board to reset.  I use this to test things like power failures, how long it will take to settle down, and to see if a reset will work before I get the ladder out to take a physical look at it.

Since I mentioned remotely resetting the board, I'll describe how I do that, but I need to explain how I use the IDE itself first.  Most of you already know that the Arduino IDE supports 'tabs', but the new folk haven't caught up to that yet.  Tabs are actually separate source files that the IDE stores in the same directory as the sketch.  These are extremely nice for organizing code so you don't have to scroll through a couple of hundred lines of code to get to something you want to work on.


What you're looking at above is the Arduino IDE with three tabs.  One is for the bulk of the barometer code, the next is for some utility routines I use a lot, and the one selected is the XBee code I use on a lot of my devices.  I opened the window that controls tabs so the new folk can get a look at the tool provided for working with tabs.  Remember, each tab is actually a separate file in the sketch directory.  When you compile something, the files are all concatenated together and passed to the compiler, this is just a convenience to help you organize your project.  There's a couple of tutorials out there that mention tabs, and I've heard of a book or two that do, but most of the instructions are great at telling you how to install it, but not how to use it effectively.  So, don't be afraid to poke around and experiment.

No, I'm not going to write a book about it.  If I did, they'd change it and I'd have to have a second revision.  I've got projects to work on instead.

Anyway, The picture shows how I decode the command and reset the board.  Over time I've worked out my own particular method of sending and receiving packets that works well for me.  Everything is done in a string; I don't try to send integers, floats or compact the data.  For home automation, interactions are measured in a minimum of seconds and plain text traveling between the machines is perfectly adequate.  It also makes monitoring what's going on much easier since I don't have to decode anything to read it, it's already text.  So my command to reset the barometer is:

Barometer,Reset

I can type that right into an XCTU session from my recliner and the barometer out on the fencepost (where it will eventually reside) will reset itself.  Yes, I have to construct a packet in API mode 2 format, but XCTU has a packet generator that works pretty well to help with that.  

Now, let's talk about AT commands to the XBee itself.  When you look at the various tutorials on the web, they love to type AT into the XBee console and look for the OK that comes back.  Then they teach you how to set up the XBee using these commands.  This is good, but not very useful if you're going to actually do a project that will run all the time and have to survive a lot of the normal problems such as RF noise, distance, flaky devices, etc.  For that you have to go to API mode and send fully constructed packets between machines.  I have already covered the various pitfalls and have instructions on how to set this up on my XBee page, so I won't go over it again here, go here <link> for that when you need it.  What I want to show you is how to send an AT command, get something from the XBee and then use it.  Doing this you can double check the configuration, get the device name, etc.  It's a really good feature to understand.  As an example, here are the first few lines executed on my barometer:


I know, it's a dumb screenshot.  I did this on purpose, to illustrate again how tabs can work for you.  I have selected the barometer tab and scrolled down a bit to the setup() routine.  See how you can flip between tabs to do things instead of scrolling though tons of lines of code and losing your train of thought?  I like this feature.

Anyway, I set up the XBee code, then call getDeviceParameters().  This routine sends a series of AT commands to gather various items and then returns.  Then I display the parameters so I can tell if I have the XBee set up the way I want it.  I actually store the string 'Barometer' in the NI command; this is the name of the device and I use it to check incoming packets to see if they're for this device and include it in the out going status packets I send.  This way I can change the name of the device by simply changing the parameter on the XBee.  Here's the code that does this:

void getDeviceParameters(){
  uint32_t addrl = 0;
  uint32_t addrh = 0;
  uint32_t daddrl = 0;
  uint32_t daddrh = 0;
  
  if(sendAtCommand((uint8_t *)"VR")){
    if (atResponse.getValueLength() != 2)
      Serial.println(F("Wrong length in VR response"));
    deviceFirmware = atResponse.getValue()[0] << 8;
    deviceFirmware += atResponse.getValue()[1];
  }
  if(sendAtCommand((uint8_t *)"AP")){
    if (atResponse.getValue()[0] != 0x02)
      Serial.println(F("Double Check the XBee AP setting, should be 2"));
  }
  if(sendAtCommand((uint8_t *)"AO")){
    if (atResponse.getValue()[0] != 0)
      Serial.println(F("Double Check the XBee A0 setting, should be 0"));
  }
  
  if(sendAtCommand((uint8_t *)"NI")){
    memset(deviceName, 0, sizeof(deviceName));
    for (int i = 0; i < atResponse.getValueLength(); i++) {
      deviceName[i] = (atResponse.getValue()[i]);
    }
    if (atResponse.getValueLength() == 0){
      Serial.println(F("Couldn't find a device name"));
    }
  }
  if(sendAtCommand((uint8_t *)"SH")){
    for (int i = 0; i < atResponse.getValueLength(); i++) {
      addrh = (addrh << 8) + atResponse.getValue()[i];
    }
  }
  if(sendAtCommand((uint8_t *)"SL")){
    for (int i = 0; i < atResponse.getValueLength(); i++) {
      addrl = (addrl << 8) + atResponse.getValue()[i];
    }
  }
  ThisDevice=XBeeAddress64(addrh,addrl);
  if(sendAtCommand((uint8_t *)"DH")){
    for (int i = 0; i < atResponse.getValueLength(); i++) {
      daddrh = (daddrh << 8) + atResponse.getValue()[i];
    }
  }
  if(sendAtCommand((uint8_t *)"DL")){
    for (int i = 0; i < atResponse.getValueLength(); i++) {
      daddrl = (daddrl << 8) + atResponse.getValue()[i];
    }
  }
  Controller=XBeeAddress64(daddrh,daddrl);
}

uint8_t frameID = 12;

boolean sendAtCommand(uint8_t *command) {
  while(1){
    frameID++;
    atRequest.setFrameId(frameID);
    atRequest.setCommand(command);
    // send the command
    xbee.send(atRequest);
    //Serial.println("sent command");
    // now wait a half second for the status response
    if (xbee.readPacket(500)) {
      // got a response!
  
      // should be an AT command response
      if (xbee.getResponse().getApiId() == AT_COMMAND_RESPONSE) {
        xbee.getResponse().getAtCommandResponse(atResponse);
  
        if (atResponse.isOk()) {
          //Serial.println("response OK");
          if (atResponse.getFrameId() == frameID){
            //Serial.print("Frame ID matched: ");
            //Serial.println(atResponse.getFrameId());
            return(true);
          }
          else {
            Serial.println("Frame Id did not match");
          }
        }
        else {
          Serial.print(F("Command returned error code: "));
          Serial.println(atResponse.getStatus(), HEX);
        }
      }
      else {
        Serial.print(F("Expected AT response but got "));
        Serial.println(xbee.getResponse().getApiId(), HEX);
      }
    } 
    else {
      // at command failed
      if (xbee.getResponse().isError()) {
        Serial.print(F("Error reading packet.  Error code: "));  
        Serial.println(xbee.getResponse().getErrorCode());
      } 
      else {
        Serial.println(F("No response from radio"));
      }
    }
  }
}

It's not perfect code, and it doesn't serve the 'general case', but it is easy to understand and does the job.  Notice in the routine sendAtCommand() that I hang up in an infinite loop waiting for a response to each AT comand I send?  Yes, that's on purpose.

On a network like mine where there are a number of XBees sending data at essentially random intervals, traffic can be mixed in with the responses to the AT commands.  I send a request for the software version at the same time as a time update is received from my house clock.  The XBee will send the packet from the clock first and I could miss the response to the AT command.  Also, the receive for the packets is interrupt driven and characters can be missed by simultaneous sends and receives.  To overcome these kinds of problems, I wait for the response about a half second, if it doesn't come back, I try it again.  This guarantees that I will accumulate the items I need during initialization regardless of traffic conditions, it just may take a couple of tries to do it.  It's actually kind of fun to watch it during high traffic conditions; it tries each command as many times as necessary to gather the data and then returns to let setup() print them out.  Here's what the output looks like when I run it:

Barometer Version 1 Init...
This is device Barometer
Device Firmware version 2370
This Device Address is 0x0013A200 0x4089D915
Default Destination Device Address is 0x0013A200 0x406F7F8C
Setup Complete
Mem = 276
Barometer: temperature: 77.5, uncorrected pressure: 937.8
See how the name of the device, long address, version, destination long address can be gathered directly from the XBee?  This is nice to guarantee that you set things up correctly when you try your code out.

Now, let's look at sending the actual data that I gathered from the barometer chip.  I've learned to like JSON encoding as a method of passing data and am converting all my devices to send data using this format.  I haven't gotten very far on that project, but the barometer is new, so I use it here.  I use this line to construct the status text that is sent:

void sendStatusXbee(){
  float temperature = bmp.readTemperature()* 9/5 +32;
  float pressure = bmp.readSealevelPressure(altitude)/100.0;
  sprintf(Dbuf, "{\"%s\":{\"temperature\":\"%s\",\"pressure\":\"%s\",\"utime\":\"%ld\"}}\n", 
            deviceName, // originally read from the XBee
            dtostrf(temperature, 1, 1, t), // This is a text conversion of a float
            dtostrf(pressure, 1, 1, p),  // same as above
            now());  // current time in seconds
  Serial.print(Dbuf); // notice this is only for the serial port
  sendXbee(Dbuf);     // out to the XBee
  Serial.println("Pressure message sent");
}

I use sprintf() because the documentation is all over the web and there are tons of examples out there for people to follow.  The problem is that the Arduino sprintf() doesn't support floating point, but does have a handy routine dtostrf() that will turn a float variable into a nice little string that can be used.  Pay particular attention to the format string I used; you don't have to do it this way.  You can assemble any string you want to and send it instead.  Like I said though, JSON is already documented on the web and I don't have to invent anything this way.

Now, I want to show you how I control where the message is sent.  I read the default address from the XBee DH and DL parameters during initialization and put it into the variable 'Destination', now I get to use the variable:

void sendXbee(const char* command){
  ZBTxRequest zbtx = ZBTxRequest(Destination, (uint8_t *)command, strlen(command));
  xbee.send(zbtx);
}

This will take the string I just constructed and put it into an XBee packet, set the address I mentioned and send it.  To change the address such that the message is broadcast to all devices, I decode a command and just change the value of Destination:

    // The ability to broadcast is sometimes used to see what it going on
    // using an XBee plugged into a laptop to monitor the data.
    else if (strcmp(nxtToken, "SendBroadcast") == 0){
      Serial.println(F("Switching Address to Broadcast"));
      Destination = Broadcast;     // set XBee destination address to Broadcast for now
    }
    else if (strcmp(nxtToken, "SendController") == 0){
      Serial.println(F("Switching Address to Controller"));
      Destination = Controller;     // set XBee destination address to Broadcast for now
    }

This is just like the Reset command above; I send

Barometer,SendBroadcast

and it switches the Destination variable to the broadcast address and from then on every XBee in the network will see the message.  To put it back to normal:

Barometer,SendController

And, this brings up another feature I like to have on my remote devices, a nice way to test the commands without having to drag out an XBee breakout board every time I want to add or try something.  Wouldn't it be nice to be able to simulate the receipt of a command through the IDE serial display?  I worked up this code to allow that very thing:

  // This allows emulation of an incoming XBee command from the serial console
  // It's not a command interpreter, it's limited to the same command string as
  // the XBee.  Can be used for testing
  if(Serial.available()){  // 
    char c = Serial.read();
    if(requestIdx < sizeof(requestBuffer)){
      requestBuffer[requestIdx] = c;
      requestIdx++;
    }
    else{  //the buffer filled up with garbage
      memset(requestBuffer,0,sizeof(requestBuffer));  //start off empty
      requestIdx = 0; // and at the beginning
      c = 0; // clear away the just-read character
    }
    if (c == '\r'){
      requestBuffer[requestIdx] = '\0';
//      Serial.println(requestBuffer);
      // now do something with the request string
      processXbee(requestBuffer,strlen(requestBuffer));
      //start collecting the next request string
      memset(requestBuffer,0,sizeof(requestBuffer));  //start off empty
      requestIdx = 0; // and at the beginning
    }
  }

This fragment will allow me to type a command that would normally come in over the XBee right into the IDE serial monitor and send it to the code.  It does this by just gathering the characters and calling the same routine I use to decode XBee commands.  This cuts short the time it takes to add another command to the device.

Now, normally, I would post the code of the entire module, but I bet you've had enough code samples in windows for one session.  Instead, I finally got the Arduino code I'm using for the various things into GitHub.  This entire module is in the 'Barometer' directory so you can grab it and modify away.  It's been running for a few days and hasn't blown up yet, but I'll certainly be adding to it over time.

If you go looking in the other directories don't expect all of these features to be in each thing.  Like I said, I developed these tricks over time and only recently started to go back and update the older devices.  Heck, I haven't changed the thermostats at all in months.  One of the modules hasn't even been compiled under an IDE version in a couple of years.  I'll work through the devices one at a time as I have the chance, and probably come up with another trick that I'll want to fit into the others.  It never ends.

Here's the link to GitHub  <link>. Have fun.

Tuesday, January 20, 2015

Barometric Pressure, Wow, that was easy

My barometric pressure sensor came in today.  I posted about ordering it a short while ago because the shipping costs at Adafruit (my preference) were too cotton-pickin' high, and man, I was totally surprised when it came in today.  The other thing that surprised me was how small this thing is.

Sure, the pictures compare it to a coin and that should give a person a good idea how small it is, but when you unwrap it and have to actually look around to find it inside the shipping envelope, it comes home full force.  This thing, breakout board and all, is tiny.


Notice how small it is compared to the Arduino?  That means I have some choices in exactly how I implement the final installation.  Another thing that surprised me was how incredibly easy it was to implement the code to read both temperature and pressure from the device.  Yes, I was standing on the shoulders of giants that have trod before me, but that's what open source is all about.  First, I grabbed the Adafruit Unified Sensor Driver and tried it out.  I had compile errors in the examples supplied.  Frankly, that was annoying since I've never encountered that before at Adafruit, so I grabbed their older, deprecated, version 1 library specifically for the BMP-085 and tried it.

Worked first time.

So, fine, I'll just use the older library because it was getting late and I really don't want to hunt down some big company's compile error.  A deprecated library that works is perfectly OK with me.  Of course I made a few changes to the example because it gives the pressure in Pascals and the temperature in Celcius, I wanted mBar and Fahrenheit.  I also want the string I use to be a JSON string when I send it over an XBee to my Raspberry Pi.

So, about twenty or so minutes later I had this script:

#include "Wire.h"
#include "Adafruit_BMP085.h"
 
Adafruit_BMP085 bmp;
char buff1[50];
 
void setup() {
  Serial.begin(9600);
  bmp.begin();  
}
 
void loop() {
    char t[10], p[10];
    
    float temperature = bmp.readTemperature()* 9/5 +32;
    float pressure = bmp.readPressure()/100.0;
    sprintf(buff1, "{barometer:{temperature:%s,pressure:%s}}\n", 
            dtostrf(temperature, 3, 1, t),dtostrf(pressure, 4, 1, p));
    Serial.print(buff1);
    delay(500);
}

Yes, that's all there is to implementing one of these devices.  How much easier can it possibly get? Here's what the output looks like:

{barometer:{temperature:74.5,pressure:930.4}}
{barometer:{temperature:74.5,pressure:930.4}}
{barometer:{temperature:74.5,pressure:930.4}}
{barometer:{temperature:74.5,pressure:930.3}}
{barometer:{temperature:74.5,pressure:930.4}}
{barometer:{temperature:74.5,pressure:930.3}}
{barometer:{temperature:74.5,pressure:930.3}}
{barometer:{temperature:74.5,pressure:930.4}}
{barometer:{temperature:74.5,pressure:930.3}}
{barometer:{temperature:74.5,pressure:930.3}}
{barometer:{temperature:74.5,pressure:930.3}}
{barometer:{temperature:74.5,pressure:930.3}}
{barometer:{temperature:74.5,pressure:930.4}}

I still have a bunch of work to do with this device, I have to hook it to an XBee, and install it into my Stephenson Screen out on the fence.  That will require waiting for the XBee shield I ordered for this project to come in.  I intend to put an Arduino, the shield, the XBee and this new BMP-085 all out on the fence in the weather and see what happens over time.  So expect at least one more post on this part of the project.

These days a basic Arduino clone can be bought for less than $10, an XBee shield for about the same and the BMP-085 for a few more dollars.  The accuracy on these things is quite good, and if necessary over time, calibration is a simple code change.  Nice project for a beginning weather buff that wants to go digital.

Have fun.

Thursday, January 15, 2015

I Want to Whine About Shipping Costs.

I don't know about the rest of you, but sometimes one of my projects costs more in shipping fees than components.  Getting a couple of chips and resistors across the country can make a project cost more than the fun of doing it is worth.  That's why more and more of my projects are based on cheap Chinese products shopped from one of the Chinese suppliers or eBay.

Sure, I'd much rather buy locally, if there was a locally available to buy from, but the industry has moved completely away from that model...  There are no electronic shops within any reasonable driving distance of my place.  The cost of getting something from somewhere else has trained me to look at the shipping cost very carefully, and my big mouth has just gotten a thread closed at one of my favorite supplier's forum.

The story: I want to follow up on my threat of getting a barometric sensor and setting it up to read temperature and air pressure inside a Stevenson screen I have on a fence post in the yard <link>, so I went to Adafruit and found what I wanted.  It was a:

BMP180 Barometric Pressure/Temperature/Altitude Sensor- 5V ready PID: 1603  $9.95

This is a great little breakout board (from the description) and would be a lot of fun to work up in the project.  However, as I was checking out, the shipping price came up:

UPS Ground: $13.23

There were other options, all more expensive than this.  What?  That's almost 150% of the cost of the item.  So, I went to their forum and looked to see if anyone else complained about this.  Of course someone did, so I did too.  I never want to be outdone in the complaint department.  When it became clear to the folk running the forum that I wasn't going to be shuffled aside by sycophants citing how hard and expensive it was to ship items, the main purpose of their business, they closed the thread. Maybe I should start a forum (not likely), because I could get the last word every time.  All you have to do is type something in, then close the thread so no one can respond.

No, I wasn't abusive, I think I was polite and reasonable, but in your leisure (those of you that have any left), take a look for yourselves <link>.

Yes, yes, I know all about how much it costs to staff and support a shipping department.  I understand completely the inconvenience of getting stuff to the post office and organizing everything.  But, isn't that what got this business going?  The huge problem of shipping is why the distributor model came to be. One company skilled in making a product sells it to another that is skilled in warehousing and shipping.  I took that class in college too.

And, they just announced a solution that is actually pretty good; they have joined with DigiKey.  So, let's see what that would do to solve my particular problem: Darn, the pressure sensor isn't listed as an Adafruit item.  Ok, I'll just pretend it was, I'll pick some other item that weighs roughly the same and compare it between Adafruit and DigiKey; I chose the:

ADS1015 12-Bit ADC - 4 Channel with Programmable Gain Amplifier PID: 1083  $9.95

and the shipping was the same at $13.23.  At DigiKey, the shipping for it was $3,23, exactly $10.00 less.

Yes, I know that ordering more stuff would spread the shipping cost across multiple items making it easier to justify, but I don't want to order a bunch of items just to stockpile them for later, they get lost or chewed on by the squirrels that sometimes get into my garage.

But just to fill in the blanks, suppose I bought two of them; would that make it better?  Nope, two boards cost exactly the same for shipping at both Adafruit and DigiKey: $13.23 and $3.23. Too bad I don't need two barometric pressure sensors.

I was going to go back and compliment them on the deal they made with DigiKey, but they had closed the thread already, so I couldn't.  But, not to be forestalled, I commented instead on the cool looking blog post they did on DigiKey (just today mind you, after I complained publicly), but the comment hasn't appeared yet.  It's a moderated blog, so it may take a while for the comment to make it past the censor (free speech doesn't apply to someone else's blog), or it may not show up at all.  I'll check back later to see if they let the comment through <link>, or maybe I'll forget about it.

Don't misunderstand, I absolutely love Adafruit.  They constantly surprise me at their offerings and the support they give to their customers.  Great company, and I'll continue to deal with them, but order from one of their distributors because I just don't want to throw money down the drain on shipping costs. Note that SparkFun still uses USPS shipping and a similar purchase would be comparable to DigiKey shipping.  And, of course, sometimes the additional shipping cost is worth it; times when you want it before the weekend, something broke and you need it fixed right now, or the ton of other times when waiting a week or more is inconvenient. That's not the case most of the time.

The barometric sensor is on its way from China, but that ADC listed above looks like it would work for another project I have in mind.  And, I have an active DigiKey account.