Tuesday, January 20, 2015

Barometric Pressure, Wow, that was easy

My barometric pressure sensor came in today.  I posted about ordering it a short while ago because the shipping costs at Adafruit (my preference) were too cotton-pickin' high, and man, I was totally surprised when it came in today.  The other thing that surprised me was how small this thing is.

Sure, the pictures compare it to a coin and that should give a person a good idea how small it is, but when you unwrap it and have to actually look around to find it inside the shipping envelope, it comes home full force.  This thing, breakout board and all, is tiny.


Notice how small it is compared to the Arduino?  That means I have some choices in exactly how I implement the final installation.  Another thing that surprised me was how incredibly easy it was to implement the code to read both temperature and pressure from the device.  Yes, I was standing on the shoulders of giants that have trod before me, but that's what open source is all about.  First, I grabbed the Adafruit Unified Sensor Driver and tried it out.  I had compile errors in the examples supplied.  Frankly, that was annoying since I've never encountered that before at Adafruit, so I grabbed their older, deprecated, version 1 library specifically for the BMP-085 and tried it.

Worked first time.

So, fine, I'll just use the older library because it was getting late and I really don't want to hunt down some big company's compile error.  A deprecated library that works is perfectly OK with me.  Of course I made a few changes to the example because it gives the pressure in Pascals and the temperature in Celcius, I wanted mBar and Fahrenheit.  I also want the string I use to be a JSON string when I send it over an XBee to my Raspberry Pi.

So, about twenty or so minutes later I had this script:

#include "Wire.h"
#include "Adafruit_BMP085.h"
 
Adafruit_BMP085 bmp;
char buff1[50];
 
void setup() {
  Serial.begin(9600);
  bmp.begin();  
}
 
void loop() {
    char t[10], p[10];
    
    float temperature = bmp.readTemperature()* 9/5 +32;
    float pressure = bmp.readPressure()/100.0;
    sprintf(buff1, "{barometer:{temperature:%s,pressure:%s}}\n", 
            dtostrf(temperature, 3, 1, t),dtostrf(pressure, 4, 1, p));
    Serial.print(buff1);
    delay(500);
}

Yes, that's all there is to implementing one of these devices.  How much easier can it possibly get? Here's what the output looks like:

{barometer:{temperature:74.5,pressure:930.4}}
{barometer:{temperature:74.5,pressure:930.4}}
{barometer:{temperature:74.5,pressure:930.4}}
{barometer:{temperature:74.5,pressure:930.3}}
{barometer:{temperature:74.5,pressure:930.4}}
{barometer:{temperature:74.5,pressure:930.3}}
{barometer:{temperature:74.5,pressure:930.3}}
{barometer:{temperature:74.5,pressure:930.4}}
{barometer:{temperature:74.5,pressure:930.3}}
{barometer:{temperature:74.5,pressure:930.3}}
{barometer:{temperature:74.5,pressure:930.3}}
{barometer:{temperature:74.5,pressure:930.3}}
{barometer:{temperature:74.5,pressure:930.4}}

I still have a bunch of work to do with this device, I have to hook it to an XBee, and install it into my Stephenson Screen out on the fence.  That will require waiting for the XBee shield I ordered for this project to come in.  I intend to put an Arduino, the shield, the XBee and this new BMP-085 all out on the fence in the weather and see what happens over time.  So expect at least one more post on this part of the project.

These days a basic Arduino clone can be bought for less than $10, an XBee shield for about the same and the BMP-085 for a few more dollars.  The accuracy on these things is quite good, and if necessary over time, calibration is a simple code change.  Nice project for a beginning weather buff that wants to go digital.

Have fun.

Thursday, January 15, 2015

I Want to Whine About Shipping Costs.

I don't know about the rest of you, but sometimes one of my projects costs more in shipping fees than components.  Getting a couple of chips and resistors across the country can make a project cost more than the fun of doing it is worth.  That's why more and more of my projects are based on cheap Chinese products shopped from one of the Chinese suppliers or eBay.

Sure, I'd much rather buy locally, if there was a locally available to buy from, but the industry has moved completely away from that model...  There are no electronic shops within any reasonable driving distance of my place.  The cost of getting something from somewhere else has trained me to look at the shipping cost very carefully, and my big mouth has just gotten a thread closed at one of my favorite supplier's forum.

The story: I want to follow up on my threat of getting a barometric sensor and setting it up to read temperature and air pressure inside a Stevenson screen I have on a fence post in the yard <link>, so I went to Adafruit and found what I wanted.  It was a:

BMP180 Barometric Pressure/Temperature/Altitude Sensor- 5V ready PID: 1603  $9.95

This is a great little breakout board (from the description) and would be a lot of fun to work up in the project.  However, as I was checking out, the shipping price came up:

UPS Ground: $13.23

There were other options, all more expensive than this.  What?  That's almost 150% of the cost of the item.  So, I went to their forum and looked to see if anyone else complained about this.  Of course someone did, so I did too.  I never want to be outdone in the complaint department.  When it became clear to the folk running the forum that I wasn't going to be shuffled aside by sycophants citing how hard and expensive it was to ship items, the main purpose of their business, they closed the thread. Maybe I should start a forum (not likely), because I could get the last word every time.  All you have to do is type something in, then close the thread so no one can respond.

No, I wasn't abusive, I think I was polite and reasonable, but in your leisure (those of you that have any left), take a look for yourselves <link>.

Yes, yes, I know all about how much it costs to staff and support a shipping department.  I understand completely the inconvenience of getting stuff to the post office and organizing everything.  But, isn't that what got this business going?  The huge problem of shipping is why the distributor model came to be. One company skilled in making a product sells it to another that is skilled in warehousing and shipping.  I took that class in college too.

And, they just announced a solution that is actually pretty good; they have joined with DigiKey.  So, let's see what that would do to solve my particular problem: Darn, the pressure sensor isn't listed as an Adafruit item.  Ok, I'll just pretend it was, I'll pick some other item that weighs roughly the same and compare it between Adafruit and DigiKey; I chose the:

ADS1015 12-Bit ADC - 4 Channel with Programmable Gain Amplifier PID: 1083  $9.95

and the shipping was the same at $13.23.  At DigiKey, the shipping for it was $3,23, exactly $10.00 less.

Yes, I know that ordering more stuff would spread the shipping cost across multiple items making it easier to justify, but I don't want to order a bunch of items just to stockpile them for later, they get lost or chewed on by the squirrels that sometimes get into my garage.

But just to fill in the blanks, suppose I bought two of them; would that make it better?  Nope, two boards cost exactly the same for shipping at both Adafruit and DigiKey: $13.23 and $3.23. Too bad I don't need two barometric pressure sensors.

I was going to go back and compliment them on the deal they made with DigiKey, but they had closed the thread already, so I couldn't.  But, not to be forestalled, I commented instead on the cool looking blog post they did on DigiKey (just today mind you, after I complained publicly), but the comment hasn't appeared yet.  It's a moderated blog, so it may take a while for the comment to make it past the censor (free speech doesn't apply to someone else's blog), or it may not show up at all.  I'll check back later to see if they let the comment through <link>, or maybe I'll forget about it.

Don't misunderstand, I absolutely love Adafruit.  They constantly surprise me at their offerings and the support they give to their customers.  Great company, and I'll continue to deal with them, but order from one of their distributors because I just don't want to throw money down the drain on shipping costs. Note that SparkFun still uses USPS shipping and a similar purchase would be comparable to DigiKey shipping.  And, of course, sometimes the additional shipping cost is worth it; times when you want it before the weekend, something broke and you need it fixed right now, or the ton of other times when waiting a week or more is inconvenient. That's not the case most of the time.

The barometric sensor is on its way from China, but that ADC listed above looks like it would work for another project I have in mind.  And, I have an active DigiKey account.

Saturday, January 10, 2015

Wemo Light Switches, a Completely New Approach

I have several posts on implementing the Wemo switches into my house <link, link, link> and of course a couple of other modifications to make things a little easier; lastly, where I finally found out what was causing them to just go away from time to time <link>.  Even with all this attention, the things still are an aggravation.  But not just for me; one of my readers, Glenn Tracy, has several of the switches, and one Wemo Insight plug that have been driving him nuts.  Glenn made the serious mistake of mentioning arp, so I enlisted him to try the various ideas out as I coded them (he probably regrets that now), and I started exploring possibilities.

The devices pretty much stay on line since I found a cause of the intermittent failure, but then they get hard to find because they don't respond well to Upnp discovery messages.  Since the IP address is assigned by DHCP, I can't rely on that to find them, plus the port they respond to can change any time the switch feels cranky.  So how the heck can I find them reliably ... arp seemed to work.  TCP/IP relies on the arp protocol to communicate, that should be reliable enough.  So, how to use it?

Fortunately, someone else has addressed this problem with a couple of tools, arping and arp-scan.  Arping relies on some knowledge in advance, but arp-scan doesn't.  You can issue an arp-scan on your home network and every machine on the network will be found.  Naturally, this isn't perfect, some devices guard against response to prevent hacking, and some devices don't respond on the first interaction.  But, I'm looking for Wemo switches, they respond, so now I have to figure out how to get the port number that the switch currently listens to.

Taking a chapter from my hacks on various other switches and devices, I decided to just try all the ports and see which one worked.  Since the switches I have only use ports 49152, 49153, and 49154, I can try all of them in a matter of seconds and not worry about it taking a ton of time searching.  If it turns out later that they go higher from time to time, I can simply add another port to the list.  Now, armed with this possibility, I tried out arp-scan.  Arp-scan is a normal linux tool, so all we have to do is install it:

sudo apt-get install arp-scan

And let the installation run.  Now we have to make it run as a root process because raw sockets need root permission to be effective:

sudo chmod +s /usr/bin/arp-scan

That will give the executable the 'suid' bit and since it's owned by root, it will get all the capabilities of a root process.  Nice, now run it and see what happens:

 arp-scan -q -v 192.168.0.0-192.168.0.255
Interface: eth0, datalink type: EN10MB (Ethernet)
Starting arp-scan 1.8.1 with 256 hosts (http://www.nta-monitor.com/tools/arp-scan/)
192.168.0.1     40:4a:03:de:41:87
192.168.0.2     00:1d:7e:91:12:0a
192.168.0.28    ec:1a:59:ce:82:71
192.168.0.34    4c:82:cf:83:c5:05
192.168.0.45    ec:e0:9b:b9:31:11
192.168.0.50    00:1d:7e:91:12:0a
192.168.0.29    ec:1a:59:ce:7c:8d
192.168.0.44    78:4b:87:41:5d:53
192.168.0.43    ec:1a:59:cd:99:55
192.168.0.22    2c:d0:5a:f6:dc:65
192.168.0.26    ec:1a:59:e8:fe:81
192.168.0.47    18:22:7e:d2:04:a0
192.168.0.202   de:ad:be:ef:fe:ed
192.168.0.203   de:ad:be:ef:fe:ee
192.168.0.204   de:ad:be:ef:fe:ef
---     Pass 1 complete
---     Pass 2 complete

18 packets received by filter, 0 packets dropped by kernel
Ending arp-scan 1.8.1: 256 hosts scanned in 1.258 seconds (203.50 hosts/sec). 15 responded

Notice a few things: I carefully chose the options to give me the information I needed while excluding the built-in capability of finding the manufacturer of the devices found by table lookup.  I was only interested in Belkin devices, and the table lookup depends on having up to date tables.  I really don't want to have to worry about keeping some table updated, so I'll handle that in the code for controlling the switch.  Also, arp-scan will take a range of addresses; since I limit my DHCP devices to the range 0-50, I should be able to scan all addresses in a few seconds.  That should be much, much faster than waiting for devices to respond to the (often ineffective) upnp discovery command.

The devices have been found, and a little code can isolate out the Belkin devices, so now it's time to find the port number the switch listens to.  This serves another purpose, to separate the switches from other Belkin devices that may show up over time.  It wouldn't do much good to tell a router or ethernet switch to turn off thinking it was a switch.  We've already discovered many intimate details of the switches including the location of the switch's setup xml description.  So a simple HTML request to switch-ip-address:possible-port/setup.xml should return a description of the switch.  Once we have the html in hand, just look inside it for the keywords and we know what we have.  I'm only interested in an Insite or a lightswitch; the other devices can wait for a later hack.  If nothing comes back, then I have the wrong port, just try the next one.  Here's an example of the setup.xml output that I sucked out of one of my switches:

<root xmlns="urn:Belkin:device-1-0"> <specVersion> <major>1</major> <minor>0</minor> </specVersion> <device> <deviceType>urn:Belkin:device:lightswitch:1</deviceType> <friendlyName>outsidegarage</friendlyName> <manufacturer>Belkin International Inc.</manufacturer> <manufacturerURL>http://www.belkin.com</manufacturerURL> <modelDescription>Belkin Plugin Socket 1.0</modelDescription> <modelName>LightSwitch</modelName> <modelNumber>1.0</modelNumber> <modelURL>http://www.belkin.com/plugin/</modelURL> <serialNumber>221332K130012B</serialNumber> <UDN>uuid:Lightswitch-1_0-221332K130012B</UDN> <UPC>123456789</UPC> <macAddress>EC1A59E8FE80</macAddress> <firmwareVersion>WeMo_WW_2.00.6395.PVT</firmwareVersion> <iconVersion>3|49153</iconVersion> <binaryState>0</binaryState> <iconList> <icon> <mimetype>jpg</mimetype> <width>100</width> <height>100</height> <depth>100</depth> <url>icon.jpg</url> </icon> </iconList> <serviceList> <service> <serviceType>urn:Belkin:service:WiFiSetup:1</serviceType> <serviceId>urn:Belkin:serviceId:WiFiSetup1</serviceId> <controlURL>/upnp/control/WiFiSetup1</controlURL> <eventSubURL>/upnp/event/WiFiSetup1</eventSubURL> <SCPDURL>/setupservice.xml</SCPDURL> </service> <service> <serviceType>urn:Belkin:service:timesync:1</serviceType> <serviceId>urn:Belkin:serviceId:timesync1</serviceId> <controlURL>/upnp/control/timesync1</controlURL> <eventSubURL>/upnp/event/timesync1</eventSubURL> <SCPDURL>/timesyncservice.xml</SCPDURL> </service> <service> <serviceType>urn:Belkin:service:basicevent:1</serviceType> <serviceId>urn:Belkin:serviceId:basicevent1</serviceId> <controlURL>/upnp/control/basicevent1</controlURL> <eventSubURL>/upnp/event/basicevent1</eventSubURL> <SCPDURL>/eventservice.xml</SCPDURL> </service> <service> <serviceType>urn:Belkin:service:firmwareupdate:1</serviceType> <serviceId>urn:Belkin:serviceId:firmwareupdate1</serviceId> <controlURL>/upnp/control/firmwareupdate1</controlURL> <eventSubURL>/upnp/event/firmwareupdate1</eventSubURL> <SCPDURL>/firmwareupdate.xml</SCPDURL> </service> <service> <serviceType>urn:Belkin:service:rules:1</serviceType> <serviceId>urn:Belkin:serviceId:rules1</serviceId> <controlURL>/upnp/control/rules1</controlURL> <eventSubURL>/upnp/event/rules1</eventSubURL> <SCPDURL>/rulesservice.xml</SCPDURL> </service> <service> <serviceType>urn:Belkin:service:metainfo:1</serviceType> <serviceId>urn:Belkin:serviceId:metainfo1</serviceId> <controlURL>/upnp/control/metainfo1</controlURL> <eventSubURL>/upnp/event/metainfo1</eventSubURL> <SCPDURL>/metainfoservice.xml</SCPDURL> </service> <service> <serviceType>urn:Belkin:service:remoteaccess:1</serviceType> <serviceId>urn:Belkin:serviceId:remoteaccess1</serviceId> <controlURL>/upnp/control/remoteaccess1</controlURL> <eventSubURL>/upnp/event/remoteaccess1</eventSubURL> <SCPDURL>/remoteaccess.xml</SCPDURL> </service> <service> <serviceType>urn:Belkin:service:deviceinfo:1</serviceType> <serviceId>urn:Belkin:serviceId:deviceinfo1</serviceId> <controlURL>/upnp/control/deviceinfo1</controlURL> <eventSubURL>/upnp/event/deviceinfo1</eventSubURL> <SCPDURL>/deviceinfoservice.xml</SCPDURL> </service> <service> <serviceType>urn:Belkin:service:smartsetup:1</serviceType> <serviceId>urn:Belkin:serviceId:smartsetup1</serviceId> <controlURL>/upnp/control/smartsetup1</controlURL> <eventSubURL>/upnp/event/smartsetup1</eventSubURL> <SCPDURL>/smartsetup.xml</SCPDURL> </service> <service> <serviceType>urn:Belkin:service:manufacture:1</serviceType> <serviceId>urn:Belkin:serviceId:manufacture1</serviceId> <controlURL>/upnp/control/manufacture1</controlURL> <eventSubURL>/upnp/event/manufacture1</eventSubURL> <SCPDURL>/manufacture.xml</SCPDURL> </service> </serviceList> <presentationURL>/pluginpres.html</presentationURL> </device> </root>

The tag <modelName>LightSwitch</modelName> tells me that this is a light switch, as opposed to an Insight, which would look like <modelName>Insight</modelName>.  This matters because the Insight switch returns a different value when it's turned on. The MAC address is in there, and a lot of other things that are used during operation.  For example to find the friendly name (the name assigned by the user) look at the tag <friendlyName>outsidegarage</friendlyName>.

The very fact that I got the XML back means I found the port and the tags tell me what it is, so now I have enough information, just some code to control the switches and I should be done.  Yeah, right.  When my victim er ... uh ... partner Glenn in this effort tested it, there were a number of problems that came up.  I had to put the IP address range in the .houserc file because there just wasn't any way to reliably discover the network range we were interested in.  Then, we found out that the wemo switches took to long to respond to the arp request and it had to be retried, sometimes several times to get them to talk.  When the switches decide to change addresses, there has to be a rediscovery to get the new port number.  You know, the usual list of things that will drive you nuts using a device differently than it was intended.

But, we overcame the ton of 'glitches' that almost appear to have been designed into the switch to keep people like me from doing things like this, and in the process made it much faster, much more reliable, and able to leap tall buildings with a single bound.  There are some caveats though:  The switches can still disappear.  Yes, even with a different method of discovery, and port change retries, the things will fail to respond long enough to evade detection.  To overcome this, the control process will simply exit after a few retries, and some controlling tool can automatically restart it to do the entire discovery process all over again.  This will find the switches and set them up to be used all over again.  A simple script like this:

#!/bin/bash
while : do         echo "Starting wemoccontrol"         wemocontrol.py         sleep 10 done

can restart the process.  Yes, you have to replace the commands with something suitable to your environment, and the sleep is to keep it under control in case you make a mistake, but you can use this to test the idea.  I use upstart and a config file to accomplish this for all the processes I run on the Pi, why invent something new when a great tool already exists?

So, instead of using upnp discovery, I switched to arp discovery, I find the Belkin device, and among those I try the various ports to discover Wemo light switches and Insite devices, then I bring up a CherryPy server to control them.  When they fail to respond to a request, I try three more times to make them respond and then give up and let the process die to be restarted by some other control software.  The controls are the same as my previous tries, I record the state in a Sqlite3 database and that in turn can be read by anything running on the machine.  Sounds pretty complicated, but it's much, much simpler than the other techniques I came up with.  This is very lightweight and works quite a bit faster than the others, and so far, much more reliably.  Here's the code, and it has also been placed in Github to make it easier to grab if you want <link>.

#! /usr/bin/python
# Checking Wemo Switches
#
import subprocess
import commands
from datetime import datetime, timedelta
import time
import urllib2
import BaseHTTPServer
from socket import *
import sys
import json
import re
import argparse
import sqlite3
import cherrypy
from houseutils import lprint, getHouseValues, timer, checkTimer

#--------This is for the HTML interface 
def openSite(Url):
    #lprint (Url)
    webHandle = None
    try:
        webHandle = urllib2.urlopen(Url, timeout=2) # give up in 2 seconds
    except urllib2.HTTPError, e:
        errorDesc = BaseHTTPServer.BaseHTTPRequestHandler.responses[e.code][0]
        #print "Error: (opensite) cannot retrieve URL: " + str(e.code) + ": " + errorDesc
        raise
    except urllib2.URLError, e:
        #print "Error: (openSite) cannot retrieve URL: " + e.reason[1]
        raise
    except:  #I kept getting strange errors when I was first testing it
        e = sys.exc_info()[0]
        #print ("(opensite) Odd Error: %s" % e )
        raise
    return webHandle

def talkHTML(ip, command):
    website = openSite("HTTP://" + ip + '/' + urllib2.quote(command, safe="%/:=&?~#+!$,;'@()*[]"))
    # now (maybe) read the status that came back from it
    if website is not None:
        websiteHtml = website.read()
        return  websiteHtml
        
# and this is for the SOAP interface        
# Extract the contents of a single XML tag from the data
def extractSingleTag(data,tag):
    startTag = "<%s" % tag
    endTag = "</%s>" % tag

    try:
        tmp = data.split(startTag)[1]
        index = tmp.find('>')
        if index != -1:
            index += 1
            return tmp[index:].split(endTag)[0].strip()
    except:
        pass
    return None

def sendSoap(actionName, whichOne, actionArguments):
    argList = ''
    soapEnd = re.compile('<\/.*:envelope>')
    if not actionArguments:
        actionArguments = {}
    for item in switches:
        if item["name"] == whichOne:
            thisOne = item
            break;
    switchIp = item["ip"]
    switchPort = item["port"]
    
    for arg,(val,dt) in actionArguments.iteritems():
        argList += '<%s>%s</%s>' % (arg,val,arg)

    soapRequest = 'POST /upnp/control/basicevent1 HTTP/1.1\r\n'
    # This is the SOAP request shell, I stuff values in it to handle
    # the various actions 
    # First the body since I need the length for the headers
    soapBody =  '<?xml version="1.0"?>\n'\
            '<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">\n'\
            '<SOAP-ENV:Body>\n'\
            '\t<m:%s xmlns:m="urn:Belkin:service:basicevent:1">\n'\
            '%s\n'\
            '\t</m:%s>\n'\
            '</SOAP-ENV:Body>\n'\
            '</SOAP-ENV:Envelope>' % (actionName,argList,actionName)

    #These are the headers to send with the request
    headers =   {
            'Host':'%s:%s' % (switchIp, switchPort),
            'Content-Length':len(soapBody),
            'Content-Type':'text/xml',
            'SOAPAction':'"urn:Belkin:service:basicevent:1#%s"' % (actionName)
            }
    #Generate the final payload
    for head,value in headers.iteritems():
        soapRequest += '%s: %s\r\n' % (head,value)
    soapRequest += '\r\n%s' % soapBody
    if showXml:
        print stars
        print "***REQUEST"
        print soapRequest
 
    try:
        sock = socket(AF_INET,SOCK_STREAM)
        sock.connect((switchIp,int(switchPort)))
        sock.settimeout(3);  # don't want to hang forever, ever
        sock.send(soapRequest)
        soapResponse = ""
        while True:
            data = sock.recv(1024)
            if not data:
                break
            else:
                soapResponse += data
                if soapEnd.search(soapResponse.lower()) != None:
                    break
        if showXml:
            print "***RESPONSE"
            print soapResponse
            print stars
            print ''
        sock.close()
        (header,body) = soapResponse.split('\r\n\r\n',1)
        if not header.upper().startswith('HTTP/1.') and ' 200 ' in header.split('\r\n')[0]:
            print 'SOAP request failed with error code:',header.split('\r\n')[0].split(' ',1)[1]
            errorMsg = self.extractSingleTag(body,'errorDescription')
            if errorMsg:
                print 'SOAP error message:',errorMsg
            return None
        else:
            return body
    except Exception, e:
        lprint ('Caught exception in sending:', e, switchIp, switchPort)
        sock.close()
        return None
    except KeyboardInterrupt:
        print "Keyboard Interrupt"
        sock.close()
        return None

# This will look at the result from sendSoap, and if the
# switch disappeared, it will try and get the new port number
# and update the various items.  This should allow the code 
# to continue as if the switch never decided to change its
# port number
def sendCommand(actionName, whichOne, actionArguments):
    result = sendSoap(actionName, whichOne, actionArguments)
    if result is not None:
        return result
    # it failed, now we have to do something about it
    # first, get the switch entry to check for a port change
    for item in switches:
        if item["name"] == whichOne:
            thisOne = item
            break;
    switchIp = item["ip"]
    switchPort = item["port"]
    # try to get the port number from the switch a few times
    for i in range(0,3): # Only try this three times
        lprint ("Trying to recover the switch %s"%whichOne)
        # getPort doesn't use sendSoap, so this call won't recurs
        newEntry = getPort(switchIp)
        # if the port changed, try and get the new one
        if newEntry is not None:
            # fine, it's at least alive, grab the port number,
            # print something, and and stuff it in the database
            # if it didn't change this won't break it, but if 
            # it did change, this will fix it.
            item["port"] = newEntry["port"]
            lprint ("Switch", whichOne, "changed ip from", switchPort, "to", newEntry["port"])
            dbconn = sqlite3.connect(DATABASE)
            c = dbconn.cursor()
            try:
                c.execute("update lights " 
                    "set port=? where name = ?;",
                    (newEntry["port"], whichOne))
            except sqlite3.OperationalError:
                lprint("Database is locked, record skipped")
            dbconn.commit()
            dbconn.close()
            # now try the command again
            # if it died completely it may have come back by now,
            # or if the port changed, this will try it one more time
            # it needs a limit this because this call will recurs
            result = sendSoap(actionName, whichOne, actionArguments)
            if result is not None:
                lprint("Switch recovered")
                return result
            time.sleep(1) #give the switch time to catch its breath
        else: 
            # this means the switch is not responding to HTML
            # so try the getPort again to see if it's back yet
            # There's no point in sending the soap command yet
            time.sleep(1) #give the switch time to catch its breath
            continue
    # it failed three times, just give up, die and let the system
    # restart the process.
    exit("The switch %s went away"% whichOne)
        
        
# Step through each light and see get its current state
# then record the state in the database.
def doLights():
    for switch in switches:
        thisOne = switch['name']
        updateDatabase(thisOne,get(thisOne))

def keepAlive():
    '''
    I update the database periodically with the time so I can check to see 
    if things are holding together.  I currently use the time in the light 
    switch records for this.
    '''
    lprint(" keep alive")
    for switch in switches:
        thisOne = switch['name']
        updateDatabase(thisOne, get(thisOne), force=True)

        
def get(whichone):
    ''' 
    Returns On or Off
    '''
    resp = sendCommand('GetBinaryState', whichone, {})
    if resp is not None:
        tagValue = extractSingleTag(resp, 'BinaryState').split('|')[0]
        return 'Off' if tagValue == '0' else 'On'
    return 'Off'

def on(whichone):
    """
    BinaryState is set to 'Error' in the case that it was already on.
    """
    resp = sendCommand('SetBinaryState', whichone, {'BinaryState': (1, 'Boolean')})
    if resp is not None:
        tagValue = extractSingleTag(resp, 'BinaryState').split('|')[0]
        status = 'On' if tagValue in ['1', '8', 'Error'] else 'Off'
        handleUpdate(whichone, status)
        lprint("turned %s on"%(whichone))
        return status
    return 'Off'

def off(whichone):
    """
    BinaryState is set to 'Error' in the case that it was already off.
    """
    resp = sendCommand('SetBinaryState', whichone, {'BinaryState': (0, 'Boolean')})
    if resp is not None:
        tagValue = extractSingleTag(resp, 'BinaryState').split('|')[0]
        status = 'Off' if tagValue in ['0', 'Error'] else 'On'
        handleUpdate(whichone, status)
        lprint("turned %s off"%(whichone))
        return status
    return 'Off'
    
def toggle(whichOne):
    if (get(whichOne) == 'On'):
        off(whichOne)
    else:
        on(whichOne)
        
def outsideLightsOn():
    lprint (" Outside lights on")
    on("outsidegarage")
    on("frontporch")
    on("cactusspot")
    
def outsideLightsOff():
    lprint (" Outside lights off")
    off("outsidegarage")
    off("frontporch")
    off("cactusspot")

        
def handleUpdate(whichone, status):
    for i in switches:
        if i['name'] == whichone:
            i['status'] = status
    updateDatabase(whichone, status)
    
def updateDatabase(whichone, status, force=False):
    ''' 
    This is running on a Pi and is not event driven, so polling like
    this will result in considerable wear to the SD card.  So, I'm going to 
    read the database to see if it needs to be changed before I change it.  
    According to everything I've read, reads are free, it's the writes that
    eat up the card.
    '''
    dbconn = sqlite3.connect(DATABASE)
    c = dbconn.cursor()
    c.execute("select status from lights where name = ?;",
        (whichone,))
    oldstatus = c.fetchone()
    if oldstatus[0] != status or force == True:
        lprint ("Had to update database %s, %s"%(whichone, status))
        try:
            c.execute("update lights " 
                "set status = ?, utime = ? where name = ?;",
                (status, time.strftime("%A, %B, %d at %H:%M:%S"), whichone))
            dbconn.commit()
        except sqlite3.OperationalError:
            lprint("Database is locked, record skipped")
    dbconn.close()

# If a command comes in from somewhere, this is where it's handled.
def handleCommand(command):
    lprint(str(command))
    # the command comes in from php as something like
    # ('s:17:"AcidPump, pumpOff";', 2)
    # so command[0] is 's:17:"AcidPump, pumpOff'
    # then split it at the "  and take the second item
    try:
        c = str(command[0].split('\"')[1]).split(',')
    except IndexError:
        c = str(command[0]).split(' ')    #this is for something I sent from another process
    lprint(c)
    if (c[0] == 'OutsideLightsOn'):
        outsideLightsOn()
    elif (c[0] == 'OutsideLightsOff'):
        outsideLightsOff()
    elif (c[0] == 'fPorchToggle'):
        toggle("frontporch")
    elif(c[0] == 'garageToggle'):
        toggle("outsidegarage")
    elif (c[0] == 'cactusToggle'):
        toggle("cactusspot")
    elif (c[0] == 'patioToggle'):
        toggle("patio")
    else:
        lprint("Weird command = " + str(c))

# First the process interface, it consists of a status report and
# a command receiver.
class WemoSC(object):
    @cherrypy.expose
    @cherrypy.tools.json_out() # This allows a dictionary input to go out as JSON
    def status(self):
        status = []
        for item in switches:
            status.append({item["name"]:get(item["name"])})
        return status
        
    @cherrypy.expose
    def pCommand(self, command):
        handleCommand((command,0));
        
    @cherrypy.expose
    def index(self):
        status = "<strong>Current Wemo Light Switch Status</strong><br /><br />"
        for item in switches:
            status += item["name"] +" is " + get(item["name"]) + "&nbsp;&nbsp;"
            status += '<a href="wemocommand?whichone='+item["name"]+'"><button>Toggle</button></a>'
            status += "<br />"
        return status
        
    @cherrypy.expose
    def wemocommand(self, whichone):
        # first change the light state
        toggle(whichone)
        # now reload the index page to tell the user
        raise cherrypy.InternalRedirect('/index')

# given the ip of a Belkin device this will try the ports that
# are used on the Wemo switches to see which one works.  The assumption
# is that if none of the ports work, it's not a switch, it's a modem or
# something else.
def getPort(ip):
    entry = []
    for p in ["49153", "49154", "49155"]:
        try:
            resp = talkHTML(ip + ':' + p + "/setup.xml", "")
            if debug:
                print "\tfound one at", b[0], "port", p
            if showXml:
                print stars
                print "response from switch"
                print resp
                print stars
            name = extractSingleTag(resp, 'friendlyName')
            model = extractSingleTag(resp, 'modelName')
            entry = {"mac":b[1],"ip":b[0], "port":p, "name":name, "model":model}
            return entry
        except timeout:
            continue
        except urllib2.URLError:
            continue
        except:
            e = sys.exc_info()[0]
            print ("Unexpected Error: %s" % e )
            continue
    return None
        
####################### Actually Starts Here ################################    
debug = False
showXml = False
if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("-d", "--debug",
        action = "store_true",
        help='debug flag')
    parser.add_argument("-x", "--xml",
        action = "store_true",
        help='show xml')
    parser.add_argument('count',type=int);
    args = parser.parse_args()
    if args.debug:
        print "Running with debug on"
        debug = True
    if args.xml:
        print "Running with showXML on"
        showXml = True
    targetNumber = args.count

    stars = "*********************************************"

    #-------------------------------------------------
    # the database where I'm storing stuff
    DATABASE=getHouseValues()["database"]
    lprint("Using database ", DATABASE);
    # Get the ip address and port number you want to use
    # from the houserc file
    ipAddress=getHouseValues()["wemocontrol"]["ipAddress"]
    port = getHouseValues()["wemocontrol"]["port"]
    lprint("started looking for {} switches".format(targetNumber))

    # This works on my machine, but you may have to mess with it
    # The arp-scan below tells it not to look up the manufacturer because I
    # didn't want to worry about keeping the tables that are used up to date,
    # the -l tells it to find the local net address on its own, and 
    # -v (verbose) will print that net address so I can show it for debugging
    # I take the scan range out of the .houserc file, it's an entry under wemocontrol
    # that looks like "scanRange":"192.168.0.1-192.168.0.50" adjust this as
    # needed
    try:
        scanRange = getHouseValues()["wemocontrol"]["scanRange"]
        arpCommand = "arp-scan -q -v %s 2>&1" %(scanRange)
    except KeyError:
        print "No entry in .houserc for wemocontrol scanRange"
        exit();

    while True:
        devices = [];
        # first the devices on the network
        if debug:
            print "arp-scan command is:", arpCommand
        theList = subprocess.check_output(arpCommand,shell=True);
        # split the output of the arp-scan into lines instead of a single string
        lines = theList.splitlines()
        # this looks at each line and grabs the addresses we're interested in
        # while ignoring the lines that are just information.
        for line in lines:
            allowedDigits = set("0123456789abcdef:. \t")
            if all(c in allowedDigits for c in line):
                d = line.split()
                try:
                    devices.append([d[0], d[1]])
                except IndexError: # an empty line will pass the test
                    continue
        # arp-scan can give the same addresses back more than once
        # step through the list and remove duplicates
        temp = []
        for e in devices:
            if e not in temp:
                temp.append(e)
        devices = temp
        if debug:
            print devices
        # for each device, look up the manufacturer to see if it was registered
        # to belkin
        bDevices = []
        # I got this list direct from the IEEE database and it may
        # need to be updated in a year or two.
        belkinList = ("001150", "00173F", "001CDF", "002275", "0030BD", 
                        "08863B", "94103E", "944452", "B4750E", "C05627", "EC1A59")
        for d in devices:
            if d[1].replace(':','')[0:6].upper() in belkinList:
                    bDevices.append([d[0],d[1]])
        if debug:
            print "These are the Belkin devices on the network"
            print bDevices
        if len(bDevices) < targetNumber: 
            lprint ("Only found", len(bDevices), "Belkin devices, retrying")
            time.sleep(1)
            continue
        # Got all that were asked for, continue to the next step
        
        # Now that we have a list of the Belkin devices on the network
        # We have to examine them to be sure they are actually switches
        # and not a modem or something else.  This will also assure that 
        # they will actually respond to a request.  They still may not work,
        # but at least we have a chance.
        switches = []
        for b in bDevices:
            result = getPort(b[0])
            if result is not None:
                switches.append(result)
        # Did we find enough switches ?
        if len(switches) < targetNumber: 
            lprint ("Only found", len(switches), "of them, retrying")
            devices = []
            continue
        # Yes we did, break out.
        break;
    # Now I'm going to check the database to see if it has been
    # adjusted to hold all the items (older version didn't have
    # ip, port, and mac addresses
    dbconn = sqlite3.connect(DATABASE)
    c = dbconn.cursor()
    c.execute("pragma table_info(lights);")
    dbrow = c.fetchall()
    if not any('ip' and 'mac' and 'port' in r for r in dbrow):
        lprint ("Database needs to be adjusted")
        lprint ("to hold ip, port, and MAC")
        try:
            print "adding ip if needed"
            c.execute("alter table lights add column ip text;")
        except sqlite3.OperationalError:
            print "ip was already there"
        try:
            print "adding mac if needed"
            c.execute("alter table lights add column mac text;")
        except sqlite3.OperationalError:
            print "mac was already there"
        try:
            print "adding port if needed"
            c.execute("alter table lights add column port text;")
        except sqlite3.OperationalError:
            print "port was already there"
        dbconn.commit()
    else:
        lprint ("Database already adjusted")
    for item in switches:
        try:
            c.execute("update lights " 
                "set ip = ?, mac=?, port=?where name = ?;",
                (item["ip"], item["mac"], item["port"], item["name"]))
            dbconn.commit()
        except sqlite3.OperationalError:
            lprint("Database is locked, record skipped")
    dbconn.commit()
    dbconn.close()
    lprint ("")
    lprint ("The list of", len(switches), "switches found is")
    for item in switches:
        lprint ("Friendly name:", item["name"])
        lprint ("Model:", item["model"])
        lprint ("IP address:", item["ip"])
        lprint ("Port number:", item["port"])
        lprint ("MAC:", item["mac"])
        lprint ('')
        
    # timed things.
    checkLightsTimer = timer(doLights, seconds=2)
    keepAliveTimer = timer(keepAlive, minutes=4)
    # Now configure the cherrypy server using the values
    cherrypy.config.update({'server.socket_host' : ipAddress.encode('ascii','ignore'),
                            'server.socket_port': port,
                            'engine.autoreload.on': False,
                            })
    # Subscribe to the 'main' channel in cherrypy with my timer
    cherrypy.engine.subscribe("main", checkTimer.tick)
    lprint ("Hanging on the wait for HTTP message")
    # Now just hang on the HTTP server looking for something to 
    # come in.  The cherrypy dispatcher will update the things that
    # are subscribed which will update the timers so the light
    # status gets recorded.
    cherrypy.quickstart(WemoSC())
    
    sys.exit("Told to shut down");

Notice that it is self contained.  It doesn't require Miranda.py anymore, nor any of the Ouimeaux libraries.  That's what makes it appealing to me; it's nice to have everything in one module and reasonably easy to understand.  I added a few parameters:

wemocontrol.py -h
usage: wemocontrol.py [-h] [-d] [-x] count

positional arguments:
  count

optional arguments:
  -h, --help   show this help message and exit
  -d, --debug  debug flag
  -x, --xml    show xml

I start it with wemocontrol.py 4, because I have four switches.  It will keep looking until it finds all four switches which could be a problem if one of them is disconnected, but I'll deal with that problem as it happens.  The -x parameter displays the SOAP messages as they are sent and received.  This can be used to find problems in the protocol or to understand how it works.  The -d is for debugging when you can't get things to work.  If you turn them both on, the volume of output is huge, so be prepared for it.  The -h is because I can never remember this stuff and wanted a reminder.  To run it, you must have an entry in a file called .houserc in the home directory of the user.  I use this file for all the entries for my processes.  This way I can keep things like keys, directory paths, etc in one place easily found.  The entries I have look like this:

"wemocontrol":{
"ipAddress":"192.168.0.205",
"port": 51001,
"scanRange":"192.168.0.1-192.168.0.50"},

I have a complete description of the file and how to use it here <link>.

Now, I must thank Glenn for his help in this effort.  This thing would have been a bear to test without him.  The rest of you, have fun with it.

Monday, January 5, 2015

AcuRite Weather Station, Raspberry Pi, Fun Stuff, Part 8

Part 7 of this series is here <link>

I mentioned way back in part 1 that I visited a site and discussed a linux version of the AcuRite weather station interface with a guy on Valley Information Systems discussion board <link>.  The person was the professed author of the VIS sofware that is provided by AcuRite and he has a greatly expanded and pretty cotton pickin' slick version that can be subscribed to as well.  Michael Walsh is his name and he's pretty sharp, but very, very snarky.

You've all seen this, especially from some of the dweebs on the various forums that belittle every post that is put up on the site.  I'd show you the post so you could get a feel for what I'm talking about, but he removed it.  Yep, when I went back and posted about my success with reading the USB port on linux and starting to decode the various items, I got a subscription notice that my post had been moved to the archives.  Which generally means it can still be found by a search on the forum, but it doesn't appear in the lists of posts.

Today, I got another notice and when I clicked on it:

The topic or board you are looking for appears to be either missing or off limits to you.

Heh, it looks like it's off the board entirely.  When I looked for 'linux' on the board, there's been a couple of folk looking around for a linux version recently; I may have become competition for him.

Keep in mind, I don't know why he moved the posts, or why he eventually made them disappear, it could just be weird traffic or some accident.  But it is cool that the posts of a successful, FREE, version that reads the AcuRite console's USB output on a little Raspberry Pi that doesn't tie up your home PC have disappeared.

But, once it's on the internet, it gets read, copied, backed up, mailed about, etc.  Especially since a few folk have helped me on this project by finding bugs and running it a lot in their own home.  I just typed "acurite raspberry pi" into Goodle and guess what site came up as the first entry?  hee hee.

So, you weather station folk out there, folk that got one of these for Christmas, people that succumbed to the impulse of the cool packaging at Costco (I'm in that group) spread the word.  If enough folk get involved, this could lead to a great weather station package.  I just don't have the hardware, facilities, time, or diversity in my network to do a really good job.

But wouldn't it be cool

Monday, December 29, 2014

AcuRite Weather Station, Raspberry Pi, Refining the readings Part 7

The previous part of this project is here <link>

Yes, another post on this device.  Even with this one, I'm not done; there will be others since I want to sample the RF from the sensor at some point and also bring up a separate little computer for just the weather station.  But I was extremely lucky and one of my readers double checked my work and found a couple of mistakes.

Thanks go to Heath Paddock <link>; he double checked the wind speed and direction and found mistakes, then gave me code to correct my mistakes.  Thank you Heath.  Due to a couple of factors completely within my control I misread the wind direction conversion.  He noticed it when he was testing the device and dropped me a line.  Then he double checked the wind speed and noticed a mistake there as well.

I was getting the bits correctly, but the conversion was off; let's look at the wind direction first.  The correct table for this sensor to convert from a number to direction looks like this:

// Array to translate the integer direction provided to text
char *Direction[] = {
    "NW",  // 0
    "WSW", // 1
    "WNW", // 2
    "W",   // 3
    "NNW", // 4
    "SW",  // 5
    "N",   // 6
    "SSW", // 7
    "ENE", // 8
    "SE",  // 9
    "E",   // 10
    "ESE", // 11
    "NE",  // 12
    "SSE", // 13
    "NNE", // 14
    "S"  };

Every single value is different; when I mess up, I do it in style.  When he sent me the code, I climbed up on the roof and brought down the station to check and see what is happening.  Yep, North was a six coming out of the USB port in report R1.  I went through each of the other directions and they matched his numbers exactly, then I looked at my console and it was wrong.  That just didn't make sense at all, so like all good computer users, I reset the darn thing.  It was still wrong.  I finally unplugged the console from the USB port on the Pi, pulled the batteries out of the console, and just to be complete, pulled the batteries out of the weather head.  This is a sure sign of desperation, when you do something that you know won't help just because you can't think of anything else.

But that did it.  A value of six (the pointer on the weather station was taped in place by now) corresponded to a N reading on the console.  I have no explanation for what was going on, but it works fine now.  No, I'm not going to try and recreate the situation.  I'm quite happy with the way it works and will double check the readings each time I have to change batteries in either the console or the weather head.

At any rate, the wind direction seems to be working really well now.  Next is the wind speed; Heath noticed an odd reading and asked me about it.  I had seen the reading the day before I left for Christmas, so I knew what he was talking about, but I didn't have time to look into it.  He took the time to examine the data he was receiving and found the solution.  The wind speed sensor is granular to .5 MPH; yes MPH, not KPH, and is reported back doubled.  The console does the usual 8 bit math and puts up a reading that is slightly off and loses the .5 granularity.  It also does some degree of smoothing and only reports the speed at intervals so you have to be somewhat patient to see what's going on.  Here's the code I'm currently using:

int getWindSpeed(char *data){
    int leftSide = (data[3] & 0x1f) << 3;
    int rightSide = (data[4] & 0x70) >> 4;
    float speed = (leftSide | rightSide) / 2.0;
    return(speed);  
}

Notice that I'm using a float to preserve the .5 granularity?  I did that to get the most out of my sensor and because it looks cool on my web display.  It won't match the console exactly, but that isn't really my goal.

There will probably be something else some clever person turns up that I missed or interpreted wrong, but that's part and parcel of hacking into a device with little or no information up front.  The point is that multiple people have participated in this project over many miles of separation, and we all get to use it.  I'll have the code updated in Github in an hour or so as well. <GitHub link>

Now I have to climb back up on the roof and put it in place.  There's a storm coming and I don't want to miss it.

The next part to this series is here <link>

Friday, December 19, 2014

AcuRite Weather Station, Barometric Pressure Part 6

Part 5 of this interminable series is here <link>

Way back in part 1 of this project I mentioned that I got into a little spat with a software developer about bringing up a linux version of his code <link>?  Well, I prowled around (a lot) in his forum and discovered that he really likes to talk.  That's a good thing, a very good thing, and I certainly can't say anything bad, I've got what? 200 or so posts on various technical items on this blog, and a trail a mile wide across the internet, so something about glass houses comes to mind.

At any rate, I ran across this post of his where he discusses the barometric pressure reading from the AcuRite console <link>.  The sensor is inexpensive, but it's a high sensitivity device and capable of measuring altitude to within 20cm if it is properly used.  What does altitude have to do with barometric pressure?  Everything.

I'm not going to go into the ins and outs of altitude vs temperature vs air pressure because there's entire sites devoted to this that will do a much better job of explaining it, except where it involves getting a reading worth using from the Acurite sensor in the console of my weather station.  It's the last piece of weather information I want to grab from the station; I want battery level from the weather head, but that's not weather info.  The rest of the stuff like rainfall this week, highest recorded temperature, those things can all be derived from the other data, and is a big piece of why I broke into the unit.

But, the barometric pressure still eludes me.  So, here's what I've learned so far and I'm quite open to suggestions.  The barometric sensor is: Measurement Specialities MS5607-02BA03 and here's its spec sheet PDF file <link> and the manufacturer's web page <link>.  It's a very nice device that uses a piezo crystal to sense pressure.  Since it's subject to temperature variance, it's read in the factory and calibration parameters are stored inside the chip.  The idea is that the developer resets the chip, reads the calibration parameters and then applies them to the reading from the chip and comes out with an extremely accurate measurement for altitude.  For a device in a fixed location, the altitude reading can be directly translated to atmospheric pressure.  That would give you an extremely accurate barometer.

Getting back to the forum post above, it's pretty clear that AcuRite doesn't do this.  They have some secret algorithm that is supposed to take the need for reading the calibration out by sampling readings for about a month to derive a value that will allow them to predict the weather more accurately for the console's location.  Pardon me if I'm a bit skeptical about that.  Weather forecasters have been trying to do that for centuries, and I hardly think AcuRite achieved the 'Holy Grail' of atmospheric science.  I think they couldn't figure out how to work the chip and gave up and called marketing in for a story to cover up the problem.

Who wants to buy a barometer that you have to wait a month before the readings start making sense?

We don't need accuracy in the absolute reading, we need accuracy in the changing of the reading, and a linear response that we can calibrate and forget for a while.  Remember the old physical barometers that our farmer grandparents had in the front room where you could check it on the way out the door to the barn?  Those devices worked well.  When you got them, you checked the local weather and set the reading to be the same by turning a screw on the back.  Then you set a movable needle (ours was red) to point to the same reading.  Then if the pressure went up (clear weather) you could tell because the needle moved up from the red one.  Simple, and did the job nicely.

I logged the data from the R2 message (where the barometer is supposed to show up) for quite a while, but still haven't figured it out.  In a fit of pique, I pulled the console apart to see if I could read the chip's markings:

Tearing open the AcuRite Console


Above is an over all view of the guts of the thing.  Circled on the left near the middle is the USB port and power input; above it is the board for the local temperature and humidity sensor; and way on the right is the radio receiver.  The antenna is a wire that runs up and to the left along the top.  Yes, they covered the logic chips to try and keep us from reverse engineering the hardware.


Here's the local temperature sensor (black) and local humidity sensor (white) next to an air vent. Simple setup and should work well.


The lower logic board with the reset and radio channel select switches.


The upper logic board.  The arrow points to the pressure sensor and I'm sorry it's hard to see, but you already know what it looks like from the data sheet.  This tiny little thing was hard to read and took my best magnifying glass and a good flashlight to see the markings.

All in all, it looks like a reasonable device, it makes me wonder why they took the path they chose for the barometric sensor; they seem to have enough horsepower in there to get a really good reading.

I went back to prowling through the website above, looking specifically for hints on how to derive the pressure reading from the data the device sends and came across this explanation:
In fact, the sensors used in these consoles aren't accurate; you can't expect to get an "accurate" barometric sensor for anything less than about $1,600.  That's the reason your sensor is reporting over 40 inches of mercury; we adjust for the variations between sensors using the adjustment you were asked to do.
But I don't buy that explanation.  I've seen a bunch of gauges that cost far less than $1600, with specs that are pretty impressive, and this device claims accuracy to mere centimeters.  Nope, it's simply that the data is not corrected using the burned in parameters that should be used.

So, given the various graphs provided by the manufacturer the device is relatively linear, just way off because the calibration hasn't been done, so how the heck do I find the pressure in the data?  If I can get the pressure, then rely on the temperature inside a house to be relatively stable, we should be able to derive an offset that gives us a pretty reasonable value.  Here's a small sample of the data sent in the R2 message.

02 00 00 80 00 00 00 00 00 04 00 10 00 00 00 09 60 01 01 01 01 90 07 47 E0
02 00 00 80 00 00 00 00 00 04 00 10 00 00 00 09 60 01 01 01 01 90 05 47 E0
02 00 00 80 00 00 00 00 00 04 00 10 00 00 00 09 60 01 01 01 01 90 05 47 DC
02 00 00 80 00 00 00 00 00 04 00 10 00 00 00 09 60 01 01 01 01 90 05 47 D9
02 00 00 80 00 00 00 00 00 04 00 10 00 00 00 09 60 01 01 01 01 90 03 47 D9
02 00 00 80 00 00 00 00 00 04 00 10 00 00 00 09 60 01 01 01 01 90 03 47 D6
02 00 00 80 00 00 00 00 00 04 00 10 00 00 00 09 60 01 01 01 01 90 05 47 D6
02 00 00 80 00 00 00 00 00 04 00 10 00 00 00 09 60 01 01 01 01 90 07 47 D6
02 00 00 80 00 00 00 00 00 04 00 10 00 00 00 09 60 01 01 01 01 90 09 47 D6
02 00 00 80 00 00 00 00 00 04 00 10 00 00 00 09 60 01 01 01 01 90 09 47 D1

The reading on the console for the last line was 77 F and 30.09 in/Hg.  Notice how only the last three bytes change?  Actually though the last four bytes change.  Heres the highest and lowest values sorted from the file I'm logging to:

02 00 00 80 00 00 00 00 00 04 00 10 00 00 00 09 60 01 01 01 01 8F DF 47 B0
02 00 00 80 00 00 00 00 00 04 00 10 00 00 00 09 60 01 01 01 01 90 3B 47 BB

Yes, the other values in the data do not change.  It was reported on the forum that the calibration values were being sent, but this isn't the case I my examination.  If these are calibration values, they don't resemble the items described by the manufacturer, nor do they make sense. So, we're left with the last four 8 bit values.  The data supplied by the sensor is in 24 bit values, so these last bytes could not be all of the temperature and pressure.  Since this is made in Europe, it's likely the data is in degrees C and mbar, and just looking at the four bytes as pressure, temperature, the last line of the first block could be, 0x9009 = 38873 which doesn't seem to be reasonable.  It might be somewhat reasonable for in/Hg though as in 38.8 in/Hg; a bit high, but sort of there.  I don't get much sense out of the other number 0x47d1 = 18385 either.

But notice that the second 16 bit number is roughly half the first number; it might be that the first number is a higher conversion from the A/D convertor than the second and both of them represent the same thing, just different settings on the A/D convertor.  The chip supports this, so it's possible.

And remember, I can't compare the numbers to the reading on the front of the display since AcuRite is using some secret algorithm to compute that because their way is more accurate, meaningful, whatever...

The low pressure for the last 24 hours was 952 mbar (28.11 in/Hg) and the high was 956 (28.23) and from the graph I looked at the average was 954 (28.17), so suppose the first 16 bit number was the pressure reading (uncompensated) and I just took a 24 hour average to see what came up, a change of 23 in the number would be a 1 millibar change in pressure.  If I just go with the average of 954 and consider the mean of my high and low reading 36877 and bump it up by one tenth for every 2.3  change in the number, to see what happens:

36831 to 36923 avg 36877  difference = 92
952 to 956 avg is 954  difference = 4
That would make each milibar increase equal to a change of 23 in the reading.

So, if I set 954 as being the same as 36877 and a reading of 8fe7 (right this second) would be 36839, subtract 36877 to get -38 divide by 23 to get -1.65 and the pressure would be 952.4 mbar.  Which is more reasonable than the display which says 1017 mbar, which would put me below seal level and a bunch higher than all the stations around me.  Also, remember that what I'm really interested in is the change over time as a forecast indicator.

All told, it looks to me like the barometric pressure is a total waste of time to continue trying to decode and make sense of beyond this.  It appears that AcuRite ignores the correction factors in favor of some secret technique, and we all know how I feel about 'secrets'.  I'll try this for a few days to see how it goes, but my big inclination is to get a pressure sensor and use some of the example code out there to read it and properly apply the compensation factors to get a really good reading.  I may not use the same chip, because there's a lot of breakout boards for the BMP180 and they're cheap.

Hmm, I could hook a BMP180 up to an Arduino, get both the temperature and barometric pressure from it, send it through an XBee to my Pi controller and replace the configuration I have outside in the Stephenson screen.  That would actually be a cool project.

If you notice something I overlooked, misunderstood, or messed up, let me know.

The next part to this project is here <link>.
Also, I've made progress on a barometric sensor that actually works <link>

Tuesday, December 16, 2014

Acurite Weather Station, Raspberry Pi, USB driver, I messed up, Part 5

Part 4 of this series is here <link>

Yep, I messed up, here's the story:  I was logging data so I could try and decode the barometric sensor and was getting nowhere, so I thought I'd take the console apart to see what kind of pressure sensor it had in it.  So, about an hour later I plugged it back in and it didn't work.

No, I didn't suspect I broke it, everything worked except the actual read.  The reads timed out and then started responding that I was sending the wrong command.  No code had changed, what could be wrong?  After the usual routine of changing cables, checking power, plugging it in and out a bunch of times, it still didn't work.  I moved the console to my other Pi and started taking apart the code.

About a day later, I had tried everything I could think of then it dawned on me that something might be wrong with the initialization.  I found the problem, and it was with the way I set it up.  I removed the attachment of the hidraw driver from the port, and it was providing the initialization to the kernal that allowed the weather station to work.  That was a bad thing, since without this, the kernal couldn't talk to the station.

When I took the command out of the command.txt file and prevented the hidraw driver from connecting, the communication path was fine.  But (there's always one of those), my code couldn't connect a second time.  The first time I plugged it in, everything was fine, but if the device was reset, it started failing again.  This turned out to be caused by my code disconnecting the hidraw driver and not putting it back so it could initialize the device again.

Fortunately, there's fix for that; usblib has a call that will detach the kernal driver automatically and restore it when we're done.  I added that and the station can be plugged in and work, then work again the next time.  While I was in there, I cleaned up some comments, removed the superfluous code that I left in, and generally cleaned up a bit.

It's working fine now and back to everyday use while I look further into the barometric pressure, but that's yet another post on this device.  Meanwhile, here's the updated code for the usb driver portion, and the changes will be in Github in an hour or so.

/*
    Documentation at desert-home.com
    
    This is the actual weather module I run for controlling the house, as such, this 
    may not be what you want.  You may want other-stuff/weatherstation.c

    Experimentation with a USB interface to the Acu-Rite 5 in 1 Weatherstation
    specifically for the Raspberry Pi.  The code may work on other systems, but I 
    don't have one of those.  Contrary to other folk's thinking, I don't care if 
    this ever runs on a Windows PC or an Apple anything.
    
    I specifically used a model 2032 display with the 5 in 1 sensor that I picked
    up at one of those warehouse stores.  The display has a usb plug on it and 
    I thought it might be possible to read the usb port and massage the data myself.
    
    This code represents the result of that effort.
    
    I gathered ideas from all over the web.  I use the latest (for this second)
    libusb and about a thousand examples of various things that other people have 
    done and posted about.  Frankly, I simply can't remember all of them, so please,
    don't be offended if you see your ideas somewhere in here and it isn't attributed.
    
    I simply lost track of where I found what.
    
    This module relies on libusb version 1.0.19 which, at this time, can only be
    compiled from source on the raspberry pi.
    
    Because there's likely to be a version of libusb and the associated header file
    on a Pi, use the command line below to build it since the build of libusb-1.0.19
    places things in /usr/local
    
    cc -o weatherstation  weatherstation.c -L/usr/local/lib -lusb-1.0    
    use ldd weatherstation to check which libraries are linked in.
    If you still have trouble with compilation, remember that cc has a -v
    parameter that can help you unwind what is happening.
*/

#include <stdio.h>
#include <stdlib.h>
#include <signal.h>
#include <time.h>
#include <unistd.h>
#include <sys/time.h>
#include <libusb-1.0/libusb.h>

// The vendor id and product number for the AcuRite 5 in 1 weather head.
#define VENDOR 0x24c0
#define PRODUCT 0x0003

// I store things about the weather device USB connection here.
struct {
    libusb_device *device;
    libusb_device_handle *handle;
} weatherStation;
// These are the sensors the the 5 in 1 weather head provides
struct {
    float   windSpeed;
    time_t  wsTime;
    int     windDirection;
    time_t  wdTime;
    float   temperature;
    time_t  tTime;
    int     humidity;
    time_t  hTime;
    int     rainCounter;
    time_t  rcTime;
} weatherData;

// This is just a function prototype for the compiler
void closeUpAndLeave();

// I want to catch control-C and close down gracefully
void sig_handler(int signo)
{
  if (signo == SIGINT)
    fprintf(stderr,"Shutting down ...\n");
    closeUpAndLeave();
}

/*
This tiny thing simply takes the data and prints it so we can see it
*/
// Array to translate the integer direction provided to text
char *Direction[] = {
    "NNW",
    "NW",
    "WNW",
    "W",
    "WSW",
    "SW",
    "SSW",
    "S",
    "SSE",
    "SE",
    "ESE",
    "E",
    "ENE",
    "NE",
    "NNE",
    "N"  };
// this is a bitmapped byte to tell if the various styles of reports have
// come in.  Bit 0 is R1 first type, bit 2 is R1 type 2 and bit 3 is R2
// even though I don't use R2 yet
uint8_t reportsSeen = 0;

void showit(){

    // make sure enough reports have come in before reporting
    if( reportsSeen >= 3){  // Change this when report 3 is decoded
        fprintf(stdout, "{\"windSpeed\":{\"WS\":\"%.1f\",\"t\":\"%d\"},"
                        "\"windDirection\":{\"WD\":\"%s\",\"t\":\"%d\"},"
                        "\"temperature\":{\"T\":\"%.1f\",\"t\":\"%d\"},"
                        "\"humidity\":{\"H\":\"%d\",\"t\":\"%d\"},"
                        "\"rainCounter\":{\"RC\":\"%d\",\"t\":\"%d\"}}\n",
                weatherData.windSpeed, weatherData.wsTime,
                Direction[weatherData.windDirection],weatherData.wdTime,
                weatherData.temperature, weatherData.tTime,
                weatherData.humidity, weatherData.hTime,
                weatherData.rainCounter, weatherData.rcTime);
        fflush(stdout);
    }
}
/* 
This code translates the data from the 5 in 1 sensors to something 
that can be used by a human.
*/
float getWindSpeed(char *data){
    int leftSide = (data[3] & 0x1f) << 3;
    int rightSide = data[4] & 0x70 >> 4;
    // Yes, I use mph, never got used to kilometers.
    return((float)(leftSide | rightSide) * 0.62);
}
int getWindDirection(char *data){
    return(data[4] & 0x0f);
}
float getTemp(char *data){
    // This item spans bytes, have to reconstruct it
    int leftSide = (data[4] & 0x0f) << 7;
    int rightSide = data[5] & 0x7f;
    float combined = leftSide | rightSide;
    return((combined - 400) / 10.0);
}
int getHumidity(char *data){
    int howWet = data[6] &0x7f;
    return(howWet);
}
int getRainCount(char *data){
    int count = data[6] &0x7f;
    return(count);
}
// Now that I have the data from the station, do something useful with it.

void decode(char *data, int length, int noisy){
    //int i;
    //for(i=0; i<length; i++){
    //    fprintf(stderr,"%0.2X ",data[i]);
    //}
    //fprintf(stderr,"\n"); */
    reportsSeen |= 0x04;  // just pretend I've seen report 2 already
    time_t seconds = time (NULL);
    //There are two varieties of data, both of them have wind speed
    // first variety of the data
    if ((data[2] & 0x0f) == 1){ // this has wind speed, direction and rainfall
        if(noisy)
            fprintf(stderr,"Wind Speed: %.1f ",getWindSpeed(data));
        weatherData.windSpeed = getWindSpeed(data);
        weatherData.wsTime = seconds;
        if(noisy)
            fprintf(stderr,"Wind Direction: %s ",Direction[getWindDirection(data)]);
        weatherData.wdTime = seconds;
        weatherData.windDirection = getWindDirection(data);
        if(noisy){
            fprintf(stderr,"Rain Counter: %d ",getRainCount(data));
            fprintf(stderr,"\n");
        }
        weatherData.rainCounter = getRainCount(data);
        weatherData.rcTime = seconds;
        reportsSeen |= 0x01; //I've seen report 1 now
    }
    // this is the other variety
    if ((data[2] & 0x0f) == 8){ // this has wind speed, temp and relative humidity
        if(noisy)
            fprintf(stderr,"Wind Speed: %.1f ",getWindSpeed(data));
        weatherData.windSpeed = getWindSpeed(data);
        weatherData.wsTime = seconds;
        if(noisy)
            fprintf(stderr,"Temperature: %.1f ",getTemp(data));
        weatherData.temperature = getTemp(data);
        weatherData.tTime = seconds;
        if(noisy){
            fprintf(stderr,"Humidity: %d ", getHumidity(data));
            fprintf(stderr,"\n");
        }
        weatherData.humidity = getHumidity(data);
        weatherData.hTime = seconds;
        reportsSeen |= 0x02;  // I've seen report 2 now

    }
}
/*
This code is related to dealing with the USB device
*/
// This searches the USB bus tree to find the device
int findDevice(libusb_device **devs)
{
    libusb_device *dev;
    int err = 0, i = 0, j = 0;
    uint8_t path[8]; 
    
    while ((dev = devs[i++]) != NULL) {
        struct libusb_device_descriptor desc;
        int r = libusb_get_device_descriptor(dev, &desc);
        if (r < 0) {
            fprintf(stderr,"Couldn't get device descriptor, %s\n", libusb_strerror(err));
            return(1);
        }

        fprintf(stderr,"%04x:%04x (bus %d, device %d)",
            desc.idVendor, desc.idProduct,
            libusb_get_bus_number(dev), libusb_get_device_address(dev));
        fprintf(stderr,"\n");
        if (desc.idVendor == VENDOR && desc.idProduct == PRODUCT){
            fprintf(stderr,"Found the weather station\n");
            weatherStation.device = dev;
            return (1);
        }
    }
    return(0);
}

// to handle testing and try to be clean about closing the USB device,
// I'll catch the signal and close off.
void closeUpAndLeave(){
    //OK, done with it, close off and let it go.
    fprintf(stderr,"Done with device, release and close it\n");
    int err = libusb_release_interface(weatherStation.handle, 0); //release the claimed interface
    if(err) {
        fprintf(stderr,"Couldn't release interface, %s\n", libusb_strerror(err));
    }
    fprintf(stderr, " Released interface and restored kernal driver\n");
    libusb_close(weatherStation.handle);
    fprintf(stderr, " Closed Weatherstation device\n");
    libusb_exit(NULL);
    fprintf(stderr, " Closed USB interface\n");
    exit(0);
}

// This is where I read the USB device to get the latest data.
unsigned char data[50]; // where we want the data to go

int getit(int whichOne, int noisy){
    int actual; // how many bytes were actually read
    
    // The second parameter is bmRequestType and is a bitfield
    // See http://www.beyondlogic.org/usbnutshell/usb6.shtml
    // for the definitions of the various bits.  With libusb, the 
    // #defines for these are at:
    // http://libusb.sourceforge.net/api-1.0/group__misc.html#gga0b0933ae70744726cde11254c39fac91a20eca62c34d2d25be7e1776510184209

    actual = libusb_control_transfer(weatherStation.handle, 
        LIBUSB_REQUEST_TYPE_CLASS | LIBUSB_RECIPIENT_INTERFACE | LIBUSB_ENDPOINT_IN,
        //These bytes were stolen with a USB sniffer
        0x01,0x0100+whichOne,0,
        data, sizeof(data), 10000);
    if (actual < 0){
        fprintf(stderr,"Read didn't work for report %d, %s\n", whichOne, libusb_strerror(actual));
    }
    else {
        // If you want both of the reports that the station provides,
        // just allow for it.  Right this second, I've found every thing
        // I need in report 1.  When I look further at report 2, this will
        // change
        //fprintf(stderr,"R%d:%d:", whichOne, actual);
        //int i;
        //for(i=0; i<actual; i++){
        //    fprintf(stderr,"%0.2X ",data[i]);
        //}
        //fprintf(stderr,"\n");
        if (whichOne == 1)
            // The actual data starts after the first byte
            // The first byte is the report number returned by 
            // the usb read.
            decode(&data[1], actual-1, noisy);
    }
}
// I do several things here that aren't strictly necessary.  As I learned about
// libusb, I tried things and also used various techniques to learn about the 
// weatherstation's implementation.  I left a lot of it in here in case I needed to
// use it later.  Someone may find it useful to hack into some other device.
int main(int argc, char **argv)
{
    char *usage = {"usage: %s -u -n\n"};
    int libusbDebug = 0; //This will turn on the DEBUG for libusb
    int noisy = 0;       //This will print the packets as they come in
    libusb_device **devs;
    int r, err, c;
    ssize_t cnt;
    
    while ((c = getopt (argc, argv, "unh")) != -1)
        switch (c){
            case 'u':
                libusbDebug = 1;
                break;
            case 'n':
                noisy = 1;
                break;
            case 'h':
                fprintf(stderr, usage, argv[0]);
            case '?':
                exit(1);
            default:
                exit(1);
       }
    fprintf(stderr,"%s Starting ... ",argv[0]);
    fprintf(stderr,"libusbDebug = %d, noisy = %d\n", libusbDebug, noisy);
    // The Pi linker can give you fits.  If you get a linker error on
    // the next line, set LD_LIBRARY_PATH to /usr/local/lib 
    fprintf(stderr,"This is not an error!! Checking linker, %s\n", libusb_strerror(0));

    if (signal(SIGINT, sig_handler) == SIG_ERR)
        fprintf(stderr,"Couldn't set up signal handler\n"); 
    err = libusb_init(NULL);
    if (err < 0){
        fprintf(stderr,"Couldn't init usblib, %s\n", libusb_strerror(err));
        exit(1);
    }
    // This is where you can get debug output from libusb.
    // just set it to LIBUSB_LOG_LEVEL_DEBUG
    if (libusbDebug)
        libusb_set_debug(NULL, LIBUSB_LOG_LEVEL_DEBUG);
    else
        libusb_set_debug(NULL, LIBUSB_LOG_LEVEL_INFO);

    
    cnt = libusb_get_device_list(NULL, &devs);
    if (cnt < 0){
        fprintf(stderr,"Couldn't get device list, %s\n", libusb_strerror(err));
        exit(1);
    }
    // go get the device; the device handle is saved in weatherStation struct.
    if (!findDevice(devs)){
        fprintf(stderr,"Couldn't find the device\n");
        exit(1);
    }
    // Now I've found the weather station and can start to try stuff
    // So, I'll get the device descriptor
    struct libusb_device_descriptor deviceDesc;
    err = libusb_get_device_descriptor(weatherStation.device, &deviceDesc);
    if (err){
        fprintf(stderr,"Couldn't get device descriptor, %s\n", libusb_strerror(err));
        exit(1);
    }
    fprintf(stderr,"got the device descriptor back\n");
    
    // Open the device and save the handle in the weatherStation struct
    err = libusb_open(weatherStation.device, &weatherStation.handle);
    if (err){
        fprintf(stderr,"Open failed, %s\n", libusb_strerror(err));
        closeUpAndLeave();    }
    fprintf(stderr,"I was able to open it\n");
    // Now that it's opened, I can free the list of all devices
    libusb_free_device_list(devs, 1); // Documentation says to get rid of the list
                                      // Once I have the device I need
    fprintf(stderr,"Released the device list\n");
    err = libusb_set_auto_detach_kernel_driver(weatherStation.handle, 1);
    if(err){
        fprintf(stderr,"Some problem with detach %s\n", libusb_strerror(err));
        closeUpAndLeave();
    }
    int activeConfig;
    err =libusb_get_configuration(weatherStation.handle, &activeConfig);
    if (err){
        fprintf(stderr,"Can't get current active configuration, %s\n", libusb_strerror(err));;
        closeUpAndLeave();
    }
    fprintf(stderr,"Currently active configuration is %d\n", activeConfig);
    if(activeConfig != 1){
        err = libusb_set_configuration  (weatherStation.handle, 1);
        if (err){
            fprintf(stderr,"Cannot set configuration, %s\n", libusb_strerror(err));;
        closeUpAndLeave();
        }
        fprintf(stderr,"Just did the set configuration\n");
    }
    
    err = libusb_claim_interface(weatherStation.handle, 0); //claim interface 0 (the first) of device (mine had just 1)
    if(err) {
        fprintf(stderr,"Cannot claim interface, %s\n", libusb_strerror(err));
        closeUpAndLeave();
    }
    fprintf(stderr,"Claimed Interface\n");
    // I don't want to just hang up and read the reports as fast as I can, so
    // I'll space them out a bit.  It's weather, and it doesn't change very fast.
    sleep(1);
    fprintf(stderr,"Setting the device 'idle' before starting reads\n");
    err = libusb_control_transfer(weatherStation.handle, 
        0x21, 0x0a, 0, 0, 0, 0, 1000);
    sleep(1);
    int tickcounter= 0;
    fprintf(stderr,"Starting reads\n");
    while(1){
        sleep(1);
        if(tickcounter++ % 10 == 0){
            getit(1, noisy);
        }
        if(tickcounter % 30 == 0){
            getit(2, noisy);
        }
        if (tickcounter % 15 == 0){
            showit();
        }
    }
}

There's more details on what not to do in the post where I described this in detail <link>, yep, I fessed up right in the post where the instructions are.

For the folk that have already grabbed this, sorry, I'll try a bit harder next time.

Part 6 of this series is here <link>