Showing posts with label Wemo. Show all posts
Showing posts with label Wemo. Show all posts

Saturday, January 10, 2015

Wemo Light Switches, a Completely New Approach

I have several posts on implementing the Wemo switches into my house <link, link, link> and of course a couple of other modifications to make things a little easier; lastly, where I finally found out what was causing them to just go away from time to time <link>.  Even with all this attention, the things still are an aggravation.  But not just for me; one of my readers, Glenn Tracy, has several of the switches, and one Wemo Insight plug that have been driving him nuts.  Glenn made the serious mistake of mentioning arp, so I enlisted him to try the various ideas out as I coded them (he probably regrets that now), and I started exploring possibilities.

The devices pretty much stay on line since I found a cause of the intermittent failure, but then they get hard to find because they don't respond well to Upnp discovery messages.  Since the IP address is assigned by DHCP, I can't rely on that to find them, plus the port they respond to can change any time the switch feels cranky.  So how the heck can I find them reliably ... arp seemed to work.  TCP/IP relies on the arp protocol to communicate, that should be reliable enough.  So, how to use it?

Fortunately, someone else has addressed this problem with a couple of tools, arping and arp-scan.  Arping relies on some knowledge in advance, but arp-scan doesn't.  You can issue an arp-scan on your home network and every machine on the network will be found.  Naturally, this isn't perfect, some devices guard against response to prevent hacking, and some devices don't respond on the first interaction.  But, I'm looking for Wemo switches, they respond, so now I have to figure out how to get the port number that the switch currently listens to.

Taking a chapter from my hacks on various other switches and devices, I decided to just try all the ports and see which one worked.  Since the switches I have only use ports 49152, 49153, and 49154, I can try all of them in a matter of seconds and not worry about it taking a ton of time searching.  If it turns out later that they go higher from time to time, I can simply add another port to the list.  Now, armed with this possibility, I tried out arp-scan.  Arp-scan is a normal linux tool, so all we have to do is install it:

sudo apt-get install arp-scan

And let the installation run.  Now we have to make it run as a root process because raw sockets need root permission to be effective:

sudo chmod +s /usr/bin/arp-scan

That will give the executable the 'suid' bit and since it's owned by root, it will get all the capabilities of a root process.  Nice, now run it and see what happens:

 arp-scan -q -v 192.168.0.0-192.168.0.255
Interface: eth0, datalink type: EN10MB (Ethernet)
Starting arp-scan 1.8.1 with 256 hosts (http://www.nta-monitor.com/tools/arp-scan/)
192.168.0.1     40:4a:03:de:41:87
192.168.0.2     00:1d:7e:91:12:0a
192.168.0.28    ec:1a:59:ce:82:71
192.168.0.34    4c:82:cf:83:c5:05
192.168.0.45    ec:e0:9b:b9:31:11
192.168.0.50    00:1d:7e:91:12:0a
192.168.0.29    ec:1a:59:ce:7c:8d
192.168.0.44    78:4b:87:41:5d:53
192.168.0.43    ec:1a:59:cd:99:55
192.168.0.22    2c:d0:5a:f6:dc:65
192.168.0.26    ec:1a:59:e8:fe:81
192.168.0.47    18:22:7e:d2:04:a0
192.168.0.202   de:ad:be:ef:fe:ed
192.168.0.203   de:ad:be:ef:fe:ee
192.168.0.204   de:ad:be:ef:fe:ef
---     Pass 1 complete
---     Pass 2 complete

18 packets received by filter, 0 packets dropped by kernel
Ending arp-scan 1.8.1: 256 hosts scanned in 1.258 seconds (203.50 hosts/sec). 15 responded

Notice a few things: I carefully chose the options to give me the information I needed while excluding the built-in capability of finding the manufacturer of the devices found by table lookup.  I was only interested in Belkin devices, and the table lookup depends on having up to date tables.  I really don't want to have to worry about keeping some table updated, so I'll handle that in the code for controlling the switch.  Also, arp-scan will take a range of addresses; since I limit my DHCP devices to the range 0-50, I should be able to scan all addresses in a few seconds.  That should be much, much faster than waiting for devices to respond to the (often ineffective) upnp discovery command.

The devices have been found, and a little code can isolate out the Belkin devices, so now it's time to find the port number the switch listens to.  This serves another purpose, to separate the switches from other Belkin devices that may show up over time.  It wouldn't do much good to tell a router or ethernet switch to turn off thinking it was a switch.  We've already discovered many intimate details of the switches including the location of the switch's setup xml description.  So a simple HTML request to switch-ip-address:possible-port/setup.xml should return a description of the switch.  Once we have the html in hand, just look inside it for the keywords and we know what we have.  I'm only interested in an Insite or a lightswitch; the other devices can wait for a later hack.  If nothing comes back, then I have the wrong port, just try the next one.  Here's an example of the setup.xml output that I sucked out of one of my switches:

<root xmlns="urn:Belkin:device-1-0"> <specVersion> <major>1</major> <minor>0</minor> </specVersion> <device> <deviceType>urn:Belkin:device:lightswitch:1</deviceType> <friendlyName>outsidegarage</friendlyName> <manufacturer>Belkin International Inc.</manufacturer> <manufacturerURL>http://www.belkin.com</manufacturerURL> <modelDescription>Belkin Plugin Socket 1.0</modelDescription> <modelName>LightSwitch</modelName> <modelNumber>1.0</modelNumber> <modelURL>http://www.belkin.com/plugin/</modelURL> <serialNumber>221332K130012B</serialNumber> <UDN>uuid:Lightswitch-1_0-221332K130012B</UDN> <UPC>123456789</UPC> <macAddress>EC1A59E8FE80</macAddress> <firmwareVersion>WeMo_WW_2.00.6395.PVT</firmwareVersion> <iconVersion>3|49153</iconVersion> <binaryState>0</binaryState> <iconList> <icon> <mimetype>jpg</mimetype> <width>100</width> <height>100</height> <depth>100</depth> <url>icon.jpg</url> </icon> </iconList> <serviceList> <service> <serviceType>urn:Belkin:service:WiFiSetup:1</serviceType> <serviceId>urn:Belkin:serviceId:WiFiSetup1</serviceId> <controlURL>/upnp/control/WiFiSetup1</controlURL> <eventSubURL>/upnp/event/WiFiSetup1</eventSubURL> <SCPDURL>/setupservice.xml</SCPDURL> </service> <service> <serviceType>urn:Belkin:service:timesync:1</serviceType> <serviceId>urn:Belkin:serviceId:timesync1</serviceId> <controlURL>/upnp/control/timesync1</controlURL> <eventSubURL>/upnp/event/timesync1</eventSubURL> <SCPDURL>/timesyncservice.xml</SCPDURL> </service> <service> <serviceType>urn:Belkin:service:basicevent:1</serviceType> <serviceId>urn:Belkin:serviceId:basicevent1</serviceId> <controlURL>/upnp/control/basicevent1</controlURL> <eventSubURL>/upnp/event/basicevent1</eventSubURL> <SCPDURL>/eventservice.xml</SCPDURL> </service> <service> <serviceType>urn:Belkin:service:firmwareupdate:1</serviceType> <serviceId>urn:Belkin:serviceId:firmwareupdate1</serviceId> <controlURL>/upnp/control/firmwareupdate1</controlURL> <eventSubURL>/upnp/event/firmwareupdate1</eventSubURL> <SCPDURL>/firmwareupdate.xml</SCPDURL> </service> <service> <serviceType>urn:Belkin:service:rules:1</serviceType> <serviceId>urn:Belkin:serviceId:rules1</serviceId> <controlURL>/upnp/control/rules1</controlURL> <eventSubURL>/upnp/event/rules1</eventSubURL> <SCPDURL>/rulesservice.xml</SCPDURL> </service> <service> <serviceType>urn:Belkin:service:metainfo:1</serviceType> <serviceId>urn:Belkin:serviceId:metainfo1</serviceId> <controlURL>/upnp/control/metainfo1</controlURL> <eventSubURL>/upnp/event/metainfo1</eventSubURL> <SCPDURL>/metainfoservice.xml</SCPDURL> </service> <service> <serviceType>urn:Belkin:service:remoteaccess:1</serviceType> <serviceId>urn:Belkin:serviceId:remoteaccess1</serviceId> <controlURL>/upnp/control/remoteaccess1</controlURL> <eventSubURL>/upnp/event/remoteaccess1</eventSubURL> <SCPDURL>/remoteaccess.xml</SCPDURL> </service> <service> <serviceType>urn:Belkin:service:deviceinfo:1</serviceType> <serviceId>urn:Belkin:serviceId:deviceinfo1</serviceId> <controlURL>/upnp/control/deviceinfo1</controlURL> <eventSubURL>/upnp/event/deviceinfo1</eventSubURL> <SCPDURL>/deviceinfoservice.xml</SCPDURL> </service> <service> <serviceType>urn:Belkin:service:smartsetup:1</serviceType> <serviceId>urn:Belkin:serviceId:smartsetup1</serviceId> <controlURL>/upnp/control/smartsetup1</controlURL> <eventSubURL>/upnp/event/smartsetup1</eventSubURL> <SCPDURL>/smartsetup.xml</SCPDURL> </service> <service> <serviceType>urn:Belkin:service:manufacture:1</serviceType> <serviceId>urn:Belkin:serviceId:manufacture1</serviceId> <controlURL>/upnp/control/manufacture1</controlURL> <eventSubURL>/upnp/event/manufacture1</eventSubURL> <SCPDURL>/manufacture.xml</SCPDURL> </service> </serviceList> <presentationURL>/pluginpres.html</presentationURL> </device> </root>

The tag <modelName>LightSwitch</modelName> tells me that this is a light switch, as opposed to an Insight, which would look like <modelName>Insight</modelName>.  This matters because the Insight switch returns a different value when it's turned on. The MAC address is in there, and a lot of other things that are used during operation.  For example to find the friendly name (the name assigned by the user) look at the tag <friendlyName>outsidegarage</friendlyName>.

The very fact that I got the XML back means I found the port and the tags tell me what it is, so now I have enough information, just some code to control the switches and I should be done.  Yeah, right.  When my victim er ... uh ... partner Glenn in this effort tested it, there were a number of problems that came up.  I had to put the IP address range in the .houserc file because there just wasn't any way to reliably discover the network range we were interested in.  Then, we found out that the wemo switches took to long to respond to the arp request and it had to be retried, sometimes several times to get them to talk.  When the switches decide to change addresses, there has to be a rediscovery to get the new port number.  You know, the usual list of things that will drive you nuts using a device differently than it was intended.

But, we overcame the ton of 'glitches' that almost appear to have been designed into the switch to keep people like me from doing things like this, and in the process made it much faster, much more reliable, and able to leap tall buildings with a single bound.  There are some caveats though:  The switches can still disappear.  Yes, even with a different method of discovery, and port change retries, the things will fail to respond long enough to evade detection.  To overcome this, the control process will simply exit after a few retries, and some controlling tool can automatically restart it to do the entire discovery process all over again.  This will find the switches and set them up to be used all over again.  A simple script like this:

#!/bin/bash
while : do         echo "Starting wemoccontrol"         wemocontrol.py         sleep 10 done

can restart the process.  Yes, you have to replace the commands with something suitable to your environment, and the sleep is to keep it under control in case you make a mistake, but you can use this to test the idea.  I use upstart and a config file to accomplish this for all the processes I run on the Pi, why invent something new when a great tool already exists?

So, instead of using upnp discovery, I switched to arp discovery, I find the Belkin device, and among those I try the various ports to discover Wemo light switches and Insite devices, then I bring up a CherryPy server to control them.  When they fail to respond to a request, I try three more times to make them respond and then give up and let the process die to be restarted by some other control software.  The controls are the same as my previous tries, I record the state in a Sqlite3 database and that in turn can be read by anything running on the machine.  Sounds pretty complicated, but it's much, much simpler than the other techniques I came up with.  This is very lightweight and works quite a bit faster than the others, and so far, much more reliably.  Here's the code, and it has also been placed in Github to make it easier to grab if you want <link>.

#! /usr/bin/python
# Checking Wemo Switches
#
import subprocess
import commands
from datetime import datetime, timedelta
import time
import urllib2
import BaseHTTPServer
from socket import *
import sys
import json
import re
import argparse
import sqlite3
import cherrypy
from houseutils import lprint, getHouseValues, timer, checkTimer

#--------This is for the HTML interface 
def openSite(Url):
    #lprint (Url)
    webHandle = None
    try:
        webHandle = urllib2.urlopen(Url, timeout=2) # give up in 2 seconds
    except urllib2.HTTPError, e:
        errorDesc = BaseHTTPServer.BaseHTTPRequestHandler.responses[e.code][0]
        #print "Error: (opensite) cannot retrieve URL: " + str(e.code) + ": " + errorDesc
        raise
    except urllib2.URLError, e:
        #print "Error: (openSite) cannot retrieve URL: " + e.reason[1]
        raise
    except:  #I kept getting strange errors when I was first testing it
        e = sys.exc_info()[0]
        #print ("(opensite) Odd Error: %s" % e )
        raise
    return webHandle

def talkHTML(ip, command):
    website = openSite("HTTP://" + ip + '/' + urllib2.quote(command, safe="%/:=&?~#+!$,;'@()*[]"))
    # now (maybe) read the status that came back from it
    if website is not None:
        websiteHtml = website.read()
        return  websiteHtml
        
# and this is for the SOAP interface        
# Extract the contents of a single XML tag from the data
def extractSingleTag(data,tag):
    startTag = "<%s" % tag
    endTag = "</%s>" % tag

    try:
        tmp = data.split(startTag)[1]
        index = tmp.find('>')
        if index != -1:
            index += 1
            return tmp[index:].split(endTag)[0].strip()
    except:
        pass
    return None

def sendSoap(actionName, whichOne, actionArguments):
    argList = ''
    soapEnd = re.compile('<\/.*:envelope>')
    if not actionArguments:
        actionArguments = {}
    for item in switches:
        if item["name"] == whichOne:
            thisOne = item
            break;
    switchIp = item["ip"]
    switchPort = item["port"]
    
    for arg,(val,dt) in actionArguments.iteritems():
        argList += '<%s>%s</%s>' % (arg,val,arg)

    soapRequest = 'POST /upnp/control/basicevent1 HTTP/1.1\r\n'
    # This is the SOAP request shell, I stuff values in it to handle
    # the various actions 
    # First the body since I need the length for the headers
    soapBody =  '<?xml version="1.0"?>\n'\
            '<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">\n'\
            '<SOAP-ENV:Body>\n'\
            '\t<m:%s xmlns:m="urn:Belkin:service:basicevent:1">\n'\
            '%s\n'\
            '\t</m:%s>\n'\
            '</SOAP-ENV:Body>\n'\
            '</SOAP-ENV:Envelope>' % (actionName,argList,actionName)

    #These are the headers to send with the request
    headers =   {
            'Host':'%s:%s' % (switchIp, switchPort),
            'Content-Length':len(soapBody),
            'Content-Type':'text/xml',
            'SOAPAction':'"urn:Belkin:service:basicevent:1#%s"' % (actionName)
            }
    #Generate the final payload
    for head,value in headers.iteritems():
        soapRequest += '%s: %s\r\n' % (head,value)
    soapRequest += '\r\n%s' % soapBody
    if showXml:
        print stars
        print "***REQUEST"
        print soapRequest
 
    try:
        sock = socket(AF_INET,SOCK_STREAM)
        sock.connect((switchIp,int(switchPort)))
        sock.settimeout(3);  # don't want to hang forever, ever
        sock.send(soapRequest)
        soapResponse = ""
        while True:
            data = sock.recv(1024)
            if not data:
                break
            else:
                soapResponse += data
                if soapEnd.search(soapResponse.lower()) != None:
                    break
        if showXml:
            print "***RESPONSE"
            print soapResponse
            print stars
            print ''
        sock.close()
        (header,body) = soapResponse.split('\r\n\r\n',1)
        if not header.upper().startswith('HTTP/1.') and ' 200 ' in header.split('\r\n')[0]:
            print 'SOAP request failed with error code:',header.split('\r\n')[0].split(' ',1)[1]
            errorMsg = self.extractSingleTag(body,'errorDescription')
            if errorMsg:
                print 'SOAP error message:',errorMsg
            return None
        else:
            return body
    except Exception, e:
        lprint ('Caught exception in sending:', e, switchIp, switchPort)
        sock.close()
        return None
    except KeyboardInterrupt:
        print "Keyboard Interrupt"
        sock.close()
        return None

# This will look at the result from sendSoap, and if the
# switch disappeared, it will try and get the new port number
# and update the various items.  This should allow the code 
# to continue as if the switch never decided to change its
# port number
def sendCommand(actionName, whichOne, actionArguments):
    result = sendSoap(actionName, whichOne, actionArguments)
    if result is not None:
        return result
    # it failed, now we have to do something about it
    # first, get the switch entry to check for a port change
    for item in switches:
        if item["name"] == whichOne:
            thisOne = item
            break;
    switchIp = item["ip"]
    switchPort = item["port"]
    # try to get the port number from the switch a few times
    for i in range(0,3): # Only try this three times
        lprint ("Trying to recover the switch %s"%whichOne)
        # getPort doesn't use sendSoap, so this call won't recurs
        newEntry = getPort(switchIp)
        # if the port changed, try and get the new one
        if newEntry is not None:
            # fine, it's at least alive, grab the port number,
            # print something, and and stuff it in the database
            # if it didn't change this won't break it, but if 
            # it did change, this will fix it.
            item["port"] = newEntry["port"]
            lprint ("Switch", whichOne, "changed ip from", switchPort, "to", newEntry["port"])
            dbconn = sqlite3.connect(DATABASE)
            c = dbconn.cursor()
            try:
                c.execute("update lights " 
                    "set port=? where name = ?;",
                    (newEntry["port"], whichOne))
            except sqlite3.OperationalError:
                lprint("Database is locked, record skipped")
            dbconn.commit()
            dbconn.close()
            # now try the command again
            # if it died completely it may have come back by now,
            # or if the port changed, this will try it one more time
            # it needs a limit this because this call will recurs
            result = sendSoap(actionName, whichOne, actionArguments)
            if result is not None:
                lprint("Switch recovered")
                return result
            time.sleep(1) #give the switch time to catch its breath
        else: 
            # this means the switch is not responding to HTML
            # so try the getPort again to see if it's back yet
            # There's no point in sending the soap command yet
            time.sleep(1) #give the switch time to catch its breath
            continue
    # it failed three times, just give up, die and let the system
    # restart the process.
    exit("The switch %s went away"% whichOne)
        
        
# Step through each light and see get its current state
# then record the state in the database.
def doLights():
    for switch in switches:
        thisOne = switch['name']
        updateDatabase(thisOne,get(thisOne))

def keepAlive():
    '''
    I update the database periodically with the time so I can check to see 
    if things are holding together.  I currently use the time in the light 
    switch records for this.
    '''
    lprint(" keep alive")
    for switch in switches:
        thisOne = switch['name']
        updateDatabase(thisOne, get(thisOne), force=True)

        
def get(whichone):
    ''' 
    Returns On or Off
    '''
    resp = sendCommand('GetBinaryState', whichone, {})
    if resp is not None:
        tagValue = extractSingleTag(resp, 'BinaryState').split('|')[0]
        return 'Off' if tagValue == '0' else 'On'
    return 'Off'

def on(whichone):
    """
    BinaryState is set to 'Error' in the case that it was already on.
    """
    resp = sendCommand('SetBinaryState', whichone, {'BinaryState': (1, 'Boolean')})
    if resp is not None:
        tagValue = extractSingleTag(resp, 'BinaryState').split('|')[0]
        status = 'On' if tagValue in ['1', '8', 'Error'] else 'Off'
        handleUpdate(whichone, status)
        lprint("turned %s on"%(whichone))
        return status
    return 'Off'

def off(whichone):
    """
    BinaryState is set to 'Error' in the case that it was already off.
    """
    resp = sendCommand('SetBinaryState', whichone, {'BinaryState': (0, 'Boolean')})
    if resp is not None:
        tagValue = extractSingleTag(resp, 'BinaryState').split('|')[0]
        status = 'Off' if tagValue in ['0', 'Error'] else 'On'
        handleUpdate(whichone, status)
        lprint("turned %s off"%(whichone))
        return status
    return 'Off'
    
def toggle(whichOne):
    if (get(whichOne) == 'On'):
        off(whichOne)
    else:
        on(whichOne)
        
def outsideLightsOn():
    lprint (" Outside lights on")
    on("outsidegarage")
    on("frontporch")
    on("cactusspot")
    
def outsideLightsOff():
    lprint (" Outside lights off")
    off("outsidegarage")
    off("frontporch")
    off("cactusspot")

        
def handleUpdate(whichone, status):
    for i in switches:
        if i['name'] == whichone:
            i['status'] = status
    updateDatabase(whichone, status)
    
def updateDatabase(whichone, status, force=False):
    ''' 
    This is running on a Pi and is not event driven, so polling like
    this will result in considerable wear to the SD card.  So, I'm going to 
    read the database to see if it needs to be changed before I change it.  
    According to everything I've read, reads are free, it's the writes that
    eat up the card.
    '''
    dbconn = sqlite3.connect(DATABASE)
    c = dbconn.cursor()
    c.execute("select status from lights where name = ?;",
        (whichone,))
    oldstatus = c.fetchone()
    if oldstatus[0] != status or force == True:
        lprint ("Had to update database %s, %s"%(whichone, status))
        try:
            c.execute("update lights " 
                "set status = ?, utime = ? where name = ?;",
                (status, time.strftime("%A, %B, %d at %H:%M:%S"), whichone))
            dbconn.commit()
        except sqlite3.OperationalError:
            lprint("Database is locked, record skipped")
    dbconn.close()

# If a command comes in from somewhere, this is where it's handled.
def handleCommand(command):
    lprint(str(command))
    # the command comes in from php as something like
    # ('s:17:"AcidPump, pumpOff";', 2)
    # so command[0] is 's:17:"AcidPump, pumpOff'
    # then split it at the "  and take the second item
    try:
        c = str(command[0].split('\"')[1]).split(',')
    except IndexError:
        c = str(command[0]).split(' ')    #this is for something I sent from another process
    lprint(c)
    if (c[0] == 'OutsideLightsOn'):
        outsideLightsOn()
    elif (c[0] == 'OutsideLightsOff'):
        outsideLightsOff()
    elif (c[0] == 'fPorchToggle'):
        toggle("frontporch")
    elif(c[0] == 'garageToggle'):
        toggle("outsidegarage")
    elif (c[0] == 'cactusToggle'):
        toggle("cactusspot")
    elif (c[0] == 'patioToggle'):
        toggle("patio")
    else:
        lprint("Weird command = " + str(c))

# First the process interface, it consists of a status report and
# a command receiver.
class WemoSC(object):
    @cherrypy.expose
    @cherrypy.tools.json_out() # This allows a dictionary input to go out as JSON
    def status(self):
        status = []
        for item in switches:
            status.append({item["name"]:get(item["name"])})
        return status
        
    @cherrypy.expose
    def pCommand(self, command):
        handleCommand((command,0));
        
    @cherrypy.expose
    def index(self):
        status = "<strong>Current Wemo Light Switch Status</strong><br /><br />"
        for item in switches:
            status += item["name"] +" is " + get(item["name"]) + "&nbsp;&nbsp;"
            status += '<a href="wemocommand?whichone='+item["name"]+'"><button>Toggle</button></a>'
            status += "<br />"
        return status
        
    @cherrypy.expose
    def wemocommand(self, whichone):
        # first change the light state
        toggle(whichone)
        # now reload the index page to tell the user
        raise cherrypy.InternalRedirect('/index')

# given the ip of a Belkin device this will try the ports that
# are used on the Wemo switches to see which one works.  The assumption
# is that if none of the ports work, it's not a switch, it's a modem or
# something else.
def getPort(ip):
    entry = []
    for p in ["49153", "49154", "49155"]:
        try:
            resp = talkHTML(ip + ':' + p + "/setup.xml", "")
            if debug:
                print "\tfound one at", b[0], "port", p
            if showXml:
                print stars
                print "response from switch"
                print resp
                print stars
            name = extractSingleTag(resp, 'friendlyName')
            model = extractSingleTag(resp, 'modelName')
            entry = {"mac":b[1],"ip":b[0], "port":p, "name":name, "model":model}
            return entry
        except timeout:
            continue
        except urllib2.URLError:
            continue
        except:
            e = sys.exc_info()[0]
            print ("Unexpected Error: %s" % e )
            continue
    return None
        
####################### Actually Starts Here ################################    
debug = False
showXml = False
if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("-d", "--debug",
        action = "store_true",
        help='debug flag')
    parser.add_argument("-x", "--xml",
        action = "store_true",
        help='show xml')
    parser.add_argument('count',type=int);
    args = parser.parse_args()
    if args.debug:
        print "Running with debug on"
        debug = True
    if args.xml:
        print "Running with showXML on"
        showXml = True
    targetNumber = args.count

    stars = "*********************************************"

    #-------------------------------------------------
    # the database where I'm storing stuff
    DATABASE=getHouseValues()["database"]
    lprint("Using database ", DATABASE);
    # Get the ip address and port number you want to use
    # from the houserc file
    ipAddress=getHouseValues()["wemocontrol"]["ipAddress"]
    port = getHouseValues()["wemocontrol"]["port"]
    lprint("started looking for {} switches".format(targetNumber))

    # This works on my machine, but you may have to mess with it
    # The arp-scan below tells it not to look up the manufacturer because I
    # didn't want to worry about keeping the tables that are used up to date,
    # the -l tells it to find the local net address on its own, and 
    # -v (verbose) will print that net address so I can show it for debugging
    # I take the scan range out of the .houserc file, it's an entry under wemocontrol
    # that looks like "scanRange":"192.168.0.1-192.168.0.50" adjust this as
    # needed
    try:
        scanRange = getHouseValues()["wemocontrol"]["scanRange"]
        arpCommand = "arp-scan -q -v %s 2>&1" %(scanRange)
    except KeyError:
        print "No entry in .houserc for wemocontrol scanRange"
        exit();

    while True:
        devices = [];
        # first the devices on the network
        if debug:
            print "arp-scan command is:", arpCommand
        theList = subprocess.check_output(arpCommand,shell=True);
        # split the output of the arp-scan into lines instead of a single string
        lines = theList.splitlines()
        # this looks at each line and grabs the addresses we're interested in
        # while ignoring the lines that are just information.
        for line in lines:
            allowedDigits = set("0123456789abcdef:. \t")
            if all(c in allowedDigits for c in line):
                d = line.split()
                try:
                    devices.append([d[0], d[1]])
                except IndexError: # an empty line will pass the test
                    continue
        # arp-scan can give the same addresses back more than once
        # step through the list and remove duplicates
        temp = []
        for e in devices:
            if e not in temp:
                temp.append(e)
        devices = temp
        if debug:
            print devices
        # for each device, look up the manufacturer to see if it was registered
        # to belkin
        bDevices = []
        # I got this list direct from the IEEE database and it may
        # need to be updated in a year or two.
        belkinList = ("001150", "00173F", "001CDF", "002275", "0030BD", 
                        "08863B", "94103E", "944452", "B4750E", "C05627", "EC1A59")
        for d in devices:
            if d[1].replace(':','')[0:6].upper() in belkinList:
                    bDevices.append([d[0],d[1]])
        if debug:
            print "These are the Belkin devices on the network"
            print bDevices
        if len(bDevices) < targetNumber: 
            lprint ("Only found", len(bDevices), "Belkin devices, retrying")
            time.sleep(1)
            continue
        # Got all that were asked for, continue to the next step
        
        # Now that we have a list of the Belkin devices on the network
        # We have to examine them to be sure they are actually switches
        # and not a modem or something else.  This will also assure that 
        # they will actually respond to a request.  They still may not work,
        # but at least we have a chance.
        switches = []
        for b in bDevices:
            result = getPort(b[0])
            if result is not None:
                switches.append(result)
        # Did we find enough switches ?
        if len(switches) < targetNumber: 
            lprint ("Only found", len(switches), "of them, retrying")
            devices = []
            continue
        # Yes we did, break out.
        break;
    # Now I'm going to check the database to see if it has been
    # adjusted to hold all the items (older version didn't have
    # ip, port, and mac addresses
    dbconn = sqlite3.connect(DATABASE)
    c = dbconn.cursor()
    c.execute("pragma table_info(lights);")
    dbrow = c.fetchall()
    if not any('ip' and 'mac' and 'port' in r for r in dbrow):
        lprint ("Database needs to be adjusted")
        lprint ("to hold ip, port, and MAC")
        try:
            print "adding ip if needed"
            c.execute("alter table lights add column ip text;")
        except sqlite3.OperationalError:
            print "ip was already there"
        try:
            print "adding mac if needed"
            c.execute("alter table lights add column mac text;")
        except sqlite3.OperationalError:
            print "mac was already there"
        try:
            print "adding port if needed"
            c.execute("alter table lights add column port text;")
        except sqlite3.OperationalError:
            print "port was already there"
        dbconn.commit()
    else:
        lprint ("Database already adjusted")
    for item in switches:
        try:
            c.execute("update lights " 
                "set ip = ?, mac=?, port=?where name = ?;",
                (item["ip"], item["mac"], item["port"], item["name"]))
            dbconn.commit()
        except sqlite3.OperationalError:
            lprint("Database is locked, record skipped")
    dbconn.commit()
    dbconn.close()
    lprint ("")
    lprint ("The list of", len(switches), "switches found is")
    for item in switches:
        lprint ("Friendly name:", item["name"])
        lprint ("Model:", item["model"])
        lprint ("IP address:", item["ip"])
        lprint ("Port number:", item["port"])
        lprint ("MAC:", item["mac"])
        lprint ('')
        
    # timed things.
    checkLightsTimer = timer(doLights, seconds=2)
    keepAliveTimer = timer(keepAlive, minutes=4)
    # Now configure the cherrypy server using the values
    cherrypy.config.update({'server.socket_host' : ipAddress.encode('ascii','ignore'),
                            'server.socket_port': port,
                            'engine.autoreload.on': False,
                            })
    # Subscribe to the 'main' channel in cherrypy with my timer
    cherrypy.engine.subscribe("main", checkTimer.tick)
    lprint ("Hanging on the wait for HTTP message")
    # Now just hang on the HTTP server looking for something to 
    # come in.  The cherrypy dispatcher will update the things that
    # are subscribed which will update the timers so the light
    # status gets recorded.
    cherrypy.quickstart(WemoSC())
    
    sys.exit("Told to shut down");

Notice that it is self contained.  It doesn't require Miranda.py anymore, nor any of the Ouimeaux libraries.  That's what makes it appealing to me; it's nice to have everything in one module and reasonably easy to understand.  I added a few parameters:

wemocontrol.py -h
usage: wemocontrol.py [-h] [-d] [-x] count

positional arguments:
  count

optional arguments:
  -h, --help   show this help message and exit
  -d, --debug  debug flag
  -x, --xml    show xml

I start it with wemocontrol.py 4, because I have four switches.  It will keep looking until it finds all four switches which could be a problem if one of them is disconnected, but I'll deal with that problem as it happens.  The -x parameter displays the SOAP messages as they are sent and received.  This can be used to find problems in the protocol or to understand how it works.  The -d is for debugging when you can't get things to work.  If you turn them both on, the volume of output is huge, so be prepared for it.  The -h is because I can never remember this stuff and wanted a reminder.  To run it, you must have an entry in a file called .houserc in the home directory of the user.  I use this file for all the entries for my processes.  This way I can keep things like keys, directory paths, etc in one place easily found.  The entries I have look like this:

"wemocontrol":{
"ipAddress":"192.168.0.205",
"port": 51001,
"scanRange":"192.168.0.1-192.168.0.50"},

I have a complete description of the file and how to use it here <link>.

Now, I must thank Glenn for his help in this effort.  This thing would have been a bear to test without him.  The rest of you, have fun with it.

Monday, November 10, 2014

Wemo Light Switches and Intermittent Failures ... Found it!

I have four Wemo light switches operating around the house on outside lights.  You know the ones.  They're the switches that you remember after you get in bed and have the covers pulled up.  In my case, I just grab a phone and turn the darn things off from the bed ... except ...

Every once in a while they don't respond.  This has driven me nuts ever since I got the switches and I'm not alone.  There are other folk out there that have been annoyed by this behavior.  After I simplified my code running the switches <link>, and the problem was still there; I set out to see what the heck was going on.  At first the code would hang waiting for a response from the switches, so I put a timeout on the socket read.

When the inevitable timeout came, I at least had a clue; the switch just quit responding.  So, I caught the timeout and printed the URL that was involved.  A few days later, it timed out again and I discovered that the IP address was ok, but the port number the Wemo was listening to had changed.  There's simply no reason for the port number to change ... unless the switch had done it intentionally.  Naw, no one would do that.

Then, the power failed to my house and everything rebooted.  Fine, I'd wait until it happened again.

One evening, one of the switches failed.  I got an email telling me the switch had gone off line.  Yes, an email.  One of the great things about working on a Pi is that it can do things like this.  I had put code in to send me an email when one of the switches got into this state.  I checked the logs (another great thing) and sure enough, the port number had changed.  A little while later, another switch failed, then the other two as well.  The period of working was roughly three days, and then, the switches would change the port number they responded to.

I don't have definitive proof, but I firmly believe they reboot the switches every three days.  Over time I could probably accumulate evidence to support this, but I don't want evidence; I want the darn switches to work.

What I had done in the code was to set a timeout on the socket receive from a polling request to the switch.  When it timed out, it would send me mail and then my process would exit.  Since the process is controlled by upstart (google it), it would start right back up and rediscover the switches.  Everything should be fine then because the new port number would be recorded and control would be set up correctly.  I actually rebooted the control process to just start over.  Not very elegant, but it worked.  When the four switches went south, the process restarted each time and got control back.

Ha, take that Belkin.

Of course there's a downside.  During the time the process is restarting, there could be a command to the switches missed.  It takes about 30 seconds to discover the switches and get all set up so the window is small, but it's there.  I think I can live with that.  I removed the code that sends mail to keep the changes I make to Miranda to an absolute minimum and have the changes running right now.  After a couple of weeks, I'll know how well it worked out.

It's very important to me to avoid relying on an external service to control something around my house, so things like If This Then That, are not an option.  Sure, I like the Wemo app for checking the lights from town, but I don't want to rely on it.  The Internet is just too flaky, companies change policies, and people like to muck with other people's stuff.  I want control inside my own house.

I'm getting closer now.

Sunday, October 5, 2014

Controlling Wemo Light Switches, Another Look

No, I'm not dead, and yes, I still work on electronic projects.  I've been busy with projects that are much more mundane and normal.  Things like trying to build a pad for my motorcycle trailer, fighting with my chlorine generator for the pool, and recovering from my house flooding during the monsoon rains here in AZ.  Yep, I had water come in one side of the house and go out the other.  I'm still working on the damage from that episode.

But people don't read my posts to hear about that kind of thing, they probably have worse problems around the house than I do ... I don't get snow or tornadoes.

For months now I've had Wemo light switches running.  They're a nice little device that Belkin puts out and seem to be gaining in popularity.  Heck, Home Depot sells them in the electrical section now.  However, they rely on an app on the cell phone and a cloud server.  That may not be a concern for most people, but I personally hate having things I depend on subject to the whim of a company that might just decide to start charging me for the service in the future.  That's why I put together some software to run the switches from my Raspberry Pi based house controller <link><link><link>.

However the library and toolset I used, while a great effort and really classy, was too complex and kept failing.  Let me be perfectly clear, the ouimeaux library I used was good, the problem is that the wemo switches don't always work like Universal Plug and Play (upnp) devices are supposed to.  I don't want to get into a lengthy description of upnp here, there are a bunch of those out there, but suffice it to say that the subscription capabilities of the switches fails.

So, you discover the devices, set up a subscription where the devices let you know when the state changes happen, and then sometime later, the switch quits responding.  That means sometime or other, the light doesn't turn on or off and you have to restart the process that controls them to get the switch working again.  Additionally, the ouimeaux library uses a LOT of other tools and provides features that I'm just not interested in.  It uses gevent to handle asynchronous operations, has a little web server, client software, enough stuff for a major project.  I just want to turn them on and off and look to see which way they are now.  Using something large like this to do a simple job leads to huge problems when something fails.  There's just so much there that you can go nuts chasing a problem.

You guessed it, I wrote my own.  Well, that's actually an exaggeration.  I stole some stuff and used it would be closer to the truth.  I started out by writing a handler to discover upnp devices on my network.  If you've never done that, I recommend you look into it just for fun.  I discovered my satellite receiver, my Roku, the DSL modem, all the Wemo switches, and several other devices that I didn't know had plug and play interfaces.  That was a royal pain.  I had to learn about SSDP, SOAP, multicast, XML, and the list was growing.  Shoot, all I want to do was control the darn switches, not create a dissertation on half the jargon on the web.  I did learn a lot about discovering the devices on my network, but that wasn't my goal.

That led me to a few days of searching on the web for upnp libraries.  They're out there, and some of them are really good, but they're written in something besides python.  Sure a couple  have interfaces, but the documentation went totally over my head.  There were some in python, but a couple had disappeared off the web, and one just didn't work (the samples even had syntax errors).  What should I do?  I remembered reading somewhere that python can import modules, and that the python tool Miranda worked reasonably well to experiment with upnp devices ... hmmm.

I got the latest Miranda and combined it with my own code and came up with a way to read and control the switches that uses only the basic python libraries.  It was a challenge hooking into someone else's code that was designed to run as a standalone tool, but it worked.  That way I leveraged the huge amount of work the author of Miranda did to make my job easier.  The big bonus was that Miranda is also really good as a upnp learning tool.  Ha, I have a debugger and a controller all in one.

Since I had spent a lot of time debugging the switch operation to determine why they would quit, I didn't want to use the subscription service.  The subscription facility is asynchronous, so the switch will send an XML message over UDP to a registered URL any time the state changes (off to on); too bad it doesn't work all the time.  That means that I would have to poll the switches to see if something or someone other than my code on the controller had changed their state.  Let me elaborate on that a bit.  These are wall switches; I can walk over and push the button and turn off the lights.  Since my software can't see that, it has to ask the switch what state it is in right now.

That's fine, I can poll it every couple of seconds and see what is happening, BUT there's a problem with that.  I love APScheduler a great timer tool in python; I have it in almost everything I do, but it uses threads to control when things happen.  Using threads and database updates is a path to disaster.  When I tried my new code out, about half the time it would leave the database locked because something jerked execution away at an inopportune time.  I even updated my code and libraries to the latest version of APScheduler which allows jobs to be paused, and still had the problem.  That meant that I had to figure out how to schedule polling of the switches within a single thread.

After a few magnificent failures I came up with a pretty simple way to do it; my method is in the code below.  I may have to use this in some of the previous work I've done, it's really small and simple, so it could decrease the load on the poor little Pi.

Even though this took an inordinate amount of time and study, it's still pretty darn nice.  I haven't ran it for months, but it's held up for a short time pretty well.  I could well develop problems with sockets hanging or something, but the code is simple and straight forward, so I should be able to isolate whatever problems turn up and fix them.  I don't have eight libraries of very complex code written by someone else to step through trying to find out how something works.  It's me and Miranda here, ... well there are the system libraries, but nothing terribly exotic.

Here's the code.  It's written in my usual style: variable names that are too darn long, comments for practically every darn line, over indentation, etc.  Basically everything I need to pick it up a year from now to fix a bug or add a feature.

#! /usr/bin/python
from miranda import upnp
from miranda import msearch
from miranda import set
import sys
#from apscheduler.schedulers.blocking import BlockingScheduler
import datetime
from datetime import timedelta
from datetime import datetime
import time
import sysv_ipc
import logging
import sqlite3
import pdb #yes, I had trouble and had to use this !!
#-------------------------------------------------
# the database where I'm storing stuff
DATABASE='/home/pi/database/desert-home'

def lprint(text):
 print time.strftime("%A, %B, %d at %H:%M:%S"),text
 sys.stdout.flush()
 
def _send(action, whichone, args):
 if not args:
  args = {}
 entry = (item for item in lightSwitches if item["name"] == whichone).next()
 index =entry['index']
 host_info = conn.ENUM_HOSTS[index]
 device_name = 'lightswitch'
 service_name = 'basicevent'
 controlURL = host_info['proto'] + host_info['name']
 controlURL2 = host_info['deviceList'][device_name]['services'][service_name]['controlURL']
 if not controlURL.endswith('/') and not controlURL2.startswith('/'):
  controlURL += '/'
 controlURL += controlURL2

 resp = conn.sendSOAP(
  host_info['name'],
  'urn:Belkin:service:basicevent:1',
  controlURL,
  action,
  args
 )
 return resp
 
def get(whichone):
 ''' 
 Returns True if the light is on and False if not
 '''
 resp = _send('GetBinaryState', whichone, {})
 tagValue = conn.extractSingleTag(resp, 'BinaryState')
 return 'On' if tagValue == '1' else 'Off'

def handleUpdate(whichone, status):
 for i in lightSwitches:
  if i['name'] == whichone:
   i['status'] = status
 updateDatabase(whichone, status)

def on(whichone):
 """
 BinaryState is set to 'Error' in the case that it was already on.
 """
 resp = _send('SetBinaryState', whichone, {'BinaryState': (1, 'Boolean')})
 tagValue = conn.extractSingleTag(resp, 'BinaryState')
 status = 'On' if tagValue in ['1', 'Error'] else 'Off'
 handleUpdate(whichone, status)
 lprint("turned %s on"%(whichone))
 return status

def off(whichone):
 """
 BinaryState is set to 'Error' in the case that it was already off.
 """
 resp = _send('SetBinaryState', whichone, {'BinaryState': (0, 'Boolean')})
 tagValue = conn.extractSingleTag(resp, 'BinaryState')
 status = 'Off' if tagValue in ['0', 'Error'] else 'On'
 handleUpdate(whichone, status)
 lprint("turned %s off"%(whichone))
 return status

def doLights():
 for switch in lightSwitches:
  thisOne = switch['name']
  updateDatabase(thisOne,get(thisOne))

def doComm():
 global firstTime
 #global scheditem
 
 try:
  if (firstTime):
   while(True):
    try:
     # commands could have piled up while this was 
     # not running. Clear them out.
     junk = Cqueue.receive(block=False, type=0)
     print "purging leftover commands", str(junk)
    except sysv_ipc.BusyError:
     break
   firstTime=False
  while(True):
   newCommand = Cqueue.receive(block=False, type=0)
   # type=0 above means suck every message off the
   # queue.  If I used a number above that, I'd
   # have to worry about the type in other ways.
   # note, I'm reserving type 1 messages for 
   # test messages I may send from time to 
   # time.  Type 2 are messages that are
   # sent by the php code in the web interface.
   # Type 3 are from the event handler. This is just like
   # the house monitor code in that respect.
   # I haven't decided on any others yet.
   handleCommand(newCommand)
 except sysv_ipc.BusyError:
  pass # Only means there wasn't anything there 


def handleCommand(command):
 #lprint(" " + str(command))
 # the command comes in from php as something like
 # ('s:17:"AcidPump, pumpOff";', 2)
 # so command[0] is 's:17:"AcidPump, pumpOff'
 # then split it at the "  and take the second item
 try:
  c = str(command[0].split('\"')[1]).split(',')
 except IndexError:
  c = str(command[0]).split(' ')   #this is for something I sent from another process
 #lprint(c)
 if (c[0] == 'OutsideLightsOn'):
  outsideLightsOn()
 elif (c[0] == 'OutsideLightsOff'):
  outsideLightsOff()
 elif (c[0] == 'fPorchToggle'):
  toggle("frontporch")
 elif(c[0] == 'garageToggle'):
  toggle("outsidegarage")
 elif (c[0] == 'cactusToggle'):
  toggle("cactusspot")
 elif (c[0] == 'patioToggle'):
  toggle("patio")
 else:
  lprint(" Weird command = " + str(c))

def outsideLightsOn():
 lprint (" Outside lights on")
 on("outsidegarage")
 on("frontporch")
 on("cactusspot")
 
def outsideLightsOff():
 lprint (" Outside lights off")
 off("outsidegarage")
 off("frontporch")
 off("cactusspot")

def toggle(whichOne):
 if (get(whichOne) == 'On'):
  off(whichOne)
 else:
  on(whichOne)
  
def keepAlive():
 '''
 For my own purposes, I update the database periodically with the time
 so I can check to see if things are holding together.  I currently use the
 time in the light switch records for this.
 '''
 lprint(" keep alive")
 for switch in lightSwitches:
  thisOne = switch['name']
  updateDatabase(thisOne, get(thisOne), force=True)

def updateDatabase(whichone, status, force=False):
 ''' 
 This is running on a Pi and is not event driven, so polling like
 this will result in considerable wear to the SD card.  So, I'm going to 
 read the database to see if it needs to be changed before I change it.  
 According to everything I've read, reads are free, it's the writes that
 eat up the card.
 '''
 dbconn = sqlite3.connect(DATABASE)
 c = dbconn.cursor()
 c.execute("select status from lights where name = ?;",
  (whichone,))
 oldstatus = c.fetchone()
 if oldstatus[0] != status or force == True:
  lprint ("Had to update database %s, %s"%(whichone, status))
  c.execute("update lights " 
   "set status = ?, utime = ? where name = ?;",
   (status, time.strftime("%A, %B, %d at %H:%M:%S"), whichone))
  dbconn.commit()
 dbconn.close()
  
if __name__ == "__main__":
 #When looking at a log, this will tell me when it is restarted
 lprint (" started")
 firstTime = True
 debug = False
 if not debug:
  conn = upnp(False,False,None,0)
  ''' 
  I don't want the search for devices to run forever 
  So, I set the timeout for miranda to some number of seconds
  to limit it.
  '''
  set(3, ["set","timeout", "10"], conn)
  ''' 
  This looks at the devices that responded and gathers more data about
  them by sending them a request to itemize their capabilities.

  Sometimes a upnp device goes nuts and responds way out of
  proportion.  You can get the same device in the tables
  many times, so set the uniq to True
  
  Also, the Wemo switches don't always respond to a discover specific to 
  them.  That means I have to do a general discover and get all the devices
  on the network.  This sucks because it slows things down, so if anyone
  overcomes this problem, let me know how.
  '''
  set(3, ["set","uniq", True], conn)
  ''' This is the actual search '''
  msearch(1,[msearch],conn)
  ''' and now do the interaction '''
  for index, hostInfo in conn.ENUM_HOSTS.iteritems():
   #print "************** ", index, " of ", len(conn.ENUM_HOSTS) - 1
   ''' on my network, I have a rogue device that reports badly '''
   if hostInfo['name'].find('192.168.16.254') == 0:
    print "Odd device, ignoring"
    continue
   ''' if you want to see them as they come in, uncomment this '''
   #print hostInfo
   if hostInfo['dataComplete'] == False:
    xmlHeaders, xmlData = conn.getXML(hostInfo['xmlFile'])
    conn.getHostInfo(xmlData,xmlHeaders,index)
    
  ''' 
  now to select only the light switches from the various devices 
  that responded 
  '''
  lightSwitches=[]
  for index, host_info in conn.ENUM_HOSTS.iteritems():
   if "deviceList" in host_info:
    if "lightswitch" in host_info["deviceList"]:
     name = host_info["deviceList"]["lightswitch"]["friendlyName"]
     lightSwitches.append({"name": name, "index": index, "status" : 'unknown'})
  ''' 
  OK, now I have the list of Wemo light switches that are around the 
  house, so print it and show the state of each one 
  '''
  print "this is the list of the", len(lightSwitches), "Wemo switches found."
  for switch in lightSwitches:
   switch['status'] = get(switch['name'])
   print switch
   
 # Create the message queue where commands can be read
 # I just chose an identifier of 13 because the house monitor
 # already took the number 12.
 Cqueue = sysv_ipc.MessageQueue(13, sysv_ipc.IPC_CREAT,mode=0666)
 '''
 This is a poor man's timer for task control.  I may put this in a class
 after I've run it for a while.  The reason I did it this way is that 
 APSscheduler creates a separate thread to handle timers and I don't 
 want the contention to the database of separate threads doing things
 that way.
 
 To use it, just put another entry into the table.
 '''
 lprint (" Setting up timed items")
 startTime = int(time.time())
 tasks = [{'whichOne': doLights,'interval' : 2, 'next': startTime+10},
  {'whichOne': keepAlive, 'interval' : (4*60), 'next' : startTime+15}]
 lprint ("going into the processing loop")
 print startTime
 while True:
  
  #pdb.set_trace()
  now = int(time.time())
  for task in tasks:
   if task['next'] <= now:
    task['next'] = now + task['interval']
    #print task['whichOne']
    task['whichOne']()
  doComm()
  ''' 
  doing a sleep here releases the cpu for longer than the program runs
  That way I reduce the load on the machine so it can do more stuff
  '''
  time.sleep(0.25) 
  pass 
 
 sys.exit("done");

It's only fair to thank Ian McCracken who wrote the Ouimeaux library, Isaac Kelly who did much of the initial discovery work, and Craig Heffner the author of Miranda.  My work is built on theirs.

No, I don't have support in here for the Wemo motion sensor, crock pot, or other items they may come up with.  If you want to grab it and extend it to support those devices, feel free; just let me know so I can get the improvements as well.

There's also support in here for the sysv messages I use to communicate between processes on the machine.  Feel free to rip that out and put in anything you want.  I wanted to give you an example that I'm actually running right now, not something I cut up to look pretty.

Just for fun, here's the output of the code that is captured to my log:

Sunday, October, 05 at 16:22:06  started
action timeout
action uniq
Show unique hosts set to: True
argc 1
argv [<function msearch at 0xb69d94f0>]
Entering discovery mode for 'upnp:rootdevice', Ctl+C to stop...

Error updating command completer structure; some command completion features mig
ht not work...
****************************************************************
SSDP reply message from 192.168.0.34:49312
XML file is located at http://192.168.0.34:49312/device.xml
Device is running Linux/2.6.37.6-4.0, UPnP/1.0, Portable SDK for UPnP devices/1.
6.17
****************************************************************

Error updating command completer structure; some command completion features mig
ht not work...
****************************************************************
SSDP reply message from 192.168.0.34:49200
XML file is located at http://192.168.0.34:49200/device.xml
Device is running Linux/2.6.37.6-4.0, UPnP/1.0, Portable SDK for UPnP devices/1.
6.17
****************************************************************

Error updating command completer structure; some command completion features mig
ht not work...
****************************************************************
SSDP reply message from 192.168.0.28:49154
XML file is located at http://192.168.0.28:49154/setup.xml
Device is running Unspecified, UPnP/1.0, Unspecified
****************************************************************

Error updating command completer structure; some command completion features mig
ht not work...
****************************************************************
SSDP reply message from 192.168.0.29:49153
XML file is located at http://192.168.0.29:49153/setup.xml
Device is running Unspecified, UPnP/1.0, Unspecified
****************************************************************

Error updating command completer structure; some command completion features mig
ht not work...
****************************************************************
SSDP reply message from 192.168.0.43:49154
XML file is located at http://192.168.0.43:49154/setup.xml
Device is running Unspecified, UPnP/1.0, Unspecified
****************************************************************

Error updating command completer structure; some command completion features mig
ht not work...
****************************************************************
SSDP reply message from 192.168.0.26:49153
XML file is located at http://192.168.0.26:49153/setup.xml
Device is running Unspecified, UPnP/1.0, Unspecified
****************************************************************

Error updating command completer structure; some command completion features mig
ht not work...
****************************************************************
SSDP reply message from 192.168.0.22:2869
XML file is located at http://192.168.0.22:2869/upnphost/udhisapi.dll?content=uu
id:3a682b82-e803-4105-bfeb-114ed775cab1
Device is running Microsoft-Windows/6.3 UPnP/1.0 UPnP-Device-Host/1.0
****************************************************************

Error updating command completer structure; some command completion features mig
ht not work...
****************************************************************
SSDP reply message from 192.168.0.14:8060
XML file is located at http://192.168.0.14:8060/
Device is running Roku UPnP/1.0 MiniUPnPd/1.4
****************************************************************

Error updating command completer structure; some command completion features mig
ht not work...
****************************************************************
SSDP reply message from 192.168.0.1:80
XML file is located at http://192.168.0.1:80/DeviceDescription.xml
Device is running ZyXEL-RomPlug/4.51 UPnP/1.0 IGD/1.00
****************************************************************


Discover mode halted...
Failed to get argument name for OpenInstaAP
Failed to get argument name for CloseInstaAP
Failed to get argument name for OpenInstaAP
Failed to get argument name for CloseInstaAP
Failed to get argument name for OpenInstaAP
Failed to get argument name for CloseInstaAP
Failed to get argument name for OpenInstaAP
Failed to get argument name for CloseInstaAP
Caught exception while parsing device service list: list index out of range
At index  7
Caught exception while parsing device service list: list index out of range
At index  8
this is the list of the 4 Wemo switches found.
{'status': 'Off', 'index': 2, 'name': 'cactusspot'}
{'status': 'Off', 'index': 3, 'name': 'frontporch'}
{'status': 'Off', 'index': 4, 'name': 'patio'}
{'status': 'Off', 'index': 5, 'name': 'outsidegarage'}
Sunday, October, 05 at 16:22:42  Setting up timed items
Sunday, October, 05 at 16:22:42 going into the processing loop
1412551362
Sunday, October, 05 at 16:22:56 Had to update database patio, On
Sunday, October, 05 at 16:22:56 turned patio on
Sunday, October, 05 at 16:22:57  keep alive
Sunday, October, 05 at 16:22:57 Had to update database cactusspot, Off
Sunday, October, 05 at 16:22:57 Had to update database frontporch, Off
Sunday, October, 05 at 16:22:57 Had to update database patio, On
Sunday, October, 05 at 16:22:58 Had to update database outsidegarage, Off
Sunday, October, 05 at 16:23:04 Had to update database patio, Off
Sunday, October, 05 at 16:23:04 turned patio off
Sunday, October, 05 at 16:23:12  Outside lights on
Sunday, October, 05 at 16:23:12 Had to update database outsidegarage, On
Sunday, October, 05 at 16:23:12 turned outsidegarage on
Sunday, October, 05 at 16:23:12 Had to update database frontporch, On
Sunday, October, 05 at 16:23:12 turned frontporch on
Sunday, October, 05 at 16:23:12 Had to update database cactusspot, On
Sunday, October, 05 at 16:23:12 turned cactusspot on
Sunday, October, 05 at 16:23:17  Outside lights off
Sunday, October, 05 at 16:23:17 Had to update database outsidegarage, Off
Sunday, October, 05 at 16:23:17 turned outsidegarage off
Sunday, October, 05 at 16:23:17 Had to update database frontporch, Off
Sunday, October, 05 at 16:23:17 turned frontporch off
Sunday, October, 05 at 16:23:17 Had to update database cactusspot, Off
Sunday, October, 05 at 16:23:17 turned cactusspot off
At the top is where I set some parameters in Miranda to control what I discover.  Then the discovery process prints out a summary of the devices it finds. The errors there are caused by the Wemo switches not responding properly.  Next are messages from the conversation with the switches.  Once again, the switches don't behave.  Finally the last step of initialization where the status and names of the switches show up.  The rest of the log is normal operation.  I poll the switches every two seconds to see if they're on or off. and every four minutes to make sure the database is kept current.  I use database currency to monitor the health of the controller.

If you have these switches, or are thinking about using them, grab the code and start playing.

Wednesday, May 14, 2014

Wemo Devices Are Getting Interesting Now

Previous posts about the Wemo light switch: Part 1, Part 2

I'm not one of those guys that gathers tech news items and reposts them to increase my page views or hits, but this is going to be an exception.  I was prowling around looking for ideas for a particular problem I'm having with the Wemo devices I have and ran across a cool news item <link>.  Basically, it says that Belkin is expanding their line of WiFi controlled devices to include light bulbs, slow cookers, coffee makers, humidfiers, and who knows what else.

This is so cool.  Imagine, a coffee pot with an internet address.

I thought I'd capture a picture from a 'coming soon' article that really caught my eye:


I stole the picture and put it here because the advertisement will probably disappear soon.  The link is from the CrockPot web site <link>, and is for a Wemo based slow cooker that is going on my 'gottahavethatthing' list.  Being a widower, I use a slow cooker a lot and this would be great to have.

But look, space heaters, humidifiers, coffee makers, what will they think of next?  Sure, I use things like this to control or monitor a lot of devices and I could probably build my own remote controlled coffee maker, but the neighbors can't.  Now, they can just buy it.

So, it looks like Belkin is in this marketplace big time.  They actually published their API for Android (yes, I downloaded it) so other people can get into the arena.  With the library work in Python and several other languages, the home hacker can start building nice controls for their own use.  Normal folk can use the free phone app and do much the same with no work (or personal satisfaction).

Home automation is looking better all the time.

Wednesday, January 15, 2014

Belkin's Wemo Light Switch Part 2

It's been over a month (well, almost two months) since I wrote about the Wemo switch.  During that time, Belkin has come out with the Insight switch and a couple of iterations of the Android software.  It still doesn't work well on my phone, their online server has problems when you add switches.  Seems they forget you already had other switches.  I'm not willing to reinitialize everything in my house each time I do something to one of them.  But:

I have three of the switches now and they're all hooked into my Raspberry Pi house controller.  Over the last couple of months I've found a couple of problems with the ouimeaux python library that the author is looking at, and of course, with my code to run them.  Unlike the Android software, the wall switches work really well.  Occasionally the switch that is at the edge of my Wifi network will fail to report and twice now I've had one of the switches respond to a command with an error.  Really good performance for a remote switch and far, far better than the X10 stuff they replaced.

The really cool thing is that the switches report when they change state.  So, if I change it over the phone, the switch tells my Pi that it happened.  If I push the button, same thing.  This is great because you can check the state of a switch at any time to see if you left the lights on.  Using my Pi to control them works great as well.  I don't have to rely on Belkin's server or that 'if this then that' website to control things.  I can set up as many schedules as I want and don't have to worry about somebody else's server having problems.

For you folk out there that would like to mess with these switches, here's the code I'm using to control them:

The Python Script
#! /usr/bin/python
import os
import signal
import sys
import logging
from apscheduler.scheduler import Scheduler
from threading import Thread
import datetime
from datetime import timedelta
import time
import sysv_ipc


from ouimeaux.upnp import UPnPLoopbackException
from ouimeaux.environment import Environment
from ouimeaux.config import get_cache, in_home, WemoConfiguration
from ouimeaux.utils import matcher

#-------------------------------------------------

def lprint(text):
 print time.strftime("%A, %B, %d at %H:%M:%S"),text
 sys.stdout.flush()
 
def signal_handler(signal,frame):
 lprint (" Exit Signal")
 sys.exit(0)

def on_switch(switch):
 lprint (" Switch found! " + switch.name)

def on_motion(motion):
 lprint (" Motion found!" + switch.motion)
 
def outsideLightsOn():
 lprint (" Outside lights on")
 garageLightSwitch.on()
 cactusLightSwitch.on()
 fPorchLightSwitch.on()
 
def outsideLightsOff():
 lprint (" Outside lights off")
 garageLightSwitch.off()
 cactusLightSwitch.off()
 fPorchLightSwitch.off()
 
def fPorchToggle():
 if (int(fPorchLightSwitch.get_state()) == 1):
  fPorchLightSwitch.off()
 else:
  fPorchLightSwitch.on()
  
def garageToggle():
 if (int(garageLightSwitch.get_state()) == 1):
  garageLightSwitch.off()
 else:
  garageLightSwitch.on()
  
def cactusToggle():
 if (int(cactusLightSwitch.get_state()) == 1):
  cactusLightSwitch.off()
 else:
  cactusLightSwitch.on()
 
def updateDatabase(whichone, status):
  pass

def cactusSwitchChange(value):
 if (int(value) == 1):
  lprint (" Cactus Spot On")
  updateDatabase('cactusspot', 'On')
 elif (int(value) == 0):
  lprint (" Cactus Spot Off")
  updateDatabase('cactusspot', 'Off')
 else:
  lprint (" Cactus Spot Switch Unknown value: " + value)
  
def garageSwitchChange(value):
 if (int(value) == 1):
  lprint (" Garage Outside Light On")
  updateDatabase('outsidegarage', 'On')
 elif (int(value) == 0):
  lprint (" Garage Outside Light Off")
  updateDatabase('outsidegarage', 'Off')
 else:
  lprint (" Garage Outside Light Switch Unknown value: " + value)
  
def fPorchSwitchChange(value):
 if (int(value) == 1):
  lprint (" Front Porch Light On")
  updateDatabase('frontporch', 'On')
 elif (int(value) == 0):
  lprint (" Front Porch Light Off")
  updateDatabase('frontporch', 'Off')
 else:
  lprint (" Front Porch Light Switch Unknown value: " + value)
  
def handleCommand(command):
 #lprint(" " + str(command))
 # the command comes in from php as something like
 # ('s:17:"AcidPump, pumpOff";', 2)
 # so command[0] is 's:17:"AcidPump, pumpOff'
 # then split it at the "  and take the second item
 try:
  c = str(command[0].split('\"')[1]).split(',')
 except IndexError:
  c = str(command[0]).split(' ')    #this is for something I sent from another process
 #lprint(c)
 if (c[0] == 'OutsideLightsOn'):
  outsideLightsOn()
 elif (c[0] == 'OutsideLightsOff'):
  outsideLightsOff()
 elif (c[0] == 'fPorchToggle'):
  fPorchToggle()
 elif(c[0] == 'garageToggle'):
  garageToggle()
 elif (c[0] == 'cactusToggle'):
  cactusToggle()
 else:
  lprint(" Weird command = " + str(c))

def showTime():
 lprint("timecheck");

if __name__ == '__main__':
 #Catch Control-C input so it isn't ugly
 signal.signal(signal.SIGINT, signal_handler)
 logging.basicConfig(format='%(levelname)s:%(message)s',level=logging.WARNING)
 #When looking at a log, this will tell me when it is restarted
 lprint (" started")

 env = Environment(on_switch, on_motion, with_cache=False)
 env.start()
 lprint (" Starting device discovery")
 env.discover(seconds=5)
 lprint (" done with discovery")
 garageLightSwitch = env.get_switch('outsidegarage')
 cactusLightSwitch = env.get_switch('cactusspot')
 fPorchLightSwitch = env.get_switch('frontporch')
 # lprint (" testing lights")
 # garageLightSwitch.basicevent.SetBinaryState(BinaryState=1)
 # cactusLightSwitch.basicevent.SetBinaryState(BinaryState=1)
 # fPorchLightSwitch.basicevent.SetBinaryState(BinaryState=1)
 # time.sleep(2)
 # lprint (' garage ' + str(garageLightSwitch.basicevent.GetBinaryState()))
 # lprint (' cactus ' + str(cactusLightSwitch.basicevent.GetBinaryState()))
 # lprint (' front porch ' + str(fPorchLightSwitch.basicevent.GetBinaryState()))
 # garageLightSwitch.basicevent.SetBinaryState(BinaryState=0)
 # cactusLightSwitch.basicevent.SetBinaryState(BinaryState=0)
 # fPorchLightSwitch.basicevent.SetBinaryState(BinaryState=0)
 # lprint (' garage ' + str(garageLightSwitch.basicevent.GetBinaryState()))
 # lprint (' cactus ' + str(cactusLightSwitch.basicevent.GetBinaryState()))
 # lprint (' front porch ' + str(fPorchLightSwitch.basicevent.GetBinaryState()))
 # If I want to see the various attributes of the device
 # lprint (' ' + garageLightSwitch.explain())
 # lprint (" done testing")
 # examine the status of the lights right now and update the database
 lprint (' checking current state of switches')
 garageSwitchChange(garageLightSwitch.get_state())
 cactusSwitchChange(cactusLightSwitch.get_state())
 fPorchSwitchChange(fPorchLightSwitch.get_state())
 
 lprint (" Setting up the callbacks for the devices")
 cactusLightSwitch.register_listener(cactusSwitchChange)
 garageLightSwitch.register_listener(garageSwitchChange)
 fPorchLightSwitch.register_listener(fPorchSwitchChange)
 
 lprint (" Setting up scheduled items")
 #------------------Stuff I schedule to happen -----
 scheditem = Scheduler()
 # every day at 1900, turn the outside lights on
 #scheditem.add_cron_job(outsideLightsOn, hour=19, minute=01)
 # every day at 2200, turn the outside lights off
 #scheditem.add_cron_job(outsideLightsOff, hour=22, minute=0)
 #just an interval timer for putting time in the log
 scheditem.add_interval_job(showTime, minutes=30)
 lprint (" starting scheduler")
 scheditem.start()
 # Create the message queue where commands can be read
 # I just chose an identifier of 13 because the house monitor
 # already took the number 12.
 Cqueue = sysv_ipc.MessageQueue(13, sysv_ipc.IPC_CREAT,mode=0666)
 firstTime = True
 
 lprint (" Going into the processing loop")
 while True:
  try:
   if (firstTime):
    while(True):
     try:
      # commands could have piled up while this was 
      # not running.  Clear them out.
      junk = Cqueue.receive(block=False, type=0)
      print "purging leftover commands", str(junk)
     except sysv_ipc.BusyError:
      break
    firstTime=False
   newCommand = Cqueue.receive(block=False, type=0)
   # type=0 above means suck every message off the
   # queue.  If I used a number above that, I'd
   # have to worry about the type in other ways.
   # note, I'm reserving type 1 messages for 
   # test messages I may send from time to 
   # time.  Type 2 are messages that are
   # sent by the php code in the web interface.
   # Type 3 are from the event handler. This is just like
   # the house monitor code in that respect.
   # I haven't decided on any others yet.
   handleCommand(newCommand)
  except sysv_ipc.BusyError:
   pass # Only means there wasn't anything there

  # Turn control over to the Wemo environment handler for a second so
  # changes happening can be recorded and handled
  env.wait(time=1)
 
 

I cut out the code to update my database since that is very specific to my implementation, and you'll notice down at the bottom, that control of the lights is handled by Sys V messages.  I don't control the lights with a console, everything comes from other processes sending messages.  For example, my web server software sends a message to this process to turn the lights on and off.  Another process sends messages to turn them on and off at certain times of the day.  That kind of thing.

Just cut that code out and insert whatever kind of control you want to use.  I also made a change to the ouimeaux python library to support asyncronous operation.  The library as it was constructed hung and waited for changes to happen.  I modified it a bit to return after a env.wait call; that's what the very last line in the code is doing; calling the wait for a second then getting back control so it can do other things.

There's a ton of things that can be done to make my control code better, but for now, it works.  I'll loop back to this project some time and add more versatility.  Maybe when the ouimeaux library is changed.