Sunday, October 12, 2014

Using an 'rc' File in My System

First though, I've been away from my house control system for a while because of other projects, and I decided to do some major updates to portions of it.  Over the months, I've learned a lot and some of that new knowledge needs to be put into controlling the house.  So, yes, I'm posting more often.

Often times, I want to post a new update to one of the processes, but I have keys and other stuff in the modules that have to be removed before I can put it on the web.  I'm also going to move the code into GitHub at some point so folk don't have to copy and paste things for their own use.  They can just grab what they want from there.  That means I have to get the keys out of the code.

Unix has long had a customary way of doing things like this, an rc file.  The 'rc' stands for run command and has a long history, look it up.  There's a lot of files out that go into the user's home directory and have a name like .bashrc, .cshrc, etc.  These are configuration commands and aliases.  This is just the tool I need to use.  I can put an rc file in the home directory and all the keys, database names, etc can go in there to be read at runtime by the applications.  Cool, that way I can make something public without having to worry about somebody updating my data feeds by mistake.

But, I hate parsing data out of a file.  I was looking at parsers and ways of storing data when I got an email about decoding a JSON file.  While I was answering the mail I realized how easy it is in python to parse through JSON, so you guessed it, my rc file is JSON.  That means that it's a tiny bit harder to create and incredibly easy to use in a program.  Here's my .houserc file (with the secrets removed):

{
"database":"/home/pi/database/database",

"xbeeport": "/dev/serial/by-id/usb-FTDI_FT232R_USB_UART_A901QLZ1-if00-port0",
"zigbeeport":"/dev/serial/by-id/usb-FTDI_FT232R_USB_UART_A901QQ2F-if00-port0",

"oldxively":{
"key":"secretkeythingieforlegacyxivelygoeshere",
"feed":"1234"},

"xively":{
"key":"secretkeythingieforthenewinterfacehere",
"feed":"1234567890"},

"emoncms":{
"key":"theyuseashorterkey"},

"grovestreams":{
"org":"grovestream-has-even-stranger-one",
"apiKey":"with-a-strange-key-as-well"},

"thingspeak":{
"key":"ASIMPLEKEYEASYTOUSE"},

"oreo":"double stuff"
}

It's just a simple JSON string that can be easily taken apart.  By using meaningful (to me anyway) names, it's pretty easy to read and edit as well.  To get the data back out in a python program, all you have to do is:

def getHouseValues():
 json_data=open("/home/pi/.houserc").read()
 return json.loads(json_data)
hv = getHouseValues()
FEED_ID = hv["oldxively"]["feed"]
API_KEY = hv["oldxively"]["key"]
# the database where I'm storing stuff
cookie = hv["oreo"]

And, since I don't want to type the directory name over and over again, I put the getHouseValues() in a python file and imported it.

No parsing, no silly obscure things to learn, just things I already had to work with doing this project.  Admittedly, it was a pain going to each module and modifying it to use the rc file, but it was worth it.  Notice I included the weird USB ports I use for the two XBees I have attached?  I can do that with other things that may get changed from time to time.  Just add them to the JSON string and I'll be able to get them back at run-time without changing code in any module.

Heck, it's not even hard to read.

Monday, October 6, 2014

My Tiny Timer Class

Just yesterday I posted about creating a new way of controlling and monitoring my Wemo light switches.  In the post I talked about having to develop a way of timing operations.  It was pretty simple and has worked well overnight, so I decided to put it in a python class and use it in the other parts of my system.

So you understand, I hate classes.  When c++ came around, and object oriented programming was all the rage, I resisted.  No, it wasn't that I couldn't accept change, it was that the crap they produced was incredibly hard to read.  Four pages of class definitions and sixteen lines of code may have been cool, but just try and debug it.  Especially with the tools available back then.  Well, it's been some years and objects have matured as well as the folk that created the mess back then.  Now, they're not as big a mess, maybe I should join the 21st century.  Sure, I use classes all the time; I just avoid writing them if at all possible; this is an exception.

So, here's my basic timer class.  Simple right?  Remember, the reason I did this was to keep control within a single thread thus avoiding deadlocks in code and locked up databases.

import time

class timer:
 _things = []
 
 def __init__(self, callback, seconds=1, minutes=0, hours=0):
  interval = (hours*60*60) + (minutes*60) + seconds
  actionTime = time.time() + interval
  self._things.append({"callback":callback,"interval":interval,"actionTime":actionTime})

 def tick(self):
  now = time.time()
  for i in self._things:
   if i["callback"] == None:
    continue
   if now >= i["actionTime"]:
    i["callback"]()
    i["actionTime"] += i["interval"]

checkTimer = timer(None)

''' This is code to test and illustrate the usage '''
def printSecond():
 print "second"
 
def printTwoSeconds():
 print "twoseconds"

def printMinute():
 print "minute"

if __name__ == "__main__":
 # First create any timers you need
 oneSecond = timer(printSecond, seconds=1)
 twoSeconds = timer(printTwoSeconds, seconds=2)
 minute = timer(printMinute, minutes=1)
 
 # now once in a while call tick to let them happen
 while True:
  checkTimer.tick()
  # a sleep lets the cpu have a rest to do other things.
  time.sleep(0.5)
It just keeps a list of the various timers I create to do things and steps through them to see if something needs to be done.  I included an instance by default so, when I use it, I don't have to create one.  It's easy to use and I may be extending it to handle other things that may need to be timed differently like: Every day at 3:00pm, feed the fish.  That takes more work parsing various parameters and such, but it's easy to extend.

Within the module is an example of how to use it.  This can be directly run under python, to see how it works, and how to use it.

I've already modified my module for the Wemo switches to use it, now I'm looking at the other modules to see if this would be appropriate there as well.  

Sunday, October 5, 2014

Controlling Wemo Light Switches, Another Look

No, I'm not dead, and yes, I still work on electronic projects.  I've been busy with projects that are much more mundane and normal.  Things like trying to build a pad for my motorcycle trailer, fighting with my chlorine generator for the pool, and recovering from my house flooding during the monsoon rains here in AZ.  Yep, I had water come in one side of the house and go out the other.  I'm still working on the damage from that episode.

But people don't read my posts to hear about that kind of thing, they probably have worse problems around the house than I do ... I don't get snow or tornadoes.

For months now I've had Wemo light switches running.  They're a nice little device that Belkin puts out and seem to be gaining in popularity.  Heck, Home Depot sells them in the electrical section now.  However, they rely on an app on the cell phone and a cloud server.  That may not be a concern for most people, but I personally hate having things I depend on subject to the whim of a company that might just decide to start charging me for the service in the future.  That's why I put together some software to run the switches from my Raspberry Pi based house controller <link><link><link>.

However the library and toolset I used, while a great effort and really classy, was too complex and kept failing.  Let me be perfectly clear, the ouimeaux library I used was good, the problem is that the wemo switches don't always work like Universal Plug and Play (upnp) devices are supposed to.  I don't want to get into a lengthy description of upnp here, there are a bunch of those out there, but suffice it to say that the subscription capabilities of the switches fails.

So, you discover the devices, set up a subscription where the devices let you know when the state changes happen, and then sometime later, the switch quits responding.  That means sometime or other, the light doesn't turn on or off and you have to restart the process that controls them to get the switch working again.  Additionally, the ouimeaux library uses a LOT of other tools and provides features that I'm just not interested in.  It uses gevent to handle asynchronous operations, has a little web server, client software, enough stuff for a major project.  I just want to turn them on and off and look to see which way they are now.  Using something large like this to do a simple job leads to huge problems when something fails.  There's just so much there that you can go nuts chasing a problem.

You guessed it, I wrote my own.  Well, that's actually an exaggeration.  I stole some stuff and used it would be closer to the truth.  I started out by writing a handler to discover upnp devices on my network.  If you've never done that, I recommend you look into it just for fun.  I discovered my satellite receiver, my Roku, the DSL modem, all the Wemo switches, and several other devices that I didn't know had plug and play interfaces.  That was a royal pain.  I had to learn about SSDP, SOAP, multicast, XML, and the list was growing.  Shoot, all I want to do was control the darn switches, not create a dissertation on half the jargon on the web.  I did learn a lot about discovering the devices on my network, but that wasn't my goal.

That led me to a few days of searching on the web for upnp libraries.  They're out there, and some of the are really good, but they're written in something besides python.  Sure a couple  have interfaces, but the documentation went totally over my head.  There were a some in python, but a couple had disappeared off the web, and one just didn't work (the samples even had syntax errors).  What should I do?  I remembered reading somewhere that python can import modules, and that the python tool Miranda worked reasonably well to experiment with upnp devices ... hmmm.

I got the latest Miranda and combined it with my own code and came up with a way to read and control the switches that uses only the basic python libraries.  It was a challenge hooking into someone else's code that was designed to run as a standalone tool, but it worked.  That way I leveraged the huge amount of work the author of Miranda did to make my job easier.  The big bonus was that Miranda is also really good as a upnp learning tool.  Ha, I have a debugger and a controller all in one.

Since I had spent a lot of time debugging the switch operation to determine why they would quit, I didn't want to use the subscription service.  The subscription facility is asynchronous, so the switch will send an XML message over UDP to a registered URL any time the state changes (off to on); too bad it doesn't work all the time.  That means that I would have to poll the switches to see if something or someone other than my code on the controller had changed their state.  Let me elaborate on that a bit.  These are wall switches; I can walk over and push the button and turn off the lights.  Since my software can't see that, it has to ask the switch what state it is in right now.

That's fine, I can poll it every couple of seconds and see what is happening, BUT there's a problem with that.  I love APScheduler a great timer tool in python; I have it in almost everything I do, but it uses threads to control when things happen.  Using threads and database updates is a path to disaster.  When I tried my new code out, about half the time it would leave the database locked because something jerked execution away at an inopportune time.  I even updated my code and libraries to the latest version of APScheduler which allows jobs to be paused, and still had the problem.  That meant that I had to figure out how to schedule polling of the switches within a single thread.

After a few magnificent failures I came up with a pretty simple way to do it; my method is in the code below.  I may have to use this in some of the previous work I've done, it's really small and simple, so it could decrease the load on the poor little Pi.

Even though this took an inordinate amount of time and study, it's still pretty darn nice.  I haven't ran it for months, but it's held up for a short time pretty well.  I could well develop problems with sockets hanging or something, but the code is simple and straight forward, so I should be able to isolate whatever problems turn up and fix them.  I don't have eight libraries of very complex code written by someone else to step through trying to find out how something works.  It's me and Miranda here, ... well there are the system libraries, but nothing terribly exotic.

Here's the code.  It's written in my usual style: variable names that are too darn long, comments for practically every darn line, over indentation, etc.  Basically everything I need to pick it up a year from now to fix a bug or add a feature.

#! /usr/bin/python
from miranda import upnp
from miranda import msearch
from miranda import set
import sys
#from apscheduler.schedulers.blocking import BlockingScheduler
import datetime
from datetime import timedelta
from datetime import datetime
import time
import sysv_ipc
import logging
import sqlite3
import pdb #yes, I had trouble and had to use this !!
#-------------------------------------------------
# the database where I'm storing stuff
DATABASE='/home/pi/database/desert-home'

def lprint(text):
 print time.strftime("%A, %B, %d at %H:%M:%S"),text
 sys.stdout.flush()
 
def _send(action, whichone, args):
 if not args:
  args = {}
 entry = (item for item in lightSwitches if item["name"] == whichone).next()
 index =entry['index']
 host_info = conn.ENUM_HOSTS[index]
 device_name = 'lightswitch'
 service_name = 'basicevent'
 controlURL = host_info['proto'] + host_info['name']
 controlURL2 = host_info['deviceList'][device_name]['services'][service_name]['controlURL']
 if not controlURL.endswith('/') and not controlURL2.startswith('/'):
  controlURL += '/'
 controlURL += controlURL2

 resp = conn.sendSOAP(
  host_info['name'],
  'urn:Belkin:service:basicevent:1',
  controlURL,
  action,
  args
 )
 return resp
 
def get(whichone):
 ''' 
 Returns True if the light is on and False if not
 '''
 resp = _send('GetBinaryState', whichone, {})
 tagValue = conn.extractSingleTag(resp, 'BinaryState')
 return 'On' if tagValue == '1' else 'Off'

def handleUpdate(whichone, status):
 for i in lightSwitches:
  if i['name'] == whichone:
   i['status'] = status
 updateDatabase(whichone, status)

def on(whichone):
 """
 BinaryState is set to 'Error' in the case that it was already on.
 """
 resp = _send('SetBinaryState', whichone, {'BinaryState': (1, 'Boolean')})
 tagValue = conn.extractSingleTag(resp, 'BinaryState')
 status = 'On' if tagValue in ['1', 'Error'] else 'Off'
 handleUpdate(whichone, status)
 lprint("turned %s on"%(whichone))
 return status

def off(whichone):
 """
 BinaryState is set to 'Error' in the case that it was already off.
 """
 resp = _send('SetBinaryState', whichone, {'BinaryState': (0, 'Boolean')})
 tagValue = conn.extractSingleTag(resp, 'BinaryState')
 status = 'Off' if tagValue in ['0', 'Error'] else 'On'
 handleUpdate(whichone, status)
 lprint("turned %s off"%(whichone))
 return status

def doLights():
 for switch in lightSwitches:
  thisOne = switch['name']
  updateDatabase(thisOne,get(thisOne))

def doComm():
 global firstTime
 #global scheditem
 
 try:
  if (firstTime):
   while(True):
    try:
     # commands could have piled up while this was 
     # not running. Clear them out.
     junk = Cqueue.receive(block=False, type=0)
     print "purging leftover commands", str(junk)
    except sysv_ipc.BusyError:
     break
   firstTime=False
  while(True):
   newCommand = Cqueue.receive(block=False, type=0)
   # type=0 above means suck every message off the
   # queue.  If I used a number above that, I'd
   # have to worry about the type in other ways.
   # note, I'm reserving type 1 messages for 
   # test messages I may send from time to 
   # time.  Type 2 are messages that are
   # sent by the php code in the web interface.
   # Type 3 are from the event handler. This is just like
   # the house monitor code in that respect.
   # I haven't decided on any others yet.
   handleCommand(newCommand)
 except sysv_ipc.BusyError:
  pass # Only means there wasn't anything there 


def handleCommand(command):
 #lprint(" " + str(command))
 # the command comes in from php as something like
 # ('s:17:"AcidPump, pumpOff";', 2)
 # so command[0] is 's:17:"AcidPump, pumpOff'
 # then split it at the "  and take the second item
 try:
  c = str(command[0].split('\"')[1]).split(',')
 except IndexError:
  c = str(command[0]).split(' ')   #this is for something I sent from another process
 #lprint(c)
 if (c[0] == 'OutsideLightsOn'):
  outsideLightsOn()
 elif (c[0] == 'OutsideLightsOff'):
  outsideLightsOff()
 elif (c[0] == 'fPorchToggle'):
  toggle("frontporch")
 elif(c[0] == 'garageToggle'):
  toggle("outsidegarage")
 elif (c[0] == 'cactusToggle'):
  toggle("cactusspot")
 elif (c[0] == 'patioToggle'):
  toggle("patio")
 else:
  lprint(" Weird command = " + str(c))

def outsideLightsOn():
 lprint (" Outside lights on")
 on("outsidegarage")
 on("frontporch")
 on("cactusspot")
 
def outsideLightsOff():
 lprint (" Outside lights off")
 off("outsidegarage")
 off("frontporch")
 off("cactusspot")

def toggle(whichOne):
 if (get(whichOne) == 'On'):
  off(whichOne)
 else:
  on(whichOne)
  
def keepAlive():
 '''
 For my own purposes, I update the database periodically with the time
 so I can check to see if things are holding together.  I currently use the
 time in the light switch records for this.
 '''
 lprint(" keep alive")
 for switch in lightSwitches:
  thisOne = switch['name']
  updateDatabase(thisOne, get(thisOne), force=True)

def updateDatabase(whichone, status, force=False):
 ''' 
 This is running on a Pi and is not event driven, so polling like
 this will result in considerable wear to the SD card.  So, I'm going to 
 read the database to see if it needs to be changed before I change it.  
 According to everything I've read, reads are free, it's the writes that
 eat up the card.
 '''
 dbconn = sqlite3.connect(DATABASE)
 c = dbconn.cursor()
 c.execute("select status from lights where name = ?;",
  (whichone,))
 oldstatus = c.fetchone()
 if oldstatus[0] != status or force == True:
  lprint ("Had to update database %s, %s"%(whichone, status))
  c.execute("update lights " 
   "set status = ?, utime = ? where name = ?;",
   (status, time.strftime("%A, %B, %d at %H:%M:%S"), whichone))
  dbconn.commit()
 dbconn.close()
  
if __name__ == "__main__":
 #When looking at a log, this will tell me when it is restarted
 lprint (" started")
 firstTime = True
 debug = False
 if not debug:
  conn = upnp(False,False,None,0)
  ''' 
  I don't want the search for devices to run forever 
  So, I set the timeout for miranda to some number of seconds
  to limit it.
  '''
  set(3, ["set","timeout", "10"], conn)
  ''' 
  This looks at the devices that responded and gathers more data about
  them by sending them a request to itemize their capabilities.

  Sometimes a upnp device goes nuts and responds way out of
  proportion.  You can get the same device in the tables
  many times, so set the uniq to True
  
  Also, the Wemo switches don't always respond to a discover specific to 
  them.  That means I have to do a general discover and get all the devices
  on the network.  This sucks because it slows things down, so if anyone
  overcomes this problem, let me know how.
  '''
  set(3, ["set","uniq", True], conn)
  ''' This is the actual search '''
  msearch(1,[msearch],conn)
  ''' and now do the interaction '''
  for index, hostInfo in conn.ENUM_HOSTS.iteritems():
   #print "************** ", index, " of ", len(conn.ENUM_HOSTS) - 1
   ''' on my network, I have a rogue device that reports badly '''
   if hostInfo['name'].find('192.168.16.254') == 0:
    print "Odd device, ignoring"
    continue
   ''' if you want to see them as they come in, uncomment this '''
   #print hostInfo
   if hostInfo['dataComplete'] == False:
    xmlHeaders, xmlData = conn.getXML(hostInfo['xmlFile'])
    conn.getHostInfo(xmlData,xmlHeaders,index)
    
  ''' 
  now to select only the light switches from the various devices 
  that responded 
  '''
  lightSwitches=[]
  for index, host_info in conn.ENUM_HOSTS.iteritems():
   if "deviceList" in host_info:
    if "lightswitch" in host_info["deviceList"]:
     name = host_info["deviceList"]["lightswitch"]["friendlyName"]
     lightSwitches.append({"name": name, "index": index, "status" : 'unknown'})
  ''' 
  OK, now I have the list of Wemo light switches that are around the 
  house, so print it and show the state of each one 
  '''
  print "this is the list of the", len(lightSwitches), "Wemo switches found."
  for switch in lightSwitches:
   switch['status'] = get(switch['name'])
   print switch
   
 # Create the message queue where commands can be read
 # I just chose an identifier of 13 because the house monitor
 # already took the number 12.
 Cqueue = sysv_ipc.MessageQueue(13, sysv_ipc.IPC_CREAT,mode=0666)
 '''
 This is a poor man's timer for task control.  I may put this in a class
 after I've run it for a while.  The reason I did it this way is that 
 APSscheduler creates a separate thread to handle timers and I don't 
 want the contention to the database of separate threads doing things
 that way.
 
 To use it, just put another entry into the table.
 '''
 lprint (" Setting up timed items")
 startTime = int(time.time())
 tasks = [{'whichOne': doLights,'interval' : 2, 'next': startTime+10},
  {'whichOne': keepAlive, 'interval' : (4*60), 'next' : startTime+15}]
 lprint ("going into the processing loop")
 print startTime
 while True:
  
  #pdb.set_trace()
  now = int(time.time())
  for task in tasks:
   if task['next'] <= now:
    task['next'] = now + task['interval']
    #print task['whichOne']
    task['whichOne']()
  doComm()
  ''' 
  doing a sleep here releases the cpu for longer than the program runs
  That way I reduce the load on the machine so it can do more stuff
  '''
  time.sleep(0.25) 
  pass 
 
 sys.exit("done");

It's only fair to thank Ian McCracken who wrote the Ouimeaux library, Isaac Kelly who did much of the initial discovery work, and Craig Heffner the author of Miranda.  My work is built on theirs.

No, I don't have support in here for the Wemo motion sensor, crock pot, or other items they may come up with.  If you want to grab it and extend it to support those devices, feel free; just let me know so I can get the improvements as well.

There's also support in here for the sysv messages I use to communicate between processes on the machine.  Feel free to rip that out and put in anything you want.  I wanted to give you an example that I'm actually running right now, not something I cut up to look pretty.

Just for fun, here's the output of the code that is captured to my log:

Sunday, October, 05 at 16:22:06  started
action timeout
action uniq
Show unique hosts set to: True
argc 1
argv [<function msearch at 0xb69d94f0>]
Entering discovery mode for 'upnp:rootdevice', Ctl+C to stop...

Error updating command completer structure; some command completion features mig
ht not work...
****************************************************************
SSDP reply message from 192.168.0.34:49312
XML file is located at http://192.168.0.34:49312/device.xml
Device is running Linux/2.6.37.6-4.0, UPnP/1.0, Portable SDK for UPnP devices/1.
6.17
****************************************************************

Error updating command completer structure; some command completion features mig
ht not work...
****************************************************************
SSDP reply message from 192.168.0.34:49200
XML file is located at http://192.168.0.34:49200/device.xml
Device is running Linux/2.6.37.6-4.0, UPnP/1.0, Portable SDK for UPnP devices/1.
6.17
****************************************************************

Error updating command completer structure; some command completion features mig
ht not work...
****************************************************************
SSDP reply message from 192.168.0.28:49154
XML file is located at http://192.168.0.28:49154/setup.xml
Device is running Unspecified, UPnP/1.0, Unspecified
****************************************************************

Error updating command completer structure; some command completion features mig
ht not work...
****************************************************************
SSDP reply message from 192.168.0.29:49153
XML file is located at http://192.168.0.29:49153/setup.xml
Device is running Unspecified, UPnP/1.0, Unspecified
****************************************************************

Error updating command completer structure; some command completion features mig
ht not work...
****************************************************************
SSDP reply message from 192.168.0.43:49154
XML file is located at http://192.168.0.43:49154/setup.xml
Device is running Unspecified, UPnP/1.0, Unspecified
****************************************************************

Error updating command completer structure; some command completion features mig
ht not work...
****************************************************************
SSDP reply message from 192.168.0.26:49153
XML file is located at http://192.168.0.26:49153/setup.xml
Device is running Unspecified, UPnP/1.0, Unspecified
****************************************************************

Error updating command completer structure; some command completion features mig
ht not work...
****************************************************************
SSDP reply message from 192.168.0.22:2869
XML file is located at http://192.168.0.22:2869/upnphost/udhisapi.dll?content=uu
id:3a682b82-e803-4105-bfeb-114ed775cab1
Device is running Microsoft-Windows/6.3 UPnP/1.0 UPnP-Device-Host/1.0
****************************************************************

Error updating command completer structure; some command completion features mig
ht not work...
****************************************************************
SSDP reply message from 192.168.0.14:8060
XML file is located at http://192.168.0.14:8060/
Device is running Roku UPnP/1.0 MiniUPnPd/1.4
****************************************************************

Error updating command completer structure; some command completion features mig
ht not work...
****************************************************************
SSDP reply message from 192.168.0.1:80
XML file is located at http://192.168.0.1:80/DeviceDescription.xml
Device is running ZyXEL-RomPlug/4.51 UPnP/1.0 IGD/1.00
****************************************************************


Discover mode halted...
Failed to get argument name for OpenInstaAP
Failed to get argument name for CloseInstaAP
Failed to get argument name for OpenInstaAP
Failed to get argument name for CloseInstaAP
Failed to get argument name for OpenInstaAP
Failed to get argument name for CloseInstaAP
Failed to get argument name for OpenInstaAP
Failed to get argument name for CloseInstaAP
Caught exception while parsing device service list: list index out of range
At index  7
Caught exception while parsing device service list: list index out of range
At index  8
this is the list of the 4 Wemo switches found.
{'status': 'Off', 'index': 2, 'name': 'cactusspot'}
{'status': 'Off', 'index': 3, 'name': 'frontporch'}
{'status': 'Off', 'index': 4, 'name': 'patio'}
{'status': 'Off', 'index': 5, 'name': 'outsidegarage'}
Sunday, October, 05 at 16:22:42  Setting up timed items
Sunday, October, 05 at 16:22:42 going into the processing loop
1412551362
Sunday, October, 05 at 16:22:56 Had to update database patio, On
Sunday, October, 05 at 16:22:56 turned patio on
Sunday, October, 05 at 16:22:57  keep alive
Sunday, October, 05 at 16:22:57 Had to update database cactusspot, Off
Sunday, October, 05 at 16:22:57 Had to update database frontporch, Off
Sunday, October, 05 at 16:22:57 Had to update database patio, On
Sunday, October, 05 at 16:22:58 Had to update database outsidegarage, Off
Sunday, October, 05 at 16:23:04 Had to update database patio, Off
Sunday, October, 05 at 16:23:04 turned patio off
Sunday, October, 05 at 16:23:12  Outside lights on
Sunday, October, 05 at 16:23:12 Had to update database outsidegarage, On
Sunday, October, 05 at 16:23:12 turned outsidegarage on
Sunday, October, 05 at 16:23:12 Had to update database frontporch, On
Sunday, October, 05 at 16:23:12 turned frontporch on
Sunday, October, 05 at 16:23:12 Had to update database cactusspot, On
Sunday, October, 05 at 16:23:12 turned cactusspot on
Sunday, October, 05 at 16:23:17  Outside lights off
Sunday, October, 05 at 16:23:17 Had to update database outsidegarage, Off
Sunday, October, 05 at 16:23:17 turned outsidegarage off
Sunday, October, 05 at 16:23:17 Had to update database frontporch, Off
Sunday, October, 05 at 16:23:17 turned frontporch off
Sunday, October, 05 at 16:23:17 Had to update database cactusspot, Off
Sunday, October, 05 at 16:23:17 turned cactusspot off
At the top is where I set some parameters in Miranda to control what I discover.  Then the discovery process prints out a summary of the devices it finds. The errors there are caused by the Wemo switches not responding properly.  Next are messages from the conversation with the switches.  Once again, the switches don't behave.  Finally the last step of initialization where the status and names of the switches show up.  The rest of the log is normal operation.  I poll the switches every two seconds to see if they're on or off. and every four minutes to make sure the database is kept current.  I use database currency to monitor the health of the controller.

If you have these switches, or are thinking about using them, grab the code and start playing.

Monday, August 4, 2014

Interaction of Appliances

One of my readers (thanks Andreas) commented about how controlling the appliances could help me.  For example, not letting the compressors of the refrigerator and freezer go on at the same time could lower my peak demand.  I answered that I was a bit leery about that because it could lead to spoiled food.  But, I decided an exercise in looking at it wouldn't take too much time and maybe it would tell me if such a thing would save me money.

For those of you just stumbling upon this site, I have demand billing from my wonderful power company.  There's a long discussion of this here <link>, but suffice it to say that from noon to 7PM I carefully control the usage of power around the house to reduce the power bill.  A single mistake that lasts for 15 minutes during this period could cost me a LOT of money.  Enough to buy several little computers to control things around the house.

So, I looked at adding the usages and creating a new graph to show the total use from my two freezers and refrigerator over a period of time, but had a brainstorm along the way.  I could simply use HighCharts ability to stack graphs.  It worked, and now I have a better understanding of my usage that I want to illustrate for people out there that are considering (or doing) the same kind of thing I do.

First though, a recap of the appliances and what their power usage profile is.  Here's my refrigerator graph as an area chart:


This appliance runs for short bursts to cool down to around 38-40 degrees.  It gets the most activity, but is pretty efficient.  Now, my freezer:


This thing runs a lot.  It has a good compressor that doesn't use much power, but it runs a lot more than I expected.  Still, it doesn't actually chew up much power.  And last, the garage freezer:


This thing runs about half the time during the summer because the garage is hot.  I don't cool the garage and it's on the south side.  

The little spikes in the charts are an artifact of the way I graph the data.  Yes, I could have hunted down the problem and fixed it, but I wanted to study the data, not spend a couple of hours chasing down a bug and fixing it.  So, just ignore the spikes, they only mean I'm lazy.

And finally, the stacked composite graph:


Spikes aside, this shows that my peak usage for all three devices is less than 500 watts.  Sure, it would be more energy efficient to stop one to allow a different one to run, but it wouldn't change my peak usage much.  The way they (the power company) company calculate this is to take a moving average of 15 minutes over the entire peak period for a month and then they bill me for the highest period.  So, any 500 watt period will cost me for the entire month.  I once messed up and used a couple of kW, so I had the freedom to really eat the power for the rest of the month.  Interesting result of peak demand billing, I wonder if they realize it.

At any rate, it doesn't look like I have to worry much about the combination of appliances getting my usage out of control, but it could be useful to keep this in mind if I add an appliance, or need to modify my lifestyle in the future.  Most people don't want to put up with their A/C units being shut off during the hottest part of the day, or their pool motor not running in the afternoon or early evening when they want to use it.  Fortunately, when relatives visit me, they understand my OCD about keeping appliances shut off, and just giggle about me when they get home, so I don't have to worry about them.

See folk, I actually take your suggestions seriously and even act on them from time to time.  It may take months, but I eventually get there.  Now I still have to think about getting a temperature sensor inside the freezer (thanks badhairday).

Monday, July 28, 2014

Controlling the Hayward Ecostar Pump

Quite some time ago I got fed up with the huge amount of power a pool pump uses and got one of those cool permanent magnet, variable speed pumps.  It's really is cool, it can run from a few RPM up to way over 3000.  All of the speeds use less power than a conventional motor and you can actually circulate water with 200 watts of power usage.  This is what they look like:

There's three pieces to these things, the controller, the actual motor and the filter-impeller.  Hayward calls the controller the 'drive'; I don't have any idea why except maybe because they make pool controllers and they don't want customers to get confused.  The drive is this part here:


Yes, I stole the image off the web.  It was too hot to go out and take my own picture.  The drive can be separated from the motor and remotely set up.  This makes it convenient for ease of use, and nice if you have the rest of the stuff inside an enclosure.

The drive has its own controls, but it can also be controlled by my Goldline P4 controller.  The problem is that the darn controller only supports two speeds: low and high.  Sure, I can set the speed for each of them to anything I want, but I only have two speeds for my variable speed motor.  That has always ticked me off.  Hayward decided that I only get two speeds for the motor that are easily done.  To do anything else I have to futz around with settings each time.  Really ??

A few weeks ago one of my readers asked if I had ever pursued breaking the protocol between my controller and the drive.  I hadn't, but it sounded like a cool project to take on.  So, armed with my motor, his motor, his skills, my lack of skills, we embarked upon a project to get control of this motor.  Naturally, there is zero documentation on the protocol, but we had some hints.

It's RS485 half duplex because it works on the same line as the other controls for the P4.  It's 19.2Kbaud for the same reason.  That meant that we could set up monitors and code to listen to the conversations between the devices as we changed things and might stand a chance to get a hint on how to control it.

Guess what?  After a few false starts, we beat it.  We can now control the motor and change the speed to anything we want to.  It turns out the protocol is relatively simple; you send a command to the motor periodically telling it the speed you want it to run.  If you stop sending, the motor stops.  If you send a speed of zero, the motor stops.  If you send a specific speed, the motor runs at that speed.

The motor protocol works in percentage.  If you send 100%, it will run at the top speed.  If you send 0%, it stops.  You can choose any percentage in between and the motor will go to that speed.  You can't send 1000 RPM, you have to send a percentage instead.  Still, it works quite nicely.  The protocol looks like this:

Off     0x10, 0x02, 0x0C, 0x01, 0x00, 0x00, 0x00, 0x1F, 0x10, 0x03

100% 0x10, 0x02, 0x0C, 0x01, 0x00, 0x64, 0x00, 0x83, 0x10, 0x03

45%   0x10, 0x02, 0x0C, 0x01, 0x00, 0x2D, 0x00, 0x4C, 0x10, 0x03

The 0x10, 0x02 is the start of packet indicator.  The 0x0C, 0x01 means from device 1 to device 12.  The 0x00 is just a zero, never saw it become anything else.  the 0x00 or 0x64 or 0x2D is the percentage in hex (0, 100, 45) the next two bytes are the checksum and the 0x10, 0x03 is the end of packet indicator.  So, to change the speed, change the percentage, recalculate the checksum and send it.  The motor will change speed.  You have to send this at half second intervals to keep the motor running.  Actually, I've gone up to a second and a half and still had the motor run, but better safe than sorry.

The checksum is a matter of adding the byte values and sticking them in place.  In the first packet above, 10 + 2 + C + 1 + zeros = 1F.  Remember, it's hexadecimal and the checksum is two bytes long, so it's 0x00, 0x1F.  Similarly, the 100% packet is 10 + 2 + C + 1 + 0 + 64 = 0x00, 0x83.  The last two bytes, 0x10, 0x03 are not used in the checksum.

Each of these conrol packets is responded to immediately by the drive unit with something like:

0x10 0x02 0x00 0x0C 0x00 0x00 0x2D 0x02 0x36 0x00 0x83 0x10 0x03

It starts off with the begin packet bytes of 0x01, 0x02.  Then 0x00 , 0x0c which would mean, from device 12 to device 0.  Strange that it would reply to device zero when device 1 sent the command, but ??  Byte 6 of the packet is the speed, in the packet above it is byte 6 counting from the first byte being index [0].  Bytes 7 & 8 are the power and they threw me for a loop. They’re in nibble form of the actual speed if you read it out loud. For example:

0x10 0x02 0x00 0x0C 0x00 0x00 0x62 0x17 0x81 0x01 0x18 0x10 0x03

Byte 6 is 0x62, for 98 percent and bytes 7 & 8 are 0x17 0x81, this reads as one, seven, eight, one or a power reading of 1,781 watts. Is that weird or what?   I tried big endian, little endian, hex, some obscure rotations I’ve seen in the past and other stuff until I just happened to read it out loud and then look at the motor. Shocked is probably the correct term for my reaction.

So, to calculate the speed: Serial.print(((long)3450 * buffer[6]) / 100); seems to work just fine. It’s a tiny bit off from the actual motor reading on its LCD sometimes, but I think that’s because of the calculations on the two devices being a little different. The power is right on for every instance I have tried.

The code for something like this was relatively simple to put together, but it gets really complex when you add in whatever method you use to control it.  I have an XBee network that I send commands through, but most folk out there want wireless ethernet or wires.  It can be controlled from a board as simple as an Arduino (like me and my colleague), or any other device that can either provide RS485 or handle an adapter to convert TTL serial.  My setup is described here <link>, I just added code to it to handle the motor separately and turned off control of the motor in the Goldline P4 I have.

The physical connection is with two wires from your RS485 device to the motor drive unit on the communications port.  You may have to use shielded cable and connect the ground up as well; these motors are extremely noisy on both the power line and RF near the unit.  So, grab some of that shielded CAT5 cable and go for it.


I cut this right out of the installation manual for the Ecostar motor.  Pin 7 is the plus (+) side of the RS485 link and they clearly label the comm connection.  This image also shows how the connection is made to a pool controller like my P4.

If you do this to control your motor, there is something that will help you.  When you send something to the motor that it can't understand, it locks up the drive unit.  I sent many a packet that was constructed incorrectly and the drive would go nuts and not respond after that.  You have to actually power down the motor and then turn it back on to continue.  Not the most robust software in the world.

There is another protocol here that I will research in more depth later.  Not only is the 'drive' controlled by the pool controller, the motor itself is controlled by the 'drive'.  Yes, the motor itself has a protocol all it's own that the drive supplies.  That's why you can separate the drive from the motor and mount it somewhere else with some wires running between.  This is also RS485 and it runs at 1200 baud.  It has a different checksum scheme and there are bits in there that have to be figured out.  We haven't made much progress on this protocol, but we can make the motor turn on without the drive unit even being there.  This may come in handy at some point in the future when the drive dies and the warranty has expired.  Those drives cost almost as much as the entire assembly ... jerks.

There you go folk, you now have control of that expensive motor Hayward has been keeping secret. Plus, for people like me, you can monitor the power usage real time to keep energy costs down.  You won't be limited by some arrangement of relays or the two speeds Hayward saw fit to allow.

That's two of their secret protocols I've put on the web now <link>; maybe I should think about hiding for a while...

Wednesday, July 23, 2014

Arduino, XBee, and a Freezer Defroster

I have an upright freezer.  Living a ways from town and the climate make this necessary.  I load a cooler in the car, go to the store get my frozen goods and some ice, pack the food in the cooler with the ice and come home.  When it's over 114F outside, you have to take some special precautions to keep the food from becoming a puddle in the back of the car.

A while back I discusssed how my freezer uses power; it's relatively efficient except for one item, the defroster <link>.  A freezer's defroster is a combination of a timer, a heater, and a thermostat to monitor the temperature of the evaporator.  What bothered me was the timing of the defrost cycle.  Every 11 hours or so, it would defrost, and this meant that the heater would be on during peak usage period.  Since I pay a lot for peak usage it would be nice to have better control of the timing of the defrost cycle.  So, an examination of the defrost circuitry showed a clear opportunity for an accurate timer and a simple SPDT relay as a replacement for the conventional method.

So, since I had a couple of arduinos that weren't doing anything, I got one of these:
This shield has both the relays and the XBee socket I wanted to use, perfect.  I have a few XBees that I picked up, so I configured one and plugged it in.  A little code later I was reading my house clock transmissions and setting off a timer every 12 hours.  I chose 08:15 and 20:15, times that are outside the peak period, now all I had to do was wire it in and test it.  Here's an image of the schematic of the circuitry:


I circled the defrost timer to make it obvious.  Notice that it simply switches between the compressor circuitry and the defrost heating assembly.  This makes it simple to wire into the circuitry, so I took out the timer and used the plug that went to it to wire into the relay of the arduino shield.  It was ready to test, and I hooked up a wall wart power supply and plugged the arduino into the same power monitor that I use on the freezer.  It worked like a charm.  Now my freezer goes into a defrost cycle at the programmed times and runs for 30 minutes.  I checked the evaporator pretty often to make sure it was running the heater long enough and everything seems fine.

While I was programming the device I threw in some code to allow me to set off the defrost cycle any time I want to as well as having it report through the XBee the status of the defroster.  This leaves a clear opportunity for installing a temperature sensor, compressor sensor, door sensor, etc.  Over time I may well do some of these things.  I could go nuts and use the arduino to control the entire freezer; these appliances are relatively simple devices and a little arduino and some relays could take over the entire operation.  I'm not sure there's any real point to that since it works well already, but I may get bored some hot summer day.

A temperature sensor would be pretty nice though.  I could send the temp to my house controller <link> and set up a piece of code to check for the temperature getting too high.  A too-high situation could easily send me mail or light up an LED somewhere to alert me to a problem.  Or both.   

Here's a chart of my freezer power usage over a 24 hour period:


The freezer is the black line.  Notice the two humps, one at 08:15 and the other at 20:15?  That's the increased power usage from the heating units (one on the evaporator and the other on the drain tube).  Now I have this power using mode scheduled for the lower rate periods.  With the new LED lights I installed in the freezer to replace the incandescent ones, this device is getting cheaper to run all the time.

Before you tell me that the wall wart and arduino probably use 5 watts continuously, remember my goal was to move the higher usage away from peak billing periods.  I'd rather have 5 watts continuous than 400 watts for thirty minutes during peak.  Peak usage is really expensive in my area.

No, it's not a major savings, but every little bit helps.  Heck, I'll probably get back the money I spent on the shield and arduino in ... what ... 10 years or so.

Saturday, July 12, 2014

OpenWrt on the HLK-RM04, Reality Check

There's a ton of pixels out there about OpenWrt.  It's open source router software that can do an excellent job of expanding the capabilities of some routers way beyond what the manufacturers provide.  I had an OpenWrt router for a while, it eventually went up in smoke and I've been using what the DSL provider recommends ever since.  However, I ran into the HLK-RM04 <link> through the suggestion of a reader and decided to look into expanding the capabilities of this little device and OpenWrt seemed like a good choice to use as software for it.

There's a few threads on the OpenWrt forum and a couple of interesting repositories on github that relate to the device and they look reasonably promising,  So, I read the threads, and all of them suggested either picking up a binary they created or building my own.  There were involved instructions on the build, so I thought, "Why not?"  Thus began my misadventures with building OpenWrt for the HLK-RM04.

First, I tried the binary that was created by one of the guys that frequents the OpenWrt forum.  This binary seemed to have great possibilities since, once you have a version of OpenWrt running, you can add to it and update it pretty easily.  So I followed the instruction to install the uboot boot loader and then installed the image that the guy put in github.  This process went pretty well, but very little worked on the router.  I could get an ethernet connection, but wireless and any outside access wouldn't work.  When I tried to enable wireless, the rm04 ran out of memory.  Fine, I'll build my own and learn about the configuration of the software while going through the process.

So, I thought to myself, I really should make sure I could restore the rm04 to factory settings.  Oops, too late.  Once you install uboot on the device, the factory software is gone and can't be recovered.  Even though it's in a different part of the device, there just isn't any way to get to it.  Sure, I could have spent about a month learning the ins and outs of the JTAG interface and bit banged myself a backup, but that didn't seem like a valuable skill to pursue.  There were two reasons I was tardy in looking at a way to back it up.  The first was because there really aren't any useful instructions out there on how to backup.  Sure there's a OpenWrt <link> wiki page on it, but I couldn't get enough information out of it to even attempt anything.  The 'How To' was full of 'could', 'maybe', and 'beware', with no real examples.  Like I said, I'd have to learn the intricacies of JTAG to have a chance.  The second was because there isn't a serial interface to control the chip.  There's a serial line, but you can't log into it and do anything.  I really don't understand why manufacturers do this, do they want to sell these things?

I guess I better make OpenWrt work.

I downloaded OpenWrt, configured it the way the instructions said and started make.  Six hours later, it failed.  Following the instructions, I used the V=s option to see what was failing.  It turns out that OpenWrt installation only installs part of the code, the makefile goes to various servers and their mirrors and downloads more.  The file it failed on was a little over two meg, so I tried it again.  Of course, it failed again.  I set up a little script to restart the make if it died and went to bed.

The next morning, it was still trying to load files.  Yes, I got the two meg and several others, but the files were getting enormous.  I watched it fail five times on an eight meg file and decide this wasn't working, so I hunted down the retry count and timeout amount in the scripts and changed them to 25 retries and a full minute timeout.  Ran it again and it seemed to stumble through some of the downloads.  However, when I watched the log, it was failing quite often with a 404 error on a file that it had been previously reading; what?  So, I changed the retry to 50 to allow for server errors and let it go again.

It chugged along for many hours and then I noticed that the file size was getting larger and larger.  There were files over 90Mb that were trying to be loaded.  The internet, my little machine, and bad luck were really conspiring because at one time I noticed the load speed was two bytes (yes bytes) a second.  Obviously, this wasn't going to work, at least in the time I have left on the planet.

Time to try another tactic.  I saw three possibilities 1) Configure various things out of the binary version I had previously tried until I got the bare minimum and see what I could get running.  2) Download and install a version I ran across that said that they had already stripped it to almost nothing.  3) Try to get the original software from somewhere and put it back, then start off with some other tactic.

I did all three.  I went to the OpenWrt image I had running, but ran out of memory, and started cutting things out.  Problem with this was that when I cut something out that I shouldn't, the little board died and I had to start over cutting things out.  This got tiring, but I got really good at loading software on the board and backing up the OpenWrt config files such that it was only tedious to restore it, not a real pain.  However, I never did get enough out of the distribution of OpenWrt to allow wireless to run without it failing from running out of memory.

So I went to step 2 above and loaded a bare minimum system on the board.  This one held some promise, but I couldn't get past the configuration changes necessary to make the board do something useful.

Step three was now the remaining option.  I dropped a note to the manufacturer asking for the very latest version of the boards software.  I hoped they would send it then I could try loading it on the board with uboot to see if that would get me back to the starting point.  I also dropped a note to a person that had posted that he had the manufacturer's files and had used them.  I fully expected to get no response from either of them, ... but:

The other customer that had the files posted them to github so folk could pick them up there.  So, the last two versions of the board's OS are out there to be tried.  The manufacturer actually answered me and sent a lot of documentation that appears to be correct.  They said they would send me the files as well, but I haven't received them yet.

This was totally unexpected.  Two total strangers responded extremely rapidly and there are possible solutions that I can try.  Gathering up enough cables and wires to make a huge mess, I tried loading the factory files to the board with uboot.  No luck there, I got the horrible message, "Bad Magic Number," and the boot failed.  I tried to look into the magic number, thinking that I could change it and get the file to execute.  No such luck.  One (get this only one) person had fought the battle against this message and posted about his trials <link>.  Turns out that it wasn't just a magic number, it's a 60 byte header that includes compression type, CRC checksum, and a bunch of other stuff I didn't know and couldn't think of a way to find out.

I'm out of options.

It's not as bad as it sounds though.  Remember, I bought this board as a tool that I could use to monitor devices and possibly make changes with.  It was truly compelling because it's relatively small and cheaper than most solutions out there.  Wait, cheap?  I ordered another chip for the board.  Since the board is just interfaces and mounts, a new chip will give me a second chance to try and get everything I need working, and I have loads of experience with it now to keep from making some of the mistakes that got me into this mess.

Back when I started experimenting with the board I managed to make a reasonable 802.11 b/g/n gateway with it.  That was a nice bonus because it meant that I definitely could get some use out of it as a wifi device.  If I can get the serial port to send data over wireless, that would just be icing on the cake.  That's worth another 10-12 bucks investment.  If I'm lucky, the newer software, better documentation, and experience could lead to a nice versatile addition to my network.

The problem with OpenWrt is that it is too versatile.  Over the years many people have contributed to the project and it has grown to the point where one needs a quad processsor running at light speed to build it the first time.  Of course that also means a quantum link to the ethernet or access to the buss of the server.  If you can get it loaded and built, you've actually got something that you can do something with because the ability to update it is really nice.  It's that first time that will test your patience.  When I finally gave up, it was over 2G and still growing; allow for this in your plans.

Now I'm waiting for the new chip and holding on to the updated manufacturer's software.  I have notes on how to do things that shouldn't kill the board and understand a whole lot more about the device.

C'mon mail.