Amazon Dot, Sending Data From Raspberry Pi

Last post <link> I showed you how to create a 'thing' and to get the necessary certificates and keys to enable communication from the Pi up to the Amazon AWS; now we need to look into sending some data up there. I tried the Amazon library that is supplied with the AWSIot, but frankly, it had a bug I couldn't tolerate as well as really pretty, but incomplete documentation. The bug is that you can't run two instances of it on a Pi. If you start up a second instance, the first one starts to fail, and I want to have more than one thing that can send data. The documentation only shows how to use it with Java and Javascript; the python descriptions are just not there. Also, each piece of it leaves things out that you need and I didn't want to spend even more hours experimenting.

The tutorials helped some, but like I said, not a one of them actually worked when I tried them, so I used Nick's example from my last post and built a process that could send data up, then I converted it to use simple mqtt calls and simplified it a whole lot. Nick's example can't receive data from Amazon, so that part had to wait until I got some experience.

To send data from your Pi up to Amazon is actually pretty simple, so let's step through some code to illustrate it. First, you have to have mqtt client installed. You don't actually need mqtt server unless you already use it for your stuff, so install mosquito. You can refer back to my post where I discovered how easy to use mqtt actually is for my instructions <link>, or use one of the hundreds of others out there. Once you've got it in place and tested a bit, you're ready to put something together that will send data up to Amazon AWSIot; yep, I'm going to use the acronyms; you should be used to them by now.

#!/usr/bin/python

import os
import sys
import time
import paho.mqtt.client as mqtt
import ssl
import json
import pprint
from houseutils import timer, checkTimer

pp = pprint.PrettyPrinter(indent=2)

def on_awsConnect(client, userdata, flags, rc):
    print("mqtt connection to AWSIoT returned result: " + str(rc) )
    # Subscribing in on_connect() means that if we lose the connection and
    # reconnect then subscriptions will be renewed. You still have to do the
    # reconnect in code because that doesn't happen automatically
    client.subscribe ([(awsShadowDelta , 1 ),
                      (awsShadowDocuments, 1)])
                      
# If you want to see the shadow documents to observe what is going on'
# uncomment the prints below.
def on_awsMessage(client, userdata, msg):
    #print "TOPIC = ",
    #print msg.topic
    #print "PAYLOAD = ",
    payload = {}
    payload = json.loads(msg.payload)
    #pp.pprint (payload)
    #print ""
         
    if msg.topic == awsShadowDocuments:
        print "got entire shadow document"
        #pp.pprint (reported)

    elif msg.topic == awsShadowDelta:
        print ("got a delta")
        #print (pp.pformat(payload["state"]))
            
def updateIotShadow():
    temperature = 79.1
    # Create report in JSON format; this should be an object, etc.
    # but for now, this will do.
    report = "{ \"state\" : { \"reported\": {"
    report += "\"temp\": \"%s\", " %(int(round(temperature)))
    report += "\"lastEntry\": \"isHere\" " #This entry is only to make it easier on me
    report += "} } }" 
    # Print something to show it's alive
    print report
    print("Tick")
    err = awsMqtt.publish(awsShadowUpdate,report)
    if err[0] != 0:
        print("got error {} on publish".format(err[0]))

if __name__ == "__main__":
    # these are the two aws subscriptions you need to operate with
    # the 'delta' is for changes that need to be taken care of
    # and the 'documents' is where the various states and such
    # are kept
    awsShadowUpdate = "$aws/things/house/shadow/update"
    awsShadowDelta = "$aws/things/house/shadow/update/delta"
    awsShadowDocuments = "$aws/things/house/shadow/update/documents"
    # create an aws mqtt client and set up the connect handlers
    awsMqtt = mqtt.Client()
    awsMqtt.on_connect = on_awsConnect
    awsMqtt.on_message = on_awsMessage
    # certificates, host and port to use
    awsHost = "data.iot.us-east-1.amazonaws.com"
    awsPort = 8883
    caPath = "/home/pi/src/house/keys/aws-iot-rootCA.crt"
    certPath = "/home/pi/src/house/keys/cert.pem"
    keyPath = "/home/pi/src/house/keys/privkey.pem"
    # now set up encryption and connect
    awsMqtt.tls_set(caPath, certfile=certPath, keyfile=keyPath, cert_reqs=ssl.CERT_REQUIRED, tls_version=ssl.PROTOCOL_TLSv1_2, ciphers=None)
    awsMqtt.connect(awsHost, awsPort, keepalive=60)
    print ("did the connect to AWSIoT")
    
    # Now that everything is ready start the mqtt loop
    awsMqtt.loop_start()
    print ("mqtt loop started")

    # this timer fires every so often to update the
    # Amazon alexa device shaddow; check 'seconds' below
    shadowUpdateTimer = timer(updateIotShadow, seconds=10)
    print("Alexa Handling started")

    # The main loop
    while True:
        # Wait a bit
        checkTimer.tick()
        time.sleep(0.5)

The code above is more than an illustration of what to do, it actually runs on my machine and will update the AWSIoT shadow I'm currently using. I took a lot out of it because you probably don't do the sensors the same way I do, and there's no point in showing how I read the sensor data out of the database and format it.

Let's start at the top and work our way down in the code. The first two routines, on_awsConnect() and on_awsMessage() are the mqtt callbacks that you'll use to connect, subscribe and publish to the AWSIoT mqtt server. In on_awsConnect() I subscribe to the general topic,

"$aws/things/house/shadow/update/documents"

and

"$aws/things/house/shadow/update/delta".

The 'documents' one is where you can see the entire shadow document and the 'delta' is where you derive the command sent to your code by Alexa. But, for now, we're only going to discuss the data we want to send up and that is published to the topic,

"$aws/things/house/shadow/update",

because that is the first thing you need to conquer in getting this running. I only illustrate the subscriptions here so you can get a feel for the message interaction.

Now, specifically in on_awsMessage() I publish to

"$aws/things/house/shadow/update",

a specific format that AWSIot expects, it actually looks like this,

{ "state" : { "reported":
   {
     "temp": "79",
     "lastEntry": "isHere"}
   }
}

if you remove all the "\" crap. I did it as a string (probably a mistake) to cut down on the time it was taking me to try things. The "lastEntry" was added by me to make it easier to add entries to the JSON string as I implemented different sensors. I just copied that line, pasted it back in and changed the stuff to be what I wanted to work on next.

Yes, it's just that simple to send an update message, the hard part was finding out what the format was and the specific mqtt topic to publish to.

Now, down in the main code there's a few declarations of the topics used to make it somewhat simpler to code, and then the objects for the mqtt client and callbacks. The next bit,

    awsHost = "data.iot.us-east-1.amazonaws.com"
    awsPort = 8883
    caPath = "/home/pi/src/house/keys/aws-iot-rootCA.crt"
    certPath = "/home/pi/src/house/keys/cert.pem"
    keyPath = "/home/pi/src/house/keys/privkey.pem"

are pretty much boiler plate for specifying the port you have to use, the actual server you send to and the fully qualified path names to your certificates. You got the certs from the previous example and should have saved them somewhere. The reason for the full path names is so this code can run in any directory. Port 8883 is the one chosen by Amazon, you can't change it. Then, 

    awsMqtt.tls_set(caPath, certfile=certPath, keyfile=keyPath, cert_reqs=ssl.CERT_REQUIRED, tls_version=ssl.PROTOCOL_TLSv1_2, ciphers=None)

    awsMqtt.connect(awsHost, awsPort, keepalive=60)

cause tls encryption to be implemented (this comes free with mosquitto client) and the actual connection to the server. Next,

    awsMqtt.loop_start()

you must have an mqtt loop of some kind so the queue can be read, and awsMqtt.loop_start() is non-blocking. This is important so you can control when the updates are sent to AWSIoT. The last little bit is using the simple timer I created to keep from starting a separate thread for timing something this simple. I used to use a much more complex timer set up, but that was a waste of resources. The timer code I used is described here <link>, but of course, you can do anything you want.

The way this code works is every so often (10 seconds in this example) the timer fires and calls updateIotShadow() which formats a message holding a temperature I just hard coded in, and sends it up to a specific topic on AWSIot using my credentials. This connects it to the right place and suddenly, I have the data available on Amazon. 

Yes, that's literally all there is to sending data and having it show up there. You can verify this by  going to the AWSIoT console and looking at your 'thing'; it will have an entry for 'temp' and a value of '79'. Now what you have to do is implement all your sensors in the message which will make you realize why I put the 'LastEntry' in there, and check it out in the console on Amazon.

So, now would be an excellent time to bookmark the Amazon AWSIoT console in your browser; you're going to be using it a lot. You'll also use bookmarks to the Amazon Lambda server, and the Alexa server a lot.

Next time, We're going to work on a Amazon Lambda function to receive requests from Alexa (I'll be using an Amazon Dot) and respond with the temperature we just sent up. It'll be in two parts because it's mildly complex and it's best to take this stuff on a bite at a time.

Have fun.

The next post in this series is here <link>

Hook a Raspberry Pi to the Amazon Dot

Earlier this year Hackster.io had a contest looking for good internet of voice examples using a Raspberry Pi and an Amazon Echo <link>; one guy Nick Triantafillou entered with a simple weather application that would have the Echo say the current temperature and humidity taken from a DHT22 connected to his Pi <link>. This sounded like the perfect place to start including an Amazon Dot (or maybe a few of them) into my house infrastructure. I'm not interested in the latest Twitter stuff, I don't follow Facebook and the streaming services  I care about are already handled by my Roku, so this should be an interesting way to get my feet wetter in dealing with voice. I already tried voice on the Pi to try and control things, that was an abysmal failure <link>, and it would certainly be a fun thing to have around the house. So, I started stepping through his article. I like Nick's style and his instructions were clear enough to follow, so I just skipped the stuff about how to hook something to the Pi and got into the meat of hooking a Pi into Amazon AWS.

Go to https://aws.amazon.com/ and sign up for AWS;  remember, this stands for Amazon Web Services and you may have already done this in the last post. Once you get in there you'll find a screen that shows the various services they provide. For now, lets take a look at what the charges are, as in money you will have to pay them to use this stuff. If you look around on the page, there is a selection for AWS Iot (Amazon Web Services Internet of Things), and if you go there, you can find the pricing. It's a bit convoluted to find, as pricing always is for web services, so here's the link to the page I found on it:

https://aws.amazon.com/iot/pricing/

The prices they charge for stuff after you have used it for 12 months are here. Yes, it's free for the first year and after that they start charging you for the various services. However, look at the actual prices for the various features and breathe a sigh of relief since they charge in MILLIONS of transactions. I calculated that, if I use their services the same way I use them at home, it'll cost me a few cents a month. I really mean a few cents, maybe a quarter if I'm really pounding on the house. Probably more like a nickel or so a month. Heck, even I can afford that, and the first year is free. So, if you don't like it, cancel it before they start charging you.

At any rate, you've signed up for AWS and you can get the super secret stuff you need to go ahead with the configuration. If you don't like it, don't use it; they shouldn't charge you for just signing up and playing around, at least for the first year.

The very first thing you're going to need is a 'user'. To create a user that will be able to authenticate and use the various features, you use another Amazon Web Service IAM (Internet Access Management). This is where you'll poke in some permissions that will allow your Pi to assume the identity of the user you create and then upload temperature readings and such. At the AWS console, take a look around at the services and choose IAM and go there. Somewhere on that page is a button to create a user. These things move around on the screen depending on various criteria, so I can't tell you exactly where it will show up, but when I do it, it looks like this:

The 'alexaControl' user is the one I created to do things with. Create your own and you'll get a screen that looks something like this:

Hit the 'Create' button and you'll move to the next screen:

NOW, THIS IS IMPORTANT, click on the show credentials link. There is a secret shown that you won't be able to get again. You'll need both of the items in the next steps to create the thing. I took a screen shot so I wouldn't mess up entry of the items. I also did a copy and paste into notepad so I could copy them out. This is what they look like:

I didn't download the credentials.  Close the screen and you'll see your new user, click on it to get a screen where you'll assign various permissions and such. You don't need a 'group', but you do need to add some permissions. Click on the permissions tab and you'll get something like this:

Permissions in AWS are handled by 'policy' documents that are JSON files that are read when something needs to be done that may have to have a permission. Attach the three policies you see in the screen shot above. That gives pretty much unlimited permission to the stuff you're going to create. You can always come back later after it works and whittle down the permissions to what is actually needed, but it's much easier not to have to fight those items when you're first trying to get it working.

After you look around a bit at what the other items are, you can leave this part of AWS and, start looking for the AWS IOT console. We won't be using it yet, but it's good to have it up so you can check it after the next part.

I mentioned last post that I stepped through the creation of a thing using the web based tools that Amazon provides, but then I ran into trouble figuring out how to get the stuff I needed on my Pi so I could actually use it. That's where Nick's article came in handy. He has a step by step implementation of the items needed. Of course, I didn't do it exactly the same way he did, I used what I wanted to have. So, let's step through creating a thing and the various certificates we need on the Pi:

First, create a new directory somewhere and 'cd' into it; you'll be creating files there and you want to be able to find them later. Load the huge hunk of software that Amazon provides as an interface to your machine. I simply used the defaults and let it install:

sudo pip install awscli

This will load the python version of the AWS Command Line Interface that we'll be using. Next, you want to configure it, the command is:

$ aws configure

Which will prompt you for the security items I told you to save above. It will also ask you for the 'region'. THIS IS ALSO IMPORTANT, use 'us-east-1' for the region. AWS has data centers globally, and normally, you'd choose one close to you, but for Alexa, you must choose us-east-1 which may show up as N. Virginia in a few places on Amazon.

Now aren't you glad you listened to me and saved the secrets from above? Now the steps you run through on the Pi to get it ready to connect:

Create  a 'thing'

$ aws iot create-thing --thing-name "house"

I used the name 'house' for mine, yours could be anything you want.

Now you can list it and check that the command worked:

$ aws iot list-things
{
"things": [
  {
    "attributes": {},
    "thingName": "house"
  }
]
}

You'll need to use encryption, so create certificates and keys

$ aws iot create-keys-and-certificate --set-as-active \ 
--certificate-pem-outfile cert.pem \ 
--public-key-outfile publicKey.pem \ 
--private-key-outfile privkey.pem

This will have a lot of output to the screen, you can safely ignore it because it also creates files in the directory you're using that you will need later.

4. Get the certificate ARN which is the AWS Resource Name. You know you're in acronym hell when you're using acronyms made from acronyms. The ARN is going to be needed later, I highlighted it below:

$ aws iot list-certificates
{
"certificates": [
  {
      "certificateArn": "arn:aws:iot:us-east-1:926342229229:cert/58debbfe90fa58cad4df8426bdf3f20a71df7644437203e231b503b994dfb8f3",
      "status": "ACTIVE",
      "creationDate": 1467893789.688,
      "certificateId": "58debbfe90fa58cad4df8426bdf3f20a71cf7644437203e251b503b994dfb8f3"
  }
]
}

Word wrap kinda messed up the copy and paste above, but you get the idea. Download the root certificate from symantec. You'll need a root certificate and we get that from Symantec.com. If you want to know what a root certificate is and does, go look it up; the explanation is complex. To get it on the Pi:

$ wget https://www.symantec.com/content/en/us/enterprise/verisign/roots/VeriSign-Class%203-Public-Primary-Certification-Authority-G5.pem -O aws-iot-rootCA.crt

Now's a good time to discuss the certificates and keys you've accumulated. It's also a good time to round them up and keep them in a place you can remember and protect. Here's the list of my keys:

$ ls keys
aws-iot-rootCA.crt  cert.pem  privkey.pem  publicKey.pem

You should have all of them as well. I keep them in a directory called 'keys' and will soon limit access to them to the root user.

Now we need to create a policy that will allow your Pi to do things by giving it permission. You do this by creating a file that you will feed to aws and it will, in turn send it up to AWS on Amazon. The file contents:

{
    "Version": "2012-10-17",
    "Statement": [{
        "Effect": "Allow",
        "Action":["iot:*"],
        "Resource": ["*"]
    }]
}

Just use an editor, copy the stuff between the braces (include the braces at the beginning and end) and name the file something you'll recognize. The date in there is important, it's the version of software at Amazon that is used, don't change it. I followed Nick's suggestion and called it 'PubSubToAnyTopic', then use the file in the next command:

$ aws iot create-policy --policy-name "PubSubToAnyTopic" --policy-document file://iotpolicy.json

You'll get back:

{
    "policyName": "PubSubToAnyTopic",
    "policyArn": "arn:aws:iot:us-east-1:704749107060:policy/PubSubToAnyTopic",
    "policyDocument": "{\n    \"Version\": \"2012-10-17\", \n    \"Statement\": [{\n        \"Effect\": \"Allow\",\n        \"Action\":[\"iot:*\"],\n        \"Resource\": [\"*\"]\n    }]\n}",
    "policyVersionId": "1"
}

Attach the principal policy you just created, using the arn highlighted from earlier:

$ aws iot attach-principal-policy --principal "arn:aws:iot:us-east-1:926342229229:cert/58debbfe90fa58cad4df8426bdf3f20a71df7644437203e231b503b994dfb8f3" --policy-name "PubSubToAnyTopic"

Now, get the endpoint that you'll send data to. This is actually a URL that accepts mqtt data and gets it into Amazon. You will need the endpoint to actually send the data; it will show up in the code as the destination for data.

$ aws iot describe-endpoint

And you'll get back:

{
    "endpointAddress": "yournumberswillbehere.iot.us-east-1.amazonaws.com"
}

Now you have a place to send data to and a policy to handle the data. You created and named a 'thing' and it's ready to use.

But, how the heck can you tell? Go to the AWS management console; you should be getting good with the names of things by now and sign in. Look at the black bar at the top, it should be 'N. Virginia' since we set up as 'us-east-1', but if it isn't use the selection and change it. Also up at the top in the black bar is 'Services' click on it and a huge drop down of services will show up. Choose 'AWS IoT and you should get a screen like this:

There's the policy, thing and endpoint that you created using the steps above. You can click on them and see stuff related to each of them. The thing is where your shadow document will be that you'll be working with going forward.

This posting has taken hours to put together and check over, so I'm going to write about sending data from the Pi up to the thing we just created next time. 

Continued here <link>

Amazon Dot and My Desert Home

I already warned you that this was going to get complicated and would probably drive you nuts in my first post on the Amazon Dot <link>, but you apparently didn't listen, so I'll go a bit deeper into the amazingly intricate process of hooking a Dot to my house, but first a tiny bit of background.


This complex diagram outlines the various processes and machines that get involved in a single voice request, and any piece of it can give you trouble implementing something. I got this picture directly from a nice description I ran across on Amazon, and let's start off going there to understand what is going on <link>.

Robert McCauley, the author, is unusual in that this article can be understood; at least the second or third time you read it. Ignore his comment about referring to the Amazon quick start documentation for IoT, it will only confuse you and it really doesn't talk about anything Robert doesn't. However, this article is a bit terse (meaning he passes the buck entirely) on how to use the Alexa voice service. The thing you want to learn from this is that Amazon uses mqtt to control your devices. You can actually interact with them using the AWS mqtt tool that Amazon provides and the instructions that Robert provides.

I actually created his water pump and messed with it, but I deleted it to keep from confusing myself later when I created my 'house' device. What I did was create a single device 'house' and it carries stuff like outside temperature, state of various lights, garage door position, etc. It would have been a real pain to create each of those devices and deal with them separately. The huge thing to understand here is the use of the JSON shadow device. You deal exclusively with the shadow that represents the last reported state of your device (my house). The shadow contains a 'reported' section, a 'desired' section, and a 'delta' section. For example, here is my current (as of the date above) shadow; remember it's a JSON document.

{
  "reported": {
    "humid": "37",
    "temp": "82",
    "barometer": "1016",
    "windspeed": "4",
    "winddirection": "south southwest",
    "raintoday": "0.0",
    "eastPatioLight": "off",
    "outsideLights": "off",
    "lastEntry": "isHere",
    "frontPorch": "off",
    "cactusSpot": "off",
    "outsideGarage": "off",
    "mbLight": "off",
    "gDoor2": "closed",
    "gDoor1": "closed"
  }
}

I don't have all the items I'm interested in yet, and I only voice control a few things so far, but this could be representative of anyone's implementation.  This is the reported section. If I wanted to control something, I'd put in a desired section that would contain what I wanted to change. So, if I want to turn on the outside lights:

{
  "desired": {
    "outsideLights": "on"
  }
}

That would get combined with the reported section and then inspected for differences. Any differences would show up in a new delta section. That would make the entire shadow document look like:

{
  "desired": {
    "outsideLights": "on"
  },
  "reported": {
    "humid": "37",
    "temp": "80",
    "barometer": "1016",
    "windspeed": "2",
    "winddirection": "east southeast",
    "raintoday": "0.0",
    "eastPatioLight": "off",
    "outsideLights": "off",
    "lastEntry": "isHere",
    "frontPorch": "off",
    "cactusSpot": "off",
    "outsideGarage": "off",
    "mbLight": "off",
    "gDoor2": "closed",
    "gDoor1": "closed"
  },
  "delta": {
    "outsideLights": "on"
  }
}

So you have a JSON document that holds what was last reported, what you want to change, and the difference. See why they call the sections 'reported', 'desired' and 'delta'? It actually makes sense. When you get a device hooked up to the mqtt server at Amazon, the delta portion is the only part you see coming in. That makes it easier to understand what you need to do with the actual device. So, on your Pi, or whatever, you subscribe to the Amazon mqtt service, and wait for a delta message to come in. When it does, you do what it says and report back the new status. There are a few complications in there that you'll have to deal with, but I hope to visit each of them as we go through this process.

In Robert's article, he interacts with the shadow document using Amazon's mqtt client to subscribe and publish to the shadow. That's exactly the kind of thing you will need to do in code to support your device. This stuff is handled by what's called a Lambda function. The Lambda funtion is simply code that you run on their machine that can bridge the gap between the Alexa voice service and the mqtt server to get something down to your machine to change something. In the other direction, you'll publish the latest state of your devices back so everything is kept up to date.

Now, do you understand why a little hands on with the Amazon IoT is necessary? Follow the lead from Robert's article and work with mqtt and the shadow a bit to see what is actually going on. Totally ignore his comments about Alexa, "Once you are familiar with the Alexa Skills Kit and understand how to create an Alexa skill;" That phrase is just about totally useless.

I have a suspicion that very few of Amazon's people that work on these projects actually do anything with them since their explanations leave so much out.

That's enough for you to get a feel for the interaction if you actually do it and then look at the messages that are passed in mqtt. You WILL need this when you start troubleshooting your own code.

Next post we're going to look at getting data from your device all the way back to Alexa. We won't actually try to change the state of a device because you have to have a reported section to the shadow before you can change it.

And, you probably won't need the water pump device that Robert had you create.

Continued here <link>

I Got One of Those New Fangled Amazon Dot Things

Obviously, this is part 1.

Yes, I took the plunge and bought an Amazon Dot before it was available and waited patiently for the release date for it to arrive. When it came in, I did the usual, "Alexa, tell me the weather." "Alexa, what time is it?" "Alexa, how tall is Scarlett Johansson?" You know, the usual.

Then, I got out the laptop and started looking at how to connect it to my house.

Let me tell you, this thing is really, really cool, but it's also the most frustrating device I have worked on in quite a while. I literally spent hours and hours on the web trying to figure out Amazon's amazingly complex and poorly documented interface to IOT (internet of things). I tried a large number of examples and tutorials out there with a singular result: they didn't work. I knew it was possible, there are youtube videos that prove it, but not a single one of them worked for me. I wrote, copied, stole thousands of lines of code and didn't get a single signal of any kind at my house. So, in a mass of frustration and disappointment, I managed to stumble (this is the third full 16 hour day of trying) across a little note on rules. It started to work part way.

Emboldened by a tiny success, I dug in and conquered the device all the way to the house, out to my patio lights and all the way back to the Alexa sitting on my kitchen counter. I had it working. No, I haven't got all the devices tied in, but I can get the weather readings taken at my house and actually turn on and off my patio light.

The problem for me, and I suspect many others, is that the interface is extremely complex and spans several of Amazon's cloud service products. For example, you speak to the Alexa and it runs many layers of recognition and permission checking that updates billing code and such before it hands something off to their Lambda service. The Lambda service runs some code that hands things off to their AWSIoT service who does some stuff and passes your signals off to Amazon's own MQTT service which will hand things off to your code at home. Your code reads this and does something returning it back to Amazon's MQTT, which passes it back to Lambda who passes it back to the Alexa service and then back to the device on the counter to be played back. In the middle of this are many, many layers of authentication, roles and permissions that have to be exactly right before things are allowed to pass. Heck, you can't even log anything without using their cloud logging service ... and you have to read it there as well.

Just forget about it being easy; it isn't. Maybe someday it will be, but it isn't right now. And, NO, I'm not going to add to the huge number of tutorials out there with my own version. Frankly, I really don't want to step through it all over again taking a hundred screen shots that will become obsolete in three or four days when they change the user interface to the various services all over again. One of the biggest problems I had was trying to work from some of the examples that were of an older interface and try finding similar capabilities on what I was looking at. You see, they have crews of people working on each piece expanding its capabilities, and even though they are very careful, things get missed. That of course means they have to fix the thing they broke and that can change the interface. It's one little guy out in the desert vs. hundreds spread around the world.

Let's talk about a specific example. Below is the session where I created the various devices using Amazon's CLI (command line interface) that I had to download and install on my Raspberry Pi. After installing it I used instructions I found out there on that world wide web thingie, and this is the console log I kept of the process. I annotated it to tell you what was actually happening after I completed the process, and it should give you a real example of what to do to set yourself up in the very beginning on the Pi:

#
# Configuring AWS ... but
# I kept the secrets that were produced by the user creation so I could use
# them here, but I forgot the step of adding permission for the user I created
# This is what happens when you forgot to add permissions to the user
#
pi@housemonitor:~/src/alexa$ aws configure
AWS Access Key ID [None]: ***Secret Stuff***
AWS Secret Access Key [None]: ***More Secret Stuff***
Default region name [None]: us-east-1
Default output format [None]:
#
# This was trying to create a 'thing' but didn't have permission 
# to do it.
#
pi@housemonitor:~/src/alexa$ aws iot create-thing --thing-name "house"

An error occurred (AccessDeniedException) when calling the CreateThing operation: User: arn:aws:iam::704749107060:user/alexaControl is not authorized to perform: iot:CreateThing on resource: arn:aws:iot:us-east-1:704749107060:thing/house
#
# After I went back and added:
#   IAMFullAccess, AWSIoTFullAccess and AWSLambdaFullAcess 
# to the user I created
#
# Then I created a 'thing' called "house"
#
pi@housemonitor:~/src/alexa$ aws iot create-thing --thing-name "house"
{
    "thingArn": "arn:aws:iot:us-east-1:704749107060:thing/house",
    "thingName": "house"
}
#
# And proved it was there by listing it
#
pi@housemonitor:~/src/alexa$ aws iot list-things
{
    "things": [
        {
            "attributes": {},
            "version": 1,
            "thingName": "house"
        }
    ]
}
#
# Custom Alexa "things" are all about security, so I generated keys
# which sent the keys to the console and scrolled the screen a bunch
#
pi@housemonitor:~/src/alexa$ aws iot create-keys-and-certificate --set-as-active --certificate-pem-outfile cert.pem --public-key-outfile publicKey.pem --private-key-outfile privkey.pem
{
    "certificateArn": "arn:aws:iot:us-east-1:704749107060:cert/e325f7755f0a4f11a59416ab7af30e20023f2c411233ecb5d2c68285843f94a4",
    "certificatePem": "-----BEGIN CERTIFICATE-----\nbunch of stuff\n-----END CERTIFICATE-----\n",
    "keyPair": {
        "PublicKey": "-----BEGIN PUBLIC KEY-----\nbunch of stuff\n-----END PUBLIC KEY-----\n",
        "PrivateKey": "-----BEGIN RSA PRIVATE KEY-----\nbunch of stuff\n-----END RSA PRIVATE KEY-----\n"
    },
    "certificateId": "even more stuff"
}
#
# But, they were in the directory I created for this
# I proved it by listing them
#
pi@housemonitor:~/src/alexa$ aws iot list-certificates
{
    "certificates": [
        {
            "certificateArn": "arn:aws:iot:us-east-1:704749107060:cert/e325f7755f0a4f11a59416ab7af30e20023f2c411233ecb5d2c68285843f94a4",
            "status": "ACTIVE",
            "creationDate": 1477339839.64,
            "certificateId": "e325f7755f0a4f11a59416ab7af30e20023f2c411233ecb5d2c68285843f94a4"
        }
    ]
}
#
# Here's the list of them
#
pi@housemonitor:~/src/alexa$ ls
cert.pem  privkey.pem  publicKey.pem
# These are the certificate, private key and public key. Keep
# the private key protected and use the public key for communication
#
# This is downloading the public key for Amazon AWS IOT
# I need this to encrypt the traffic to Amazon
#
pi@housemonitor:~/src/alexa$ wget https://www.symantec.com/content/en/us/enterprise/verisign/roots/VeriSign-Class%203-Public-Primary-Certification-Authority-G5.pem -O aws-iot-rootCA.crt
--2016-10-24 13:14:47--  https://www.symantec.com/content/en/us/enterprise/verisign/roots/VeriSign-Class%203-Public-Primary-Certification-Authority-G5.pem
Resolving www.symantec.com (www.symantec.com)... 104.100.196.23
Connecting to www.symantec.com (www.symantec.com)|104.100.196.23|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1758 (1.7K) [text/plain]
Saving to: ‘aws-iot-rootCA.crt’

aws-iot-rootCA.crt  100%[=====================>]   1.72K  --.-KB/s   in 0.001s

2016-10-24 13:14:49 (1.25 MB/s) - ‘aws-iot-rootCA.crt’ saved [1758/1758]
#
# Now I take a look at the directory to see what keys I 
# have accumulated so far
#
pi@housemonitor:~/src/alexa$ ls
aws-iot-rootCA.crt  cert.pem  privkey.pem  publicKey.pem
#
# This is where it starts to get even more complicated
# I have to have a policy in place to allow this Amazon user
# to publish data. The policy is a JSON formatted file I called
# iotpolicy.json (I totally stole that name from Nick Triantafillou)
# 
pi@housemonitor:~/src/alexa$ aws iot create-policy --policy-name "PubSubToAnyTopic" --policy-document file://iotpolicy.json
{
    "policyName": "PubSubToAnyTopic",
    "policyArn": "arn:aws:iot:us-east-1:704749107060:policy/PubSubToAnyTopic",
    "policyDocument": "{\n    \"Version\": \"2012-10-17\", \n    \"Statement\": [{\n        \"Effect\": \"Allow\",\n        \"Action\":[\"iot:*\"],\n        \"Resource\": [\"*\"]\n    }]\n}",
    "policyVersionId": "1"
}
#
# Now that a policy has been put in place up on Amazon AWS
# I have to attache it to the arn (Amazon resource name) which is 
# actually saying to attach it to the place I'll eventually send it 
# to
pi@housemonitor:~/src/alexa$  aws iot attach-principal-policy --principal "arn:aws:iot:us-east-1:704749107060:cert/e325f7755f0a4f11a59416ab7af30e20023f2c411233ecb5d2c68285843f94a4" --policy-name "PubSubToAnyTopic"
pi@housemonitor:~/src/alexa$
# I get the endpoint that I'll send data to. This is actually a
# URL that accepts mqtt data and gets it into Amazon
# I will need the endpoint to actually send the data
# This endpoint will show up in the code as the destination for data
pi@housemonitor:~/src/alexa$ aws iot describe-endpoint
{
    "endpointAddress": "ayccly8mhj4t3.iot.us-east-1.amazonaws.com"
}
#
# So I have a place to send data to and a policy to handle the data,
# I need to attach that to the 'thing' I created called 'house'
#
pi@housemonitor:~/src/alexa$ aws iot attach-thing-principal --thing-name "house" --principal  "arn:aws:iot:us-east-1:704749107060:cert/e325f7755f0a4f11a59416ab7af30e20023f2c411233ecb5d2c68285843f94a4"
pi@housemonitor:~/src/alexa$
#
# Finally, the linkage to aws is done I just have to test it.
# wish me luck

This little set of steps didn't seem to bad, but I didn't understand much of it at first. That seems to be the general rule, you poke at it until you get something to work sort of, and then you put out a tutorial or example and explain things about half way because that's all you really understand. As an example, there is a line down near the bottom of the above about attach-thing-principal. This is one line in a json file that the Amazon service holds that needs to be there. They call it a principal and you have to create it with a command they provide in the CLI. It's one line, they make it sound like something important. Well, actually it is, but only because they made it important. The Amazon interface is full of things like this.

They came up with a rather clever idea. They hold the state of your device in a regular old json file on their server. No, it isn't a database, it's just a file. This is clever because they can modify the file to represent things as they change. When I was working on getting my weather data up to Amazon I was able to add one item at a time and watch the file change as I added them. I started with the temperature (naturally) and it showed up in the file. Then I added the barometric pressure and saw it show up. I stepped through the items I wanted to check on until I got a nice little weather report I could ask for while walking around the kitchen.

Once I got that working pretty nicely, I wanted to actually change something, and this is where the way that they handle the file became really useful. You get the device to be reported and it shows up in the JSON file, then you tell alexa to change it and the file changes to show new things. It has a 'desired' entry where the new state of the device shows up, and then a 'delta' entry where the difference from the overall status and the 'desired' show up. Only the delta is sent to the Pi for it to act on. You don't have to worry about what went on before at the Pi, you just do whatever the 'delta' tells you to.

This may not sound like much, but it allows the device to go off line for periods and come back to catch up with what you last told it to do. It also provide readings for the last time the device was on line. They call this entire mess a 'shadow' and I learned to like it.

So, don't despair, over the next (however many) postings I'll tell you about parts of the system that you can step through to get your own house hooked into the Alexa. But, NO, I won't take a look at your individual implementation and show you what's wrong, nor will I answer basic questions like how do you sign on to Alexa. That stuff changes almost as fast as I could write about it. Get good at bookmarking the various pages you use on Amazon, they're hard to find a second time. Remember, I'm learning about this as I go, just like you are.

The difference is that I already put over a week into the darn thing and have actually got a light I can control with it.

Oh, before you folk ask why I went my own way instead of using Smart Things or one of the other home automation systems out there. I hate cloud services. I don't want my data stored somewhere else under someone else's control and subject to their whim. If Amazon changes their policy, I just don't use them for remote control anymore; I get out the phone and touch the screen instead. The Alexa is cool, but it won't control me or my data.

Continued here <link>

After a Long Testing Period, Moving Forward With My Temperature Sensor

A while back I came up with a battery operated temperature sensor that sent its data over my XBee network <link>. This little device has been sitting beside my bed for a long time now and works well. It has had it troubles over the months that I had to fix, but that was the point: make sure it works before moving it into a bigger project. Also, I use it to turn off my bedroom lights from the bed and have coupled it into turning off other things to be sure they're ready for the night as well.

I decided it was time to make some more of them, but the thought of hooking up thirty or so little fiddly wires on a protoboard kept me from actually doing it. What I needed was a custom PC board. I've designed a board before for a charger I came up with for the various lead acid batteries around the place <link>, but that project died when I found a really good battery maintainer to use commercially available. I still have one of the boards running, but it's only monitoring the battery voltage, not charging anymore. I may get back to that project at some point and come up with a simple battery monitor replacement, but right now I need an easier assembly technique and a custom board sounds like a great idea.

I dragged out Eagle <link> that had been hiding somewhere on my machine for months and updated it (of course) and started trying to use it again. Needless to say, I had to find a couple of tutorials to get me started again. I took the schematic for the sensor and came up with a board that looked like it would work, let it sit for a couple of days, revisited it and made a couple of changes then sent it off to OSH <link> which is SparkFun's old PC board service they farmed out.

I got the boards back a couple of days ago and assembled one of the three to see if it would work.

They're two inches square and purple. They're also thinner than I'm used to seeing, but that doesn't really seem to matter; they are strong enough to use.

When I got one of them loaded with components:

Yep, I'm still using the cheapest batteries I can find. I put the XBee and the Arduino side by side instead of vertically to meet a different form factor I couldn't try with the prototype. I'm still mounting all the active components in sockets so I can trade them out if necessary. I was lucky, it worked first try.

Well, that's partly a lie. The circuitry was fine and everything connected up OK, but I put the wrong profile on the XBee and it took some head scratching to find out what happened. Note to self: pay attention to what you name the profiles.

I went and got one of my famous rubber bands and packaged it up:

There's some things I might do differently on my next order, and in general. For example: if I flip the ftdi connector over to the other side of the Arduino it would make the height shorter and maybe easier to mount in a permanent enclosure. It might be good to actually include some holes for mounting the thing; I totally forgot that part. I did think of things like a place for a connector for the switch, but I put it too close to the switch to be easily connected. Lastly, more labels on the board. I had trouble telling which capacitor went where. Labels like C1, and C2 didn't tell me much and I had to keep looking at the schematic to assemble it. Of course, if you already have an example made, this problem doesn't exist, so maybe I'll just keep a good picture of it on my phone to refer to later.

The period from creating the artwork for the board and getting it back was long enough that I even forgot which way I pointed the Arduino and XBee. Sad I didn't put an arrow or something on the silk screen for the board. But, I guess we have to learn some things the hard way. At least I do. Nevertheless, IT WORKED !

I'm actually pretty happy with how it turned out. The idea of taking major components that I can buy for the most complex parts and just mounting them as modules on a board that interfaces them and has the interface components worked really well. I don't have to stock all the parts for a bare bones Arduino, I just use a cheap Arduino Pro Mini. I don't have to stock some radio parts or fiddle with RF alignment, I just use an XBee. The only parts are simple to install ones and not many of them.

Now, I have a bunch of work to do. I already had code on my Raspberry Pi to update the data base when a new temperature sensor appeared and that worked well, so I'm saving readings from two of them now (the prototype and the new one), but I'm not doing anything with it. The real objective for these is to put one in each strategic place around the house and use them to control the house temperature.

The plan is to measure the temperature and intelligently control the air handlers and compressors of my two heat pumps to adjust for warm spots in the summer and cold ones in the winter. I want to get out of the tub after a long soak in the winter and NOT freeze my butt off. I can use the air handlers to distribute warm air from the hot side of the house in the winter, and just reverse that in the summer.

I won't have thermostats at all, I'll network the entire thing and control it with an HTML interface. I may still leave a display up in the place of the thermostats since people expect to be able to look at one, but it won't have any buttons. OR, I may put a cheap tablet up on the wall with a browser running to control the entire house with.

This will mean a control board at each air handler so I can control various relays that work the fan, compressor and reversing switch that hooks into my XBee network as well. But hey, I talked a bit about that already <link> so I won't bore you until I actually start that part.

I'll need at least three more boards, and I may make the changes I talked about in the second order, but thinking about it, the changes are trivial and I can live without them. I also have other ideas about using something like this to monitor the moisture in the soil of my two new fruit trees, and as mentioned earlier, the state of the tractor batteries in the barn. See, I can put any sensor on the device and have it transmit whatever I want to my network. I may look at a motion sensor for the driveway to tell me when someone drives up. Doing that without wires would be really cool. That would mean some changes to the board for the different uses, but that also means that I GET TO MAKE CHANGES TO THE BOARD as well as try out some new sensor and code.

I'm definitely going to need some more batteries.

So, My Raspberry Pi Web Server Was Running Slow

I was sitting on a bar stool showing off my Pi web server that controls my house and had to wait 15-20 seconds for each screen change. Loading graphs was painfully slow and would pause in the middle with only half the graph showing. It was embarrassing. When I got home, I took a look and the load average was up in the double digits; something I had never seen before. Obviously, there was some process or other out of control that needed to be fixed. I was wrong.

Granted, it's my oldest Pi; a Pi 1 that I just keep because it's easier than bringing up my Pi3B to do the same job. I recently put a SSD on it, and it logs to a database server up in the attic, so it's been fast enough. Now, it was just crawling along.

When I went looking for what was causing it I found someone out there on the web was loading my data as fast as the process could be run. Don't misunderstand, there's nothing secret there and folk visit my site all the time to see what's different from the last time, and I've had something similar happen before; it was quite innocent. What most people don't realize is that a site that automatically updates by doing a periodic get, when put in background, will continue to update. So, you visit one of the news sites, hit the back button down on the bottom of the phone, the app disappears and you go do something else, and the app continues to run updating the screen you can't see. This can cause data overages and such, but the app is ready when you come back. This shows up in my logs as someone on the site for a very long time.

Almost all sites are polite about this auto-update and only update on a multi-minute schedule; I have my site update every 10 seconds because I want to double check and see that the garage door actually closed like it was supposed to. What was happening was someone had set up a loop that would grab the data again as soon as it was delivered. That caused a lot of database read activity and slowed the machine down a LOT. Of course, I didn't realize this at first and assumed I needed to check the efficiency of the data gathering steps.

I have a php script that gathers the data from my database and returns it to the web user called housedata.php. It's a rather simple implementation, so I took it and started timing the various operations by commenting out pieces and timing it using the 'time' command in bash. The stupid little process was taking 4.75 seconds on average to finish at first, but after some database query changes, it went way down; but all that did was allow the person out there to call my machine faster.

I looked at excluding the person by IP address using the features of the apache2 web server and succeeded in stopping the interaction quite nicely. That made me think about what else might be going on, so I took a closer look at the logs. There was the usual script kiddies trash looking for 10 year old vulnerabilities, search engines prowling around, and days worth of this person beating on my machine. I had fixed the problem, so I improved the speed of housedata.php a little more and called it done. The next morning, the person was right back in there with a slightly different IP address doing the same thing.

I added the new address to the web server exclusions and noticed that it was in a subnet of the internet provider that was being used. Ha! I excluded the entire subnet to stop the problem. The problem with excluding the IP addresses with the web server was that the web server starts a process for each hit. That takes time and machine resources, not a lot, but enough to notice over time. It looked like it was time to actually bring up a firewall to protect the little machine.

I already knew about 'iptables', but have you ever tried to use that thing? It's really hard to set up, and I could mess it up pretty badly leaving holes where there shouldn't be and locking myself out of my own machine. I shuddered at trying to get that working without a months worth of research, but then discovered 'ufw' a tool designed to help with that process. I did the dreaded apt-get update command and then an apt-get install ufw so I could try it out. Notice that I did NOT use apt-get upgrade! I'm getting really tired of having too much stuff on my machine replaced by well meaning folk out there. The last time I did that I wound up with a new slightly incompatible operating system (jessie).

Since I run headless (no keyboard or console), I was afraid of actually enabling the firewall since it would exclude port 22 and I wouldn't be able to get into the machine to do anything without dragging it to a TV set somewhere and poking around for hours sitting on the floor in front of it. Fortunately, the folk that put the package together left a note in one of the configuration files about this very thing and I did what they suggested. Gritting my teeth in expectation of failure, I started the process and it warned me that ssh sessions could be interrupted and asked for confirmation. I gritted a little harder and answered 'Y'.

It came up just fine and didn't affect the ssh session at all. I was on my way.

If you have to do this, one thing you'll find annoying is the huge amount of introductions, tutorials, promotions, and examples out there on the web that don't tell you what you want to know. Sure they tell you stuff that is valuable, but I didn't find a single one that covered what I needed to do; it was all trial and error. Painful trial and error. After about an hour I came up with an idea: look at the darn log file created by ufw to see what was going on. On the Pi, the log records are mixed in with other stuff in the file /var/log/messages. So, I set up a way to watch it and see what was happening:

tail -f /var/log/messages | grep UFW

After watching a while for the various things that were being dropped, changing the configuration, watching some more, I got it working perfectly for my purposes. I had a little annoyance with the order of the rules. See, when ufw (actually iptables, ufw is just an interface) sees a packet, it steps through its rules in order and stops when it satisfies the first one. So if you allow access to port 80 as the first line, you can't exclude a specific IP address later; it already let the packet through and stopped looking. So, put the stuff you want stopped first and then the stuff you want to allow later.

I allowed all the machines on my local network to get to the web server for various things, but only open port 80 outside the house. Here's the list I'm currently using:

pi@housemonitor:/var/log/apache2$ sudo ufw status numbered
Status: active

     To                         Action      From
     --                         ------      ----
[ 1] 22                         ALLOW IN    192.168.0.0/16
[ 2] Anywhere                   DENY IN     69.145.122.0/24
[ 3] Anywhere                   DENY IN     180.16.15.0/24
[ 4] Anywhere                   DENY IN     180.76.15.0/24
[ 5] 80                         ALLOW IN    Anywhere
[ 6] 3551                       ALLOW IN    192.168.0.0/16
[ 7] Samba4                     ALLOW IN    192.168.0.0/16
[ 8] 224.0.0.251                ALLOW IN    192.168.0.0/24
[ 9] 224.0.0.1                  ALLOW IN    192.168.0.0/24
[10] 22                         ALLOW IN    Anywhere (v6)
[11] 80                         ALLOW IN    Anywhere (v6)
[12] 3551                       ALLOW IN    Anywhere (v6)

pi@housemonitor:/var/log/apache2$

First, I allow port 22 from all internal addresses. Like I said, I worry about excluding my own access to the machine. Then a series of nets excluded because I saw them messing around. The 69.145.122.0/24 address is where the annoying traffic was coming from, the two addresses that start off with 180 are for a Chinese web crawler for a search engine. It was hitting my machine every 30 minutes from two different addresses that changed within that range. I don't mind search engines, but every 30 minutes? The IP version 6 stuff is the default, I haven't gotten to it yet.

Port 3551 is for my APC UPS that I wrote about recently. That device is working really well and controls the shutdown of all my Pi's so I want the machines to be able to interact. Of course I use Samba to move files around, so there's an entry for that. The two 224 addresses are for ARP and such, I put the entries in just to keep them out of the log.

As soon as I did that, my load level dropped to fractional numbers; the machine finally had time to actually do things.

Also, since the person out there was getting time outs from not being able to get a response from my machine, its hits dropped to every 30 seconds or so. These hits were dropped at the protocol level, so they don't cause me any problems at all.

This rather busy screen shot shows the hits roughly every 30 seconds trying to load stuff, and each of them gets dropped with no response. The really annoying thing is that this person is STILL trying to get in. I've been working on this for a few days now and this robot doesn't have the smarts to try something else. I know it's a 'bot because it goes directly after the data server code without going through the web page. After I finished with the changes I let the machine run overnight and looked again, neither the web crawler or the annoying 'bot got in.

Success !

At least so far. Sure, I'll get annoying traffic again, but now I know how to stop it and have the means to do it relatively easily.

Don't misunderstand, I don't mind people looking at the site at all; I encourage folk to take a look. It's there not only so I can close my garage doors, but to serve as an example and source of ideas and suggestions. I just ask that you don't set up scripts to mess around with it for days at a time, and until now, everyone has been really nice about it. Some of my (ahem) ideas have come from people looking around and suggesting things. I've saved many a kid from a bad grade in a computer science class because they can steal code from me. And, there have been a couple of seriously interesting term papers written based on things they found here.

I'm actually contributing ... well sort of.

One last thing came out of this exercise. I had to put in a fake 'index.html' page because of the various 'bots that want to prowl around the site. What happens is the 'bot goes for the web site and then uses the URLs inside the index page to find other stuff on the site. Then it tries various 'exploits' to break in and do something bad on each of the pages. If there is no index page, it tries to get a directory listing and from there it starts messing around. If you put in a dead end index page, it can't get a directory listing and there are no URLs in the page to leverage from; the bot gives up and moves on. The script kiddies and the constant data loading came to the attention of the web monitoring tools at my ISP and they started expiring my IP lease a couple of times a day to break up the traffic. Each time they did that, I was off the network for a short time while the DNS servers were updated. I have code in place for this kind of thing, so I didn't have to do anything, but it got annoying. We'll see if my changes remove this problem.

All in all, this was both annoying and fun. I got to learn about new stuff, make changes that work really well to some of my code and have something new to brag about when I go to the bar.