Tuesday, November 8, 2016

Hook a Raspberry Pi to the Amazon Dot

Earlier this year Hackster.io had a contest looking for good internet of voice examples using a Raspberry Pi and an Amazon Echo <link>; one guy Nick Triantafillou entered with a simple weather application that would have the Echo say the current temperature and humidity taken from a DHT22 connected to his Pi <link>. This sounded like the perfect place to start including an Amazon Dot (or maybe a few of them) into my house infrastructure. I'm not interested in the latest Twitter stuff, I don't follow Facebook and the streaming services  I care about are already handled by my Roku, so this should be an interesting way to get my feet wetter in dealing with voice. I already tried voice on the Pi to try and control things, that was an abysmal failure <link>, and it would certainly be a fun thing to have around the house. So, I started stepping through his article. I like Nick's style and his instructions were clear enough to follow, so I just skipped the stuff about how to hook something to the Pi and got into the meat of hooking a Pi into Amazon AWS.

Go to https://aws.amazon.com/ and sign up for AWS;  remember, this stands for Amazon Web Services and you may have already done this in the last post. Once you get in there you'll find a screen that shows the various services they provide. For now, lets take a look at what the charges are, as in money you will have to pay them to use this stuff. If you look around on the page, there is a selection for AWS Iot (Amazon Web Services Internet of Things), and if you go there, you can find the pricing. It's a bit convoluted to find, as pricing always is for web services, so here's the link to the page I found on it:


The prices they charge for stuff after you have used it for 12 months are here. Yes, it's free for the first year and after that they start charging you for the various services. However, look at the actual prices for the various features and breathe a sigh of relief since they charge in MILLIONS of transactions. I calculated that, if I use their services the same way I use them at home, it'll cost me a few cents a month. I really mean a few cents, maybe a quarter if I'm really pounding on the house. Probably more like a nickel or so a month. Heck, even I can afford that, and the first year is free. So, if you don't like it, cancel it before they start charging you.

At any rate, you've signed up for AWS and you can get the super secret stuff you need to go ahead with the configuration. If you don't like it, don't use it; they shouldn't charge you for just signing up and playing around, at least for the first year.

The very first thing you're going to need is a 'user'. To create a user that will be able to authenticate and use the various features, you use another Amazon Web Service IAM (Internet Access Management). This is where you'll poke in some permissions that will allow your Pi to assume the identity of the user you create and then upload temperature readings and such. At the AWS console, take a look around at the services and choose IAM and go there. Somewhere on that page is a button to create a user. These things move around on the screen depending on various criteria, so I can't tell you exactly where it will show up, but when I do it, it looks like this:


The 'alexaControl' user is the one I created to do things with. Create your own and you'll get a screen that looks something like this:


Hit the 'Create' button and you'll move to the next screen:


NOW, THIS IS IMPORTANT, click on the show credentials link. There is a secret shown that you won't be able to get again. You'll need both of the items in the next steps to create the thing. I took a screen shot so I wouldn't mess up entry of the items. I also did a copy and paste into notepad so I could copy them out. This is what they look like:


I didn't download the credentials.  Close the screen and you'll see your new user, click on it to get a screen where you'll assign various permissions and such. You don't need a 'group', but you do need to add some permissions. Click on the permissions tab and you'll get something like this:


Permissions in AWS are handled by 'policy' documents that are JSON files that are read when something needs to be done that may have to have a permission. Attach the three policies you see in the screen shot above. That gives pretty much unlimited permission to the stuff you're going to create. You can always come back later after it works and whittle down the permissions to what is actually needed, but it's much easier not to have to fight those items when you're first trying to get it working.

After you look around a bit at what the other items are, you can leave this part of AWS and, start looking for the AWS IOT console. We won't be using it yet, but it's good to have it up so you can check it after the next part.

I mentioned last post that I stepped through the creation of a thing using the web based tools that Amazon provides, but then I ran into trouble figuring out how to get the stuff I needed on my Pi so I could actually use it. That's where Nick's article came in handy. He has a step by step implementation of the items needed. Of course, I didn't do it exactly the same way he did, I used what I wanted to have. So, let's step through creating a thing and the various certificates we need on the Pi:

First, create a new directory somewhere and 'cd' into it; you'll be creating files there and you want to be able to find them later. Load the huge hunk of software that Amazon provides as an interface to your machine. I simply used the defaults and let it install:

sudo pip install awscli

This will load the python version of the AWS Command Line Interface that we'll be using. Next, you want to configure it, the command is:

$ aws configure

Which will prompt you for the security items I told you to save above. It will also ask you for the 'region'. THIS IS ALSO IMPORTANT, use 'us-east-1' for the region. AWS has data centers globally, and normally, you'd choose one close to you, but for Alexa, you must choose us-east-1 which may show up as N. Virginia in a few places on Amazon.

Now aren't you glad you listened to me and saved the secrets from above? Now the steps you run through on the Pi to get it ready to connect:

Create  a 'thing'

$ aws iot create-thing --thing-name "house"

I used the name 'house' for mine, yours could be anything you want.

Now you can list it and check that the command worked:

$ aws iot list-things
{
"things": [
  {
    "attributes": {},
    "thingName": "house"
  }
]
}

You'll need to use encryption, so create certificates and keys

$ aws iot create-keys-and-certificate --set-as-active \ 
--certificate-pem-outfile cert.pem \ 
--public-key-outfile publicKey.pem \ 
--private-key-outfile privkey.pem

This will have a lot of output to the screen, you can safely ignore it because it also creates files in the directory you're using that you will need later.

4. Get the certificate ARN which is the AWS Resource Name. You know you're in acronym hell when you're using acronyms made from acronyms. The ARN is going to be needed later, I highlighted it below:

$ aws iot list-certificates
{
"certificates": [
  {
      "certificateArn": "arn:aws:iot:us-east-1:926342229229:cert/58debbfe90fa58cad4df8426bdf3f20a71df7644437203e231b503b994dfb8f3",
      "status": "ACTIVE",
      "creationDate": 1467893789.688,
      "certificateId": "58debbfe90fa58cad4df8426bdf3f20a71cf7644437203e251b503b994dfb8f3"
  }
]
}

Word wrap kinda messed up the copy and paste above, but you get the idea. Download the root certificate from symantec. You'll need a root certificate and we get that from Symantec.com. If you want to know what a root certificate is and does, go look it up; the explanation is complex. To get it on the Pi:

$ wget https://www.symantec.com/content/en/us/enterprise/verisign/roots/VeriSign-Class%203-Public-Primary-Certification-Authority-G5.pem -O aws-iot-rootCA.crt

Now's a good time to discuss the certificates and keys you've accumulated. It's also a good time to round them up and keep them in a place you can remember and protect. Here's the list of my keys:

$ ls keys
aws-iot-rootCA.crt  cert.pem  privkey.pem  publicKey.pem

You should have all of them as well. I keep them in a directory called 'keys' and will soon limit access to them to the root user.

Now we need to create a policy that will allow your Pi to do things by giving it permission. You do this by creating a file that you will feed to aws and it will, in turn send it up to AWS on Amazon. The file contents:

{
    "Version": "2012-10-17",
    "Statement": [{
        "Effect": "Allow",
        "Action":["iot:*"],
        "Resource": ["*"]
    }]
}

Just use an editor, copy the stuff between the braces (include the braces at the beginning and end) and name the file something you'll recognize. The date in there is important, it's the version of software at Amazon that is used, don't change it. I followed Nick's suggestion and called it 'PubSubToAnyTopic', then use the file in the next command:

$ aws iot create-policy --policy-name "PubSubToAnyTopic" --policy-document file://iotpolicy.json

You'll get back:

{
    "policyName": "PubSubToAnyTopic",
    "policyArn": "arn:aws:iot:us-east-1:704749107060:policy/PubSubToAnyTopic",
    "policyDocument": "{\n    \"Version\": \"2012-10-17\", \n    \"Statement\": [{\n        \"Effect\": \"Allow\",\n        \"Action\":[\"iot:*\"],\n        \"Resource\": [\"*\"]\n    }]\n}",
    "policyVersionId": "1"
}

Attach the principal policy you just created, using the arn highlighted from earlier:

$ aws iot attach-principal-policy --principal "arn:aws:iot:us-east-1:926342229229:cert/58debbfe90fa58cad4df8426bdf3f20a71df7644437203e231b503b994dfb8f3" --policy-name "PubSubToAnyTopic"

Now, get the endpoint that you'll send data to. This is actually a URL that accepts mqtt data and gets it into Amazon. You will need the endpoint to actually send the data; it will show up in the code as the destination for data.

$ aws iot describe-endpoint

And you'll get back:

{
    "endpointAddress": "yournumberswillbehere.iot.us-east-1.amazonaws.com"
}

Now you have a place to send data to and a policy to handle the data. You created and named a 'thing' and it's ready to use.

But, how the heck can you tell? Go to the AWS management console; you should be getting good with the names of things by now and sign in. Look at the black bar at the top, it should be 'N. Virginia' since we set up as 'us-east-1', but if it isn't use the selection and change it. Also up at the top in the black bar is 'Services' click on it and a huge drop down of services will show up. Choose 'AWS IoT and you should get a screen like this:


There's the policy, thing and endpoint that you created using the steps above. You can click on them and see stuff related to each of them. The thing is where your shadow document will be that you'll be working with going forward.

This posting has taken hours to put together and check over, so I'm going to write about sending data from the Pi up to the thing we just created next time. 

Continued here <link>

Saturday, November 5, 2016

Amazon Dot and My Desert Home

I already warned you that this was going to get complicated and would probably drive you nuts in my first post on the Amazon Dot <link>, but you apparently didn't listen, so I'll go a bit deeper into the amazingly intricate process of hooking a Dot to my house, but first a tiny bit of background.


This complex diagram outlines the various processes and machines that get involved in a single voice request, and any piece of it can give you trouble implementing something. I got this picture directly from a nice description I ran across on Amazon, and let's start off going there to understand what is going on <link>.

Robert McCauley, the author, is unusual in that this article can be understood; at least the second or third time you read it. Ignore his comment about referring to the Amazon quick start documentation for IoT, it will only confuse you and it really doesn't talk about anything Robert doesn't. However, this article is a bit terse (meaning he passes the buck entirely) on how to use the Alexa voice service. The thing you want to learn from this is that Amazon uses mqtt to control your devices. You can actually interact with them using the AWS mqtt tool that Amazon provides and the instructions that Robert provides.

I actually created his water pump and messed with it, but I deleted it to keep from confusing myself later when I created my 'house' device. What I did was create a single device 'house' and it carries stuff like outside temperature, state of various lights, garage door position, etc. It would have been a real pain to create each of those devices and deal with them separately. The huge thing to understand here is the use of the JSON shadow device. You deal exclusively with the shadow that represents the last reported state of your device (my house). The shadow contains a 'reported' section, a 'desired' section, and a 'delta' section. For example, here is my current (as of the date above) shadow; remember it's a JSON document.

{
  "reported": {
    "humid": "37",
    "temp": "82",
    "barometer": "1016",
    "windspeed": "4",
    "winddirection": "south southwest",
    "raintoday": "0.0",
    "eastPatioLight": "off",
    "outsideLights": "off",
    "lastEntry": "isHere",
    "frontPorch": "off",
    "cactusSpot": "off",
    "outsideGarage": "off",
    "mbLight": "off",
    "gDoor2": "closed",
    "gDoor1": "closed"
  }
}

I don't have all the items I'm interested in yet, and I only voice control a few things so far, but this could be representative of anyone's implementation.  This is the reported section. If I wanted to control something, I'd put in a desired section that would contain what I wanted to change. So, if I want to turn on the outside lights:

{
  "desired": {
    "outsideLights": "on"
  }
}

That would get combined with the reported section and then inspected for differences. Any differences would show up in a new delta section. That would make the entire shadow document look like:

{
  "desired": {
    "outsideLights": "on"
  },
  "reported": {
    "humid": "37",
    "temp": "80",
    "barometer": "1016",
    "windspeed": "2",
    "winddirection": "east southeast",
    "raintoday": "0.0",
    "eastPatioLight": "off",
    "outsideLights": "off",
    "lastEntry": "isHere",
    "frontPorch": "off",
    "cactusSpot": "off",
    "outsideGarage": "off",
    "mbLight": "off",
    "gDoor2": "closed",
    "gDoor1": "closed"
  },
  "delta": {
    "outsideLights": "on"
  }
}

So you have a JSON document that holds what was last reported, what you want to change, and the difference. See why they call the sections 'reported', 'desired' and 'delta'? It actually makes sense. When you get a device hooked up to the mqtt server at Amazon, the delta portion is the only part you see coming in. That makes it easier to understand what you need to do with the actual device. So, on your Pi, or whatever, you subscribe to the Amazon mqtt service, and wait for a delta message to come in. When it does, you do what it says and report back the new status. There are a few complications in there that you'll have to deal with, but I hope to visit each of them as we go through this process.

In Robert's article, he interacts with the shadow document using Amazon's mqtt client to subscribe and publish to the shadow. That's exactly the kind of thing you will need to do in code to support your device. This stuff is handled by what's called a Lambda function. The Lambda funtion is simply code that you run on their machine that can bridge the gap between the Alexa voice service and the mqtt server to get something down to your machine to change something. In the other direction, you'll publish the latest state of your devices back so everything is kept up to date.

Now, do you understand why a little hands on with the Amazon IoT is necessary? Follow the lead from Robert's article and work with mqtt and the shadow a bit to see what is actually going on. Totally ignore his comments about Alexa, "Once you are familiar with the Alexa Skills Kit and understand how to create an Alexa skill;" That phrase is just about totally useless.

I have a suspicion that very few of Amazon's people that work on these projects actually do anything with them since their explanations leave so much out.

That's enough for you to get a feel for the interaction if you actually do it and then look at the messages that are passed in mqtt. You WILL need this when you start troubleshooting your own code.

Next post we're going to look at getting data from your device all the way back to Alexa. We won't actually try to change the state of a device because you have to have a reported section to the shadow before you can change it.

And, you probably won't need the water pump device that Robert had you create.

Continued here <link>

Tuesday, November 1, 2016

I Got One of Those New Fangled Amazon Dot Things

Obviously, this is part 1.

Yes, I took the plunge and bought an Amazon Dot before it was available and waited patiently for the release date for it to arrive. When it came in, I did the usual, "Alexa, tell me the weather." "Alexa, what time is it?" "Alexa, how tall is Scarlett Johansson?" You know, the usual.

Then, I got out the laptop and started looking at how to connect it to my house.

Let me tell you, this thing is really, really cool, but it's also the most frustrating device I have worked on in quite a while. I literally spent hours and hours on the web trying to figure out Amazon's amazingly complex and poorly documented interface to IOT (internet of things). I tried a large number of examples and tutorials out there with a singular result: they didn't work. I knew it was possible, there are youtube videos that prove it, but not a single one of them worked for me. I wrote, copied, stole thousands of lines of code and didn't get a single signal of any kind at my house. So, in a mass of frustration and disappointment, I managed to stumble (this is the third full 16 hour day of trying) across a little note on rules. It started to work part way.

Emboldened by a tiny success, I dug in and conquered the device all the way to the house, out to my patio lights and all the way back to the Alexa sitting on my kitchen counter. I had it working. No, I haven't got all the devices tied in, but I can get the weather readings taken at my house and actually turn on and off my patio light.

The problem for me, and I suspect many others, is that the interface is extremely complex and spans several of Amazon's cloud service products. For example, you speak to the Alexa and it runs many layers of recognition and permission checking that updates billing code and such before it hands something off to their Lambda service. The Lambda service runs some code that hands things off to their AWSIoT service who does some stuff and passes your signals off to Amazon's own MQTT service which will hand things off to your code at home. Your code reads this and does something returning it back to Amazon's MQTT, which passes it back to Lambda who passes it back to the Alexa service and then back to the device on the counter to be played back. In the middle of this are many, many layers of authentication, roles and permissions that have to be exactly right before things are allowed to pass. Heck, you can't even log anything without using their cloud logging service ... and you have to read it there as well.

Just forget about it being easy; it isn't. Maybe someday it will be, but it isn't right now. And, NO, I'm not going to add to the huge number of tutorials out there with my own version. Frankly, I really don't want to step through it all over again taking a hundred screen shots that will become obsolete in three or four days when they change the user interface to the various services all over again. One of the biggest problems I had was trying to work from some of the examples that were of an older interface and try finding similar capabilities on what I was looking at. You see, they have crews of people working on each piece expanding its capabilities, and even though they are very careful, things get missed. That of course means they have to fix the thing they broke and that can change the interface. It's one little guy out in the desert vs. hundreds spread around the world.

Let's talk about a specific example. Below is the session where I created the various devices using Amazon's CLI (command line interface) that I had to download and install on my Raspberry Pi. After installing it I used instructions I found out there on that world wide web thingie, and this is the console log I kept of the process. I annotated it to tell you what was actually happening after I completed the process, and it should give you a real example of what to do to set yourself up in the very beginning on the Pi:

#
# Configuring AWS ... but
# I kept the secrets that were produced by the user creation so I could use
# them here, but I forgot the step of adding permission for the user I created
# This is what happens when you forgot to add permissions to the user
#
pi@housemonitor:~/src/alexa$ aws configure
AWS Access Key ID [None]: ***Secret Stuff***
AWS Secret Access Key [None]: ***More Secret Stuff***
Default region name [None]: us-east-1
Default output format [None]:
#
# This was trying to create a 'thing' but didn't have permission 
# to do it.
#
pi@housemonitor:~/src/alexa$ aws iot create-thing --thing-name "house"

An error occurred (AccessDeniedException) when calling the CreateThing operation: User: arn:aws:iam::704749107060:user/alexaControl is not authorized to perform: iot:CreateThing on resource: arn:aws:iot:us-east-1:704749107060:thing/house
#
# After I went back and added:
#   IAMFullAccess, AWSIoTFullAccess and AWSLambdaFullAcess 
# to the user I created
#
# Then I created a 'thing' called "house"
#
pi@housemonitor:~/src/alexa$ aws iot create-thing --thing-name "house"
{
    "thingArn": "arn:aws:iot:us-east-1:704749107060:thing/house",
    "thingName": "house"
}
#
# And proved it was there by listing it
#
pi@housemonitor:~/src/alexa$ aws iot list-things
{
    "things": [
        {
            "attributes": {},
            "version": 1,
            "thingName": "house"
        }
    ]
}
#
# Custom Alexa "things" are all about security, so I generated keys
# which sent the keys to the console and scrolled the screen a bunch
#
pi@housemonitor:~/src/alexa$ aws iot create-keys-and-certificate --set-as-active --certificate-pem-outfile cert.pem --public-key-outfile publicKey.pem --private-key-outfile privkey.pem
{
    "certificateArn": "arn:aws:iot:us-east-1:704749107060:cert/e325f7755f0a4f11a59416ab7af30e20023f2c411233ecb5d2c68285843f94a4",
    "certificatePem": "-----BEGIN CERTIFICATE-----\nbunch of stuff\n-----END CERTIFICATE-----\n",
    "keyPair": {
        "PublicKey": "-----BEGIN PUBLIC KEY-----\nbunch of stuff\n-----END PUBLIC KEY-----\n",
        "PrivateKey": "-----BEGIN RSA PRIVATE KEY-----\nbunch of stuff\n-----END RSA PRIVATE KEY-----\n"
    },
    "certificateId": "even more stuff"
}
#
# But, they were in the directory I created for this
# I proved it by listing them
#
pi@housemonitor:~/src/alexa$ aws iot list-certificates
{
    "certificates": [
        {
            "certificateArn": "arn:aws:iot:us-east-1:704749107060:cert/e325f7755f0a4f11a59416ab7af30e20023f2c411233ecb5d2c68285843f94a4",
            "status": "ACTIVE",
            "creationDate": 1477339839.64,
            "certificateId": "e325f7755f0a4f11a59416ab7af30e20023f2c411233ecb5d2c68285843f94a4"
        }
    ]
}
#
# Here's the list of them
#
pi@housemonitor:~/src/alexa$ ls
cert.pem  privkey.pem  publicKey.pem
# These are the certificate, private key and public key. Keep
# the private key protected and use the public key for communication
#
# This is downloading the public key for Amazon AWS IOT
# I need this to encrypt the traffic to Amazon
#
pi@housemonitor:~/src/alexa$ wget https://www.symantec.com/content/en/us/enterprise/verisign/roots/VeriSign-Class%203-Public-Primary-Certification-Authority-G5.pem -O aws-iot-rootCA.crt
--2016-10-24 13:14:47--  https://www.symantec.com/content/en/us/enterprise/verisign/roots/VeriSign-Class%203-Public-Primary-Certification-Authority-G5.pem
Resolving www.symantec.com (www.symantec.com)... 104.100.196.23
Connecting to www.symantec.com (www.symantec.com)|104.100.196.23|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1758 (1.7K) [text/plain]
Saving to: ‘aws-iot-rootCA.crt’

aws-iot-rootCA.crt  100%[=====================>]   1.72K  --.-KB/s   in 0.001s

2016-10-24 13:14:49 (1.25 MB/s) - ‘aws-iot-rootCA.crt’ saved [1758/1758]
#
# Now I take a look at the directory to see what keys I 
# have accumulated so far
#
pi@housemonitor:~/src/alexa$ ls
aws-iot-rootCA.crt  cert.pem  privkey.pem  publicKey.pem
#
# This is where it starts to get even more complicated
# I have to have a policy in place to allow this Amazon user
# to publish data. The policy is a JSON formatted file I called
# iotpolicy.json (I totally stole that name from Nick Triantafillou)
# 
pi@housemonitor:~/src/alexa$ aws iot create-policy --policy-name "PubSubToAnyTopic" --policy-document file://iotpolicy.json
{
    "policyName": "PubSubToAnyTopic",
    "policyArn": "arn:aws:iot:us-east-1:704749107060:policy/PubSubToAnyTopic",
    "policyDocument": "{\n    \"Version\": \"2012-10-17\", \n    \"Statement\": [{\n        \"Effect\": \"Allow\",\n        \"Action\":[\"iot:*\"],\n        \"Resource\": [\"*\"]\n    }]\n}",
    "policyVersionId": "1"
}
#
# Now that a policy has been put in place up on Amazon AWS
# I have to attache it to the arn (Amazon resource name) which is 
# actually saying to attach it to the place I'll eventually send it 
# to
pi@housemonitor:~/src/alexa$  aws iot attach-principal-policy --principal "arn:aws:iot:us-east-1:704749107060:cert/e325f7755f0a4f11a59416ab7af30e20023f2c411233ecb5d2c68285843f94a4" --policy-name "PubSubToAnyTopic"
pi@housemonitor:~/src/alexa$
# I get the endpoint that I'll send data to. This is actually a
# URL that accepts mqtt data and gets it into Amazon
# I will need the endpoint to actually send the data
# This endpoint will show up in the code as the destination for data
pi@housemonitor:~/src/alexa$ aws iot describe-endpoint
{
    "endpointAddress": "ayccly8mhj4t3.iot.us-east-1.amazonaws.com"
}
#
# So I have a place to send data to and a policy to handle the data,
# I need to attach that to the 'thing' I created called 'house'
#
pi@housemonitor:~/src/alexa$ aws iot attach-thing-principal --thing-name "house" --principal  "arn:aws:iot:us-east-1:704749107060:cert/e325f7755f0a4f11a59416ab7af30e20023f2c411233ecb5d2c68285843f94a4"
pi@housemonitor:~/src/alexa$
#
# Finally, the linkage to aws is done I just have to test it.
# wish me luck

This little set of steps didn't seem to bad, but I didn't understand much of it at first. That seems to be the general rule, you poke at it until you get something to work sort of, and then you put out a tutorial or example and explain things about half way because that's all you really understand. As an example, there is a line down near the bottom of the above about attach-thing-principal. This is one line in a json file that the Amazon service holds that needs to be there. They call it a principal and you have to create it with a command they provide in the CLI. It's one line, they make it sound like something important. Well, actually it is, but only because they made it important. The Amazon interface is full of things like this.

They came up with a rather clever idea. They hold the state of your device in a regular old json file on their server. No, it isn't a database, it's just a file. This is clever because they can modify the file to represent things as they change. When I was working on getting my weather data up to Amazon I was able to add one item at a time and watch the file change as I added them. I started with the temperature (naturally) and it showed up in the file. Then I added the barometric pressure and saw it show up. I stepped through the items I wanted to check on until I got a nice little weather report I could ask for while walking around the kitchen.

Once I got that working pretty nicely, I wanted to actually change something, and this is where the way that they handle the file became really useful. You get the device to be reported and it shows up in the JSON file, then you tell alexa to change it and the file changes to show new things. It has a 'desired' entry where the new state of the device shows up, and then a 'delta' entry where the difference from the overall status and the 'desired' show up. Only the delta is sent to the Pi for it to act on. You don't have to worry about what went on before at the Pi, you just do whatever the 'delta' tells you to.

This may not sound like much, but it allows the device to go off line for periods and come back to catch up with what you last told it to do. It also provide readings for the last time the device was on line. They call this entire mess a 'shadow' and I learned to like it.

So, don't despair, over the next (however many) postings I'll tell you about parts of the system that you can step through to get your own house hooked into the Alexa. But, NO, I won't take a look at your individual implementation and show you what's wrong, nor will I answer basic questions like how do you sign on to Alexa. That stuff changes almost as fast as I could write about it. Get good at bookmarking the various pages you use on Amazon, they're hard to find a second time. Remember, I'm learning about this as I go, just like you are.

The difference is that I already put over a week into the darn thing and have actually got a light I can control with it.

Oh, before you folk ask why I went my own way instead of using Smart Things or one of the other home automation systems out there. I hate cloud services. I don't want my data stored somewhere else under someone else's control and subject to their whim. If Amazon changes their policy, I just don't use them for remote control anymore; I get out the phone and touch the screen instead. The Alexa is cool, but it won't control me or my data.

Continued here <link>

Wednesday, October 19, 2016

After a Long Testing Period, Moving Forward With My Temperature Sensor

A while back I came up with a battery operated temperature sensor that sent its data over my XBee network <link>. This little device has been sitting beside my bed for a long time now and works well. It has had it troubles over the months that I had to fix, but that was the point: make sure it works before moving it into a bigger project. Also, I use it to turn off my bedroom lights from the bed and have coupled it into turning off other things to be sure they're ready for the night as well.

I decided it was time to make some more of them, but the thought of hooking up thirty or so little fiddly wires on a protoboard kept me from actually doing it. What I needed was a custom PC board. I've designed a board before for a charger I came up with for the various lead acid batteries around the place <link>, but that project died when I found a really good battery maintainer to use commercially available. I still have one of the boards running, but it's only monitoring the battery voltage, not charging anymore. I may get back to that project at some point and come up with a simple battery monitor replacement, but right now I need an easier assembly technique and a custom board sounds like a great idea.

I dragged out Eagle <link> that had been hiding somewhere on my machine for months and updated it (of course) and started trying to use it again. Needless to say, I had to find a couple of tutorials to get me started again. I took the schematic for the sensor and came up with a board that looked like it would work, let it sit for a couple of days, revisited it and made a couple of changes then sent it off to OSH <link> which is SparkFun's old PC board service they farmed out.

I got the boards back a couple of days ago and assembled one of the three to see if it would work.


They're two inches square and purple. They're also thinner than I'm used to seeing, but that doesn't really seem to matter; they are strong enough to use.

When I got one of them loaded with components:


Yep, I'm still using the cheapest batteries I can find. I put the XBee and the Arduino side by side instead of vertically to meet a different form factor I couldn't try with the prototype. I'm still mounting all the active components in sockets so I can trade them out if necessary. I was lucky, it worked first try.

Well, that's partly a lie. The circuitry was fine and everything connected up OK, but I put the wrong profile on the XBee and it took some head scratching to find out what happened. Note to self: pay attention to what you name the profiles.

I went and got one of my famous rubber bands and packaged it up:


There's some things I might do differently on my next order, and in general. For example: if I flip the ftdi connector over to the other side of the Arduino it would make the height shorter and maybe easier to mount in a permanent enclosure. It might be good to actually include some holes for mounting the thing; I totally forgot that part. I did think of things like a place for a connector for the switch, but I put it too close to the switch to be easily connected. Lastly, more labels on the board. I had trouble telling which capacitor went where. Labels like C1, and C2 didn't tell me much and I had to keep looking at the schematic to assemble it. Of course, if you already have an example made, this problem doesn't exist, so maybe I'll just keep a good picture of it on my phone to refer to later.

The period from creating the artwork for the board and getting it back was long enough that I even forgot which way I pointed the Arduino and XBee. Sad I didn't put an arrow or something on the silk screen for the board. But, I guess we have to learn some things the hard way. At least I do. Nevertheless, IT WORKED !

I'm actually pretty happy with how it turned out. The idea of taking major components that I can buy for the most complex parts and just mounting them as modules on a board that interfaces them and has the interface components worked really well. I don't have to stock all the parts for a bare bones Arduino, I just use a cheap Arduino Pro Mini. I don't have to stock some radio parts or fiddle with RF alignment, I just use an XBee. The only parts are simple to install ones and not many of them.

Now, I have a bunch of work to do. I already had code on my Raspberry Pi to update the data base when a new temperature sensor appeared and that worked well, so I'm saving readings from two of them now (the prototype and the new one), but I'm not doing anything with it. The real objective for these is to put one in each strategic place around the house and use them to control the house temperature.

The plan is to measure the temperature and intelligently control the air handlers and compressors of my two heat pumps to adjust for warm spots in the summer and cold ones in the winter. I want to get out of the tub after a long soak in the winter and NOT freeze my butt off. I can use the air handlers to distribute warm air from the hot side of the house in the winter, and just reverse that in the summer.

I won't have thermostats at all, I'll network the entire thing and control it with an HTML interface. I may still leave a display up in the place of the thermostats since people expect to be able to look at one, but it won't have any buttons. OR, I may put a cheap tablet up on the wall with a browser running to control the entire house with.

This will mean a control board at each air handler so I can control various relays that work the fan, compressor and reversing switch that hooks into my XBee network as well. But hey, I talked a bit about that already <link> so I won't bore you until I actually start that part.

I'll need at least three more boards, and I may make the changes I talked about in the second order, but thinking about it, the changes are trivial and I can live without them. I also have other ideas about using something like this to monitor the moisture in the soil of my two new fruit trees, and as mentioned earlier, the state of the tractor batteries in the barn. See, I can put any sensor on the device and have it transmit whatever I want to my network. I may look at a motion sensor for the driveway to tell me when someone drives up. Doing that without wires would be really cool. That would mean some changes to the board for the different uses, but that also means that I GET TO MAKE CHANGES TO THE BOARD as well as try out some new sensor and code.

I'm definitely going to need some more batteries.

Tuesday, October 4, 2016

So, My Raspberry Pi Web Server Was Running Slow

I was sitting on a bar stool showing off my Pi web server that controls my house and had to wait 15-20 seconds for each screen change. Loading graphs was painfully slow and would pause in the middle with only half the graph showing. It was embarrassing. When I got home, I took a look and the load average was up in the double digits; something I had never seen before. Obviously, there was some process or other out of control that needed to be fixed. I was wrong.

Granted, it's my oldest Pi; a Pi 1 that I just keep because it's easier than bringing up my Pi3B to do the same job. I recently put a SSD on it, and it logs to a database server up in the attic, so it's been fast enough. Now, it was just crawling along.

When I went looking for what was causing it I found someone out there on the web was loading my data as fast as the process could be run. Don't misunderstand, there's nothing secret there and folk visit my site all the time to see what's different from the last time, and I've had something similar happen before; it was quite innocent. What most people don't realize is that a site that automatically updates by doing a periodic get, when put in background, will continue to update. So, you visit one of the news sites, hit the back button down on the bottom of the phone, the app disappears and you go do something else, and the app continues to run updating the screen you can't see. This can cause data overages and such, but the app is ready when you come back. This shows up in my logs as someone on the site for a very long time.

Almost all sites are polite about this auto-update and only update on a multi-minute schedule; I have my site update every 10 seconds because I want to double check and see that the garage door actually closed like it was supposed to. What was happening was someone had set up a loop that would grab the data again as soon as it was delivered. That caused a lot of database read activity and slowed the machine down a LOT. Of course, I didn't realize this at first and assumed I needed to check the efficiency of the data gathering steps.

I have a php script that gathers the data from my database and returns it to the web user called housedata.php. It's a rather simple implementation, so I took it and started timing the various operations by commenting out pieces and timing it using the 'time' command in bash. The stupid little process was taking 4.75 seconds on average to finish at first, but after some database query changes, it went way down; but all that did was allow the person out there to call my machine faster.

I looked at excluding the person by IP address using the features of the apache2 web server and succeeded in stopping the interaction quite nicely. That made me think about what else might be going on, so I took a closer look at the logs. There was the usual script kiddies trash looking for 10 year old vulnerabilities, search engines prowling around, and days worth of this person beating on my machine. I had fixed the problem, so I improved the speed of housedata.php a little more and called it done. The next morning, the person was right back in there with a slightly different IP address doing the same thing.

I added the new address to the web server exclusions and noticed that it was in a subnet of the internet provider that was being used. Ha! I excluded the entire subnet to stop the problem. The problem with excluding the IP addresses with the web server was that the web server starts a process for each hit. That takes time and machine resources, not a lot, but enough to notice over time. It looked like it was time to actually bring up a firewall to protect the little machine.

I already knew about 'iptables', but have you ever tried to use that thing? It's really hard to set up, and I could mess it up pretty badly leaving holes where there shouldn't be and locking myself out of my own machine. I shuddered at trying to get that working without a months worth of research, but then discovered 'ufw' a tool designed to help with that process. I did the dreaded apt-get update command and then an apt-get install ufw so I could try it out. Notice that I did NOT use apt-get upgrade! I'm getting really tired of having too much stuff on my machine replaced by well meaning folk out there. The last time I did that I wound up with a new slightly incompatible operating system (jessie).

Since I run headless (no keyboard or console), I was afraid of actually enabling the firewall since it would exclude port 22 and I wouldn't be able to get into the machine to do anything without dragging it to a TV set somewhere and poking around for hours sitting on the floor in front of it. Fortunately, the folk that put the package together left a note in one of the configuration files about this very thing and I did what they suggested. Gritting my teeth in expectation of failure, I started the process and it warned me that ssh sessions could be interrupted and asked for confirmation. I gritted a little harder and answered 'Y'.

It came up just fine and didn't affect the ssh session at all. I was on my way.

If you have to do this, one thing you'll find annoying is the huge amount of introductions, tutorials, promotions, and examples out there on the web that don't tell you what you want to know. Sure they tell you stuff that is valuable, but I didn't find a single one that covered what I needed to do; it was all trial and error. Painful trial and error. After about an hour I came up with an idea: look at the darn log file created by ufw to see what was going on. On the Pi, the log records are mixed in with other stuff in the file /var/log/messages. So, I set up a way to watch it and see what was happening:

tail -f /var/log/messages | grep UFW

After watching a while for the various things that were being dropped, changing the configuration, watching some more, I got it working perfectly for my purposes. I had a little annoyance with the order of the rules. See, when ufw (actually iptables, ufw is just an interface) sees a packet, it steps through its rules in order and stops when it satisfies the first one. So if you allow access to port 80 as the first line, you can't exclude a specific IP address later; it already let the packet through and stopped looking. So, put the stuff you want stopped first and then the stuff you want to allow later.

I allowed all the machines on my local network to get to the web server for various things, but only open port 80 outside the house. Here's the list I'm currently using:

pi@housemonitor:/var/log/apache2$ sudo ufw status numbered
Status: active

     To                         Action      From
     --                         ------      ----
[ 1] 22                         ALLOW IN    192.168.0.0/16
[ 2] Anywhere                   DENY IN     69.145.122.0/24
[ 3] Anywhere                   DENY IN     180.16.15.0/24
[ 4] Anywhere                   DENY IN     180.76.15.0/24
[ 5] 80                         ALLOW IN    Anywhere
[ 6] 3551                       ALLOW IN    192.168.0.0/16
[ 7] Samba4                     ALLOW IN    192.168.0.0/16
[ 8] 224.0.0.251                ALLOW IN    192.168.0.0/24
[ 9] 224.0.0.1                  ALLOW IN    192.168.0.0/24
[10] 22                         ALLOW IN    Anywhere (v6)
[11] 80                         ALLOW IN    Anywhere (v6)
[12] 3551                       ALLOW IN    Anywhere (v6)

pi@housemonitor:/var/log/apache2$
First, I allow port 22 from all internal addresses. Like I said, I worry about excluding my own access to the machine. Then a series of nets excluded because I saw them messing around. The 69.145.122.0/24 address is where the annoying traffic was coming from, the two addresses that start off with 180 are for a Chinese web crawler for a search engine. It was hitting my machine every 30 minutes from two different addresses that changed within that range. I don't mind search engines, but every 30 minutes? The IP version 6 stuff is the default, I haven't gotten to it yet.

Port 3551 is for my APC UPS that I wrote about recently. That device is working really well and controls the shutdown of all my Pi's so I want the machines to be able to interact. Of course I use Samba to move files around, so there's an entry for that. The two 224 addresses are for ARP and such, I put the entries in just to keep them out of the log.

As soon as I did that, my load level dropped to fractional numbers; the machine finally had time to actually do things.


Also, since the person out there was getting time outs from not being able to get a response from my machine, its hits dropped to every 30 seconds or so. These hits were dropped at the protocol level, so they don't cause me any problems at all.


This rather busy screen shot shows the hits roughly every 30 seconds trying to load stuff, and each of them gets dropped with no response. The really annoying thing is that this person is STILL trying to get in. I've been working on this for a few days now and this robot doesn't have the smarts to try something else. I know it's a 'bot because it goes directly after the data server code without going through the web page. After I finished with the changes I let the machine run overnight and looked again, neither the web crawler or the annoying 'bot got in.

Success !

At least so far. Sure, I'll get annoying traffic again, but now I know how to stop it and have the means to do it relatively easily.

Don't misunderstand, I don't mind people looking at the site at all; I encourage folk to take a look. It's there not only so I can close my garage doors, but to serve as an example and source of ideas and suggestions. I just ask that you don't set up scripts to mess around with it for days at a time, and until now, everyone has been really nice about it. Some of my (ahem) ideas have come from people looking around and suggesting things. I've saved many a kid from a bad grade in a computer science class because they can steal code from me. And, there have been a couple of seriously interesting term papers written based on things they found here.

I'm actually contributing ... well sort of.

One last thing came out of this exercise. I had to put in a fake 'index.html' page because of the various 'bots that want to prowl around the site. What happens is the 'bot goes for the web site and then uses the URLs inside the index page to find other stuff on the site. Then it tries various 'exploits' to break in and do something bad on each of the pages. If there is no index page, it tries to get a directory listing and from there it starts messing around. If you put in a dead end index page, it can't get a directory listing and there are no URLs in the page to leverage from; the bot gives up and moves on. The script kiddies and the constant data loading came to the attention of the web monitoring tools at my ISP and they started expiring my IP lease a couple of times a day to break up the traffic. Each time they did that, I was off the network for a short time while the DNS servers were updated. I have code in place for this kind of thing, so I didn't have to do anything, but it got annoying. We'll see if my changes remove this problem.

All in all, this was both annoying and fun. I got to learn about new stuff, make changes that work really well to some of my code and have something new to brag about when I go to the bar.

Wednesday, September 28, 2016

Raspberry Pi, More About Installing a SSD

I just finished helping a reader get an SSD (solid state drive) running on his Pi, and realized that the documentation that the Pi folk provided <link>, while correct, left some things kinda hanging for people doing this. Mainly, what to expect when you type the various commands, basically what to expect while actually doing it. Don't misunderstand, they didn't mess anything up, or leave something out, they did a great job, but some of us dweebs can get lost in the responses and such that come back.

I saved key parts of hooking up a SSD to a USB port and getting the system over to it, then booting from it, to serve, both as an extended example, and as notes to myself for next time. Yes, I read my own blog to see what the heck I did last time.

So, you get yourself an SSD and some kind of enclosure to hold it. I went to Amazon and grabbed one of the cheapest combinations I could find and used that. I wound up with two of them due to a shipping problem. Actually, this is a cute story, so I'll briefly tell you about it.

I ordered a drive and USB enclosure and then waited for it to come in. It didn't. The Amazon order said it was delivered, but it wasn't on my porch or in the box down the road. Naturally, I called Amazon and complained. They immediately refunded the money and put in an expedited order for new devices. I got the new devices in two days. That was absolutely great; I love Amazon customer service a lot.

A week later, the neighbor came over with the original package. They just moved in and forgot about it until it turned up again. I thanked him then paced around wondering whether I should tell Amazon, or just keep the drive. You guessed it, conscience got to me and I called Amazon. I had a choice, send it back (they pay shipping) or buy it. I bought it; since I actually need three of them, might as well keep this one. Sure, it's more expensive than others I can get, but I already had it in my greedy hands. That's the drive I used in the explanation below. I still want to get a really cheap one and try the entire process again with cheaper hardware, but that can wait for later.

Anyway, you have your new SSD in hand and want to put the Pi software on it and use the thing, so plug it in and partition it. Since this is the first USB drive, it will be recognized and assigned /dev/sda; if you have another drive, unplug it or figure out what the drive shows up as. Then:

pi@housemonitor:~$ sudo parted /dev/sda
GNU Parted 3.2
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) mktable msdos
(parted) mkpart primary fat32 0% 100M
(parted) mkpart primary ext4 100M 100%
(parted) print
Model: ASMT 2115 (scsi)
Disk /dev/sda: 63.4GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:
Number  Start   End     Size    Type     File system  Flags
 1      1049kB  99.6MB  98.6MB  primary  fat32        lba
 2      99.6MB  63.4GB  63.3GB  primary  ext4         lba
(parted)
Information: You may need to update /etc/fstab.
pi@housemonitor:~$
The things I typed in are bold so you can see them easily and you have to type control D to get out of parted. Folk that have done this before will wonder why I created a boot partition when I'm not going to use it. Well, I figure the work on alternate boot capabilities will continue and I want a boot partition on there in case I want to use it later. It doesn't take much space, and I have way more than I need, so why not? Occasionally I actually think ahead.

Now we have to put file systems on the new partitions. We do that with mkfs and here's what it looks like:

pi@housemonitor:~$ sudo mkfs.vfat -n BOOT -F 32 /dev/sda1
mkfs.fat 3.0.27 (2014-11-12)
pi@housemonitor:~$ sudo mkfs.ext4 /dev/sda2
mke2fs 1.42.12 (29-Aug-2014)
Creating filesystem with 15442176 4k blocks and 3866624 inodes
Filesystem UUID: f1df7b73-5872-43b8-b0da-29d6d139557d
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424
Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
pi@housemonitor:~$

This makes the filesystems on both the /BOOT and / partitions. Now you're ready to put something on there. In most cases, you want to take a running system, clone it on the new drive, and then use it without having to change a bunch of stuff. So, let's get them mounted so we can copy what we already have to the new drive.

pi@housemonitor:~$ sudo mkdir /mnt/target
pi@housemonitor:~$ sudo mount /dev/sda2 /mnt/target/
pi@housemonitor:~$ sudo mkdir /mnt/target/boot
pi@housemonitor:~$ sudo mount /dev/sda1 /mnt/target/boot/
This mounts both of the new partitions. Next, you may have to install rsync; it depends on which version of Jessie you started with. Jesssie lite doesn't come with rsync, while the full version does. I had it on one Pi and not on the other. If you need to go get it do an update first to be sure you get the latest.

sudo apt-get update

There will be a ton of output from this depending on how recently you updated. I always get lots of stuff since I rarely update the machines. Then actually install rsync:

sudo apt-get install rsync

This will also have lots of output. You've done this before, you know what you're going to get. Now, copy all the stuff from the SD card to the new drive:

sudo rsync -ax --progress / /boot /mnt/target

Not only will this produce voluminous output, it takes a good long time. If you have a minimal system like Jessie lite, you can wait around for it. If you have the full system with a lot of installed packages, take a break. I fed the dog, put on some laundry, called the telephone company and it still wasn't done.

Since all my Pi's are run headless (without a monitor or keyboard), I had to generate new keys for ssh. The easy way to do this is to mount the new partitions and pretend they are the real thing. If that's unclear, take a look at the documentation for chroot:

pi@housemonitor:~$ cd /mnt/target
pi@housemonitor:/mnt/target$ sudo mount --bind /dev dev
pi@housemonitor:/mnt/target$ sudo mount --bind /sys sys
pi@housemonitor:/mnt/target$ sudo mount --bind /proc proc
pi@housemonitor:/mnt/target$ sudo chroot /mnt/target
root@housemonitor:/#

Now, actually regenerate the keys. This warns that it will take a while to do this, but compared to the rsync above, it's nothing.

root@housemonitor:/# rm /etc/ssh/ssh_host*
root@housemonitor:/# dpkg-reconfigure openssh-server
Creating SSH2 RSA key; this may take some time ...
2048 1b:51:b1:64:a3:ad:2a:3c:3f:54:b0:7e:e5:e8:b2:76 /etc/ssh/ssh_host_rsa_key.pub (RSA)
Creating SSH2 DSA key; this may take some time ...
1024 fc:75:4e:17:77:f1:7e:4c:50:47:95:d7:94:3a:24:31 /etc/ssh/ssh_host_dsa_key.pub (DSA)
Creating SSH2 ECDSA key; this may take some time ...
256 3f:56:f8:02:92:fe:c6:a5:3d:3b:b7:58:f1:f1:2a:0b /etc/ssh/ssh_host_ecdsa_key.pub (ECDSA)
Creating SSH2 ED25519 key; this may take some time ...
256 cc:ba:5e:54:df:52:19:1c:ec:be:96:39:ca:55:3b:8a /etc/ssh/ssh_host_ed25519_key.pub (ED25519)
[ ok ] Restarting OpenBSD Secure Shell server: sshd.
root@housemonitor:/# exit
exit
pi@housemonitor:/mnt/target$

Whew ! Unmount the new stuff which will make sure it gets written out, and then we'll make changes to the boot files to complete the process.

pi@housemonitor:/mnt/target$ sudo umount dev
pi@housemonitor:/mnt/target$ sudo umount sys
pi@housemonitor:/mnt/target$ sudo umount proc
pi@housemonitor:/mnt/target$

This next part may confuse you some. I've noticed that all the examples out there don't show the mistakes we all make in following some tutorial or written process. I decided, for one window, to show what actually happened during some of the changes I made; this is where I edited the cmdline.txt and fstab files. I did a cat of the file contents before and after since the editor clears the screen of the changes:

pi@housemonitor:/mnt/target$ cd /boot
pi@housemonitor:/boot$ ls
bcm2708-rpi-b.dtb       COPYING.linux  LICENCE.broadcom
bcm2708-rpi-b-plus.dtb  fixup_cd.dat   LICENSE.oracle
bcm2708-rpi-cm.dtb      fixup.dat      overlays
bcm2709-rpi-2-b.dtb     fixup_db.dat   start_cd.elf
bcm2710-rpi-3-b.dtb     fixup_x.dat    start_db.elf
bootcode.bin            issue.txt      start.elf
cmdline.txt             kernel7.img    start_x.elf
config.txt              kernel.img     System Volume Information
pi@housemonitor:/boot$ cp cmdline.txt cmdline.sd
cp: cannot create regular file ‘cmdline.sd’: Permission denied
pi@housemonitor:/boot$ sudo !!
sudo cp cmdline.txt cmdline.sd
pi@housemonitor:/boot$ sudo vi cmdline.txt
pi@housemonitor:/boot$ cp cmdline.txt cmdline.usb
cp: cannot create regular file ‘cmdline.usb’: Permission denied
pi@housemonitor:/boot$ sudo !!
sudo cp cmdline.txt cmdline.usb
pi@housemonitor:/boot$
pi@housemonitor:/boot$ cd /mnt/target/etc/
pi@housemonitor:/mnt/target/etc$ cat fstab
proc            /proc           proc    defaults          0       0
/dev/mmcblk0p1  /boot           vfat    defaults          0       2
/dev/mmcblk0p2  /               ext4    defaults,noatime  0       1
# a swapfile is not a swap partition, no line here
#   use  dphys-swapfile swap[on|off]  for that
pi@housemonitor:/mnt/target/etc$ sudo vi fstab
pi@housemonitor:/mnt/target/etc$ cat fstab
proc            /proc           proc    defaults          0       0
/dev/mmcblk0p1  /boot           vfat    ro,auto,user,exec,async          0       2
/dev/sda2       /               ext4    defaults,noatime  0       1
# a swapfile is not a swap partition, no line here
#   use  dphys-swapfile swap[on|off]  for that
pi@housemonitor:/mnt/target/etc$
I'm constantly forgetting to use 'sudo' to edit system files. It really gets annoying making the same mistake over and over again. 

Now, the trial by fire. Notice above that I saved the cmdline.txt file for the SD card. That's just in case this doesn't work. If it fails all I have to do is copy the file back in place on a PC (remember, the BOOT partition is msdos) and I'm back to the SD card and can fix whatever I messed up. So, let's unmount the new disk and reboot the Pi:

pi@housemonitor:/mnt/target/etc$ cd
pi@housemonitor:~$ sudo umount /mnt/target/boot
pi@housemonitor:~$ sudo umount /mnt/target
pi@housemonitor:~$ sudo reboot

I usually find myself gritting my teeth at this point, and honestly, the Pi got into one of those situations where it wouldn't connect to the network. I had to walk over and cycle power to get it up. This happens to me once in a while, and the Pi will almost always come up after some period of retrying. I just got tired of watching it blink while it tried to connect. But, it came up and I did a 'df' to show the new disk was mounted and the SD card was read-only.

login as: pi
pi@192.168.0.205's password:

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Wed Sep 28 09:09:58 2016
pi@housemonitor:~$ df
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/root       60666940 3968532  53593592   7% /
devtmpfs          218416       0    218416   0% /dev
tmpfs             222688       0    222688   0% /dev/shm
tmpfs             222688    5052    217636   3% /run
tmpfs               5120       8      5112   1% /run/lock
tmpfs             222688       0    222688   0% /sys/fs/cgroup
/dev/mmcblk0p1     61384   20400     40984  34% /boot
tmpfs              44540       0     44540   0% /run/user/1000
pi@housemonitor:~$ ps ax|grep python
  455 ?        Sl     0:09 /usr/bin/python /home/pi/src/house/updateoldxively.py
  458 ?        S      0:01 /usr/bin/python /home/pi/src/house/savehouse.py
  461 ?        Sl     0:05 /usr/bin/python /home/pi/src/house/healthcheck.py
  462 ?        Sl     0:03 /usr/bin/python /home/pi/src/house/events.py
  464 ?        Sl     0:05 /usr/bin/python /home/pi/src/house/iriscontrol.py
  466 ?        S      0:01 /usr/bin/python /home/pi/src/house/mqttlogger.py
  468 ?        Sl     0:08 /usr/bin/python /home/pi/src/house/monitorhouse.py
  470 ?        Sl     0:04 /usr/bin/python /home/pi/src/house/wemocontrol.py 4
  807 pts/0    S+     0:00 grep python
pi@housemonitor:~$
I also did a 'ps' to check that the processes I usually run started automatically. Everything worked.

So folks, here's a real example of how to do this with a lot of the text that comes along with it. It took me less than a morning, even with the long rsync process. I didn't bother backing up the SD card since I was only changing the one file, cmdline.txt, on it and even that was backed up. The big advantage to me is that the SD card is fully bootable and is very unlikely to fail since it is never written to. I can use it to fix something if I need to at a minutes' notice.

Have fun.