Friday, March 13, 2015

Raspberry Pi - Read Only USB Stick - Pain in the bottom

Remember back last year when I added a USB stick to my Pi to increase reliability <link>? Serious mistake, it died. Not only did it die, I reacted wrongly to every single thing and messed myself up and had to rebuild the Pi from almost the ground up.

The story goes like this, but I don't have nice pretty examples of the output of the various efforts to fix and recover from this. I wasn't thinking about blogging it, I was trying to get my Pi house controller back up.

I tried to open the garage doors with a remote command it didn't work. I walked out to the garage, pushed the button and the door opened. OK, the door opener was working, so I took a quick look at the garage controller, it looked fine, all the little blinking lights were working. I took a look at the various house control logs and they were weird. I was getting a message that said that the USB stick was a read-only file system. It's important to note that all the devices were still controlling things perfectly. Over the years I made sure that their default operation didn't depend on the house controller, The house controller is a convenience and logging device, nothing totally depends on it. This is when I made my most serious mistake, I tried rebooting the device.

Naturally, it wouldn't boot. I took the whole mess of wires and components to the kitchen table and started trying to understand what was going on. I couldn't read the file system on a laptop since it was a Linux file system, so I went looking for something that could help, but instead I found a number of tools that wouldn't work with a USB stick. I didn't want to spend hours trying things out, so I just edited my boot SD card back the way it was before I added the USB stick. The little Pi booted first try with software a year old. Of course this messed up my data logging because things have changed a lot in the last months.

I searched the web for solutions to the read-only-USB-stick and found lots of people that had also encountered the problem, but nothing that would actually solve it. Time to look at my backups...I didn't have any! Yep, I relied on the stick to keep working and my source repositories on GitHub as my backup plan.

Fine, I'll just rebuild it. The problem was that I no longer had my source repository on the Pi, it was gone with the USB stick. If I had been coherent while the Pi was running read only, I could have just copied it off and gotten on with life. The source was there in GitHub, all I had to do was get it. So, I wiped my really old source directory from the SD card and downloaded a zip file from GitHub and installed it.

Now, I had the source, but the various libraries and packages that I've accumulated over the months weren't there. It became a chore of try something, get the missing stuff, try it again - repeat for each and every process I'd put together and modified over time.

My notes here on the blog were extremely valuable. I kept going back to see what I did in a particular instance which gave me clues to what I needed to add and modify. As I stepped through things, I even noticed a couple of bugs that hadn't given me problems before. Basically, I recreated my house controller from notes on this blog. But, my configuration file 'houserc' was a total loss and I built it up from scratch one line at a time, rediscovering the various entries I needed as processes printed error messages. I can't put that one on GitHub, it has the keys and passwords used for all the cloud servers. Eventually though, the pieces came together and everything started working again.

I managed to recreate my source repository so I could get back in sync with GitHub and update the minor changes I made to the code. That turned out to relatively easy. My objections to GitHub from the past seem really silly now. You folk should consider using it.

The USB stick is a total loss. I can't format it because it's read only, and I can't read it on a laptop because I can't format it back to DOS. Guess I'll take it apart and see what's in there.

So, did I learn anything? I learned not to trust USB sticks, they just aren't meant to be the primary device on a system. How about making backups more often? Well, I'm backed up now, but when life starts interfering, I'm sure I'll get lax about it. Automate the backup process? There's a possibility, but it costs money in the form of somewhere to back it up to, and time rotating the backups so they stay relevant. There's no really good solution, just a bunch of compromises.

I guess I'll start saving for a network appliance, but I still think the 3D printer is more important.


  1. Maybe this is an idea: I use it to backup my Raspberry based home automation system.

    You have a great blog.

    1. That's not a bad idea. I should probably look into it soon while I still remember all the pieces that need to be backed up. Funny how we forget stuff like logrotate.conf and .bashrc over time.


  2. I have a 4 TB Seagate network drive that I use for backup of photos and code etc. It draws 6 watts. I also mounted some directories to it on a couple of RPis that I use for security cameras. They write files to it every few mins.

    1. I'm really leaning toward a local big drive on the network. Which one did you get and does it work well?

      6 watts is really good. Does it get hot?

  3. It's a Seagate Central 4TB Personal Cloud Storage NAS, I just got another one the other day online on sale from Best Buy for $150.

    I got the first one last spring and I have not had any problems with it. You can ssh into it too.
    It's running Linux and you can issue some commands:

    SeagateDrive:~$ uname -a
    Linux SeagateDrive #1 Thu Apr 10 23:57:57 PDT 2014 armv6l GNU/Linux

    SeagateDrive:~$ uptime
    19:24:55 up 40 days, 22:17, load average: 2.03, 2.12, 2.12

    It was either 6w or 8w. I have a Kill-A-Watt meter, I can stick it on it again to check if you want.

    I can access it from our PCs or Linux boxes.


  4. I almost forgot, it's in my mechanical room and it runs warm, not hot at all.

    1. Thanks, I'm going to look into it. I wonder if I can put a big ol' database on it.

  5. You should be able to. You can turn a Pi into one:

    1. I'll do some serious looking. I'm sure there's a way. As for the Pi, I'd rather have a device specifically designed for this kind of thing. I originally thought I'd use a Pi for this, but the network appliances are so much faster, it really doesn't make much sense not to use one.

      Thanks again for the pointer.

  6. There may be others that will work better. Some NAS are really expensive, but they will have RAID too.

    My wife and I would like to move to Tucson in the next couple of years. We plan on trying to stay for a couple of months next winter. It's such a different climate compared to the east coast. We were just out that way last fall. The house we are in now we built years ago with lots of insulation. Here we have to worry about heat in the summer and cold in the winter. There I would guess mostly heat.

    1. I've personally seen it as low as 16F, but that only lasts a day. Usually, it's in the high 60s during the day in the coldest part of the year. Nights can freeze, but the daytime sun takes care of that. Summer, totally different animal. It's flat out hot and dry. When you come from a humid environment, expect nose bleeds during the first summer because your sinus will dry out. Your wife will be a frequent user of various creams to moisten the skin. Women here carry it in their purse all the time.

      Never move here in the summer. You need a few months of gradual temperature change to get used to the summer heat.

    2. Thanks. We have been going out that way the last 3 years, in May and once in Sept. We were in Death Valley when it was 110F & 10% humidity one day. Cooled down at night. Last fall we were at Lost Dutchman State Park and it was 106F. In the evening we went for a hike and it was OK. It is dry! Here in the winter it gets dry, it was 23% yesterday. But then it will get humid again.

      As for building, I guess they don't have basements out that way. You would think it would help keep the homes cooler.

    3. I've only seen one basement out here. I have seen one house that is almost totally underground though. It would keep them cooler, but basements didn't seem to catch on.

    4. Hi Dave.

      Check this out: I used this guide to move everything of to a USB powered Hard Disk. I am sure the guide you used to moved to USB stick was similar.

      This way you can have a very small, cheap SD card containing only the boot partition Since the boot is essentially Read-Only, it should last 'forever'. All the rest is on the USB powered external hard drive. The hard drive I used unfortunately went to sleep every now and then, and I am not a linux expert to know how to keep it awake. I will tell you that it worked fine whilst it was awake, and I would simply 'dump' the MySQL database once in a while as a backup.

      My previous reply disappeared when I Previewed, so I hope I am not duplicating...

      Cheers, Abrie.

    5. Thanks Abrie, I'm seriously considering doing exactly that, or something similar. My problem has been finding the right drive to try out. I lean toward a small SSD because it's quiet and should last forever.