Sunday, May 3, 2015

My New Network Attached Storage

Well, I did it. I finally broke down and bought a NAS (network attached Storage). I researched these things for days looking for something that would fit my needs and couldn't make up my mind. I was tempted to just take a Raspberry Pi and add a disk drive, but I wanted more capability in the device than that would have given me. I asked other folk, and it came down to two manufacturers of this kind of device: Synology and QNAP. Frankly, they both sound great and have more good reviews than bad.

I settled on QNAP, not because it won out in the specification, but because I caught a nice sale. TigerDirect had the QNAP TS-231 on sale for $30 bucks off, so I ordered it. I got two 3TB WD red drives from Amazon and set it up. It's up in my attic data center working its little heart out.

When I was looking at what could be run on it, MySQL was listed. That meant that I could put my house databases up there and get them off the Pi's I have running. It also meant that I could expand the database and actually log data locally for my house.

Wow, that would mean all my griping about cloud services would end.

First step, set up a MySQL database on the server; that took about 10 minutes. Then understand how to put data on it; that took a day. Then convert my code to use the new databases on the NAS; that took a stinking week. The very basic use of MySQL is different enough from the way to use SQLite3 that I had to learn a whole lot to get it working.

But, when I did, it really sped things up. I'm not clear on why exactly; I'm making data base transactions across an Ethernet link instead of locally, but it runs faster. Since the NAS is set up Raid 1, I have 3TB to play with and I expect the databases to use a hunk of that so it may slow down over time as the data bases grow.

My impetus for finally biting the bullet on one of these devices was storage failures on the Raspberry Pi. That's the most significant problem I've had with the Pi, the SD cards wear out, and I had worse luck using a USB stick. Well, I won't be wearing the cards out as fast, and I don't need the extra space of a USB stick anymore. However, the SD cards will still wear out because all the system logs and such are beating on them.

To this end, I'm going to put the operating system on the NAS as well, but I haven't gotten that far yet. The idea is to put everything I can on the NAS and only boot the Pi from the SD card. That should make it last a while. Sort of the best of both worlds; the separate processor and hardware of the Pi combined with the storage and reliability of a NAS. I'll inherit the possibility of things quitting if the Ethernet fails, but my house already relies on Ethernet anyway. I have the various Arduinos set up to work in a default mode, so that problem hasn't gotten any worse ... I think.

One of the side-effects (affects?? I can never remember which one to use) is that I cleaned up a bunch of code, removed things I really wasn't using any more and restructured a lot of my processes. So, most of the processes that wrote data to the various cloud services I looked into are gone now. The code is still out there on GitHub (the internet is forever), but it's not in the latest update. If you need it, just look at a previous commit.

Another thing that changed is since I'm not updating the various clouds, the live graphs I have scattered in earlier posts may not work any more. I replaced some of them with pictures to keep it somewhat coherent, but gave up after a few. Also, the architecture of the overall house control and monitoring is a bit different. I guess I'll delete the page on that and try to create a new picture and description soon.

I'm keeping my legacy feeds on Xively running for now since the majority of my data is there. I wouldn't mind losing that data, I'll make more, but I want to annoy them as long a possible with a personal feed. They've been hinting that they may remove the personal feeds for a while now, and I just plan on staying there until they kick me off.

But, this thing is so cool. It just works and accepts transactions without complaining at all. MySOL is way capable and can do things I hadn't even thought of.

So, Dave's place has Arduinos running things like the garage, pool and air conditioning that talk to a Raspberry Pi that saves it's status to a NAS. There's also a weather head that sends RF data to another Raspberry Pi that saves weather data cumulatively to the NAS as well.

The code for the new way of doing things is out on GitHub, if you want to grab something.

Unfortunately, this exercise has given me a bunch of ideas on things I can do to improve the operation. Now that I can essentially have unlimited Raspberry Pi devices doing things and coordinating activities of other devices, I can put the house configuration file on the NAS and read it from any Ethernet connected device, I can separate the web services from everything else and increase house Ethernet security. I can save data on each room using the new temperature sensor in various ways. I can ... I can go completely nuts,

If I haven't already.

8 comments:

  1. I have thought about unloading a bunch of stuff to my NAS but have stayed away from it due to it beine an older device (D-Link DNS-323) and not wanting to slow it to a crawl. I had Transmission on it but it KILLED the performance for serving media in the house and that took priority. Will be watching to see how it works out for you though. Might move the database eventually but I'm not keeping big historical numbers so it wouldn't affect me much. BTW - it's side effect. :-) You had it right....

    ReplyDelete
    Replies
    1. I literally have to look up usage for effect vs affect every time I use the darn words.

      I don't plan on running too much on the NAS, I will leave the web server it has running up and will continue to experiment with the database to see what happens. Like you, the big deal is storage that isn't likely to die on me.

      I'm not one of those folk that wants to keep tons of movies and music on the thing. My entire porn collection consists of the latest episode of Game of Thrones

      Delete
    2. I might have the entire series so far (legally obtained of course)....

      Delete
  2. I'm so glad to see my nagging about NAS paid off. Now you do need to get that UPS to protect it. QNAP is a great product not unlike a number of others out there. I have been really happy with mine. One thing I have running on mine is DOKUWIKI. I run this for guests. It has a section on WIFI, so I just give them the URL to it and they can set their own stuff up. I also have all the details about our place on it for them to browse when we are not going out for ribs or etouffe, or some other great Texas delicacy.Even with it streaming audio to my Apple devices it still runs MySQL nice and quick.

    Glenn.

    ReplyDelete
  3. One other thing I want to mention about using a NAS device is in my case I have mine setup to keep my DDNS address updated. A side benefit of this is that my NAS will also sense a power failure and report it to me. This happens because I have my DSL router connected to non UPS power. When the power goes out the NAS senses a new DDNS IP address and sends me an email with the time stamp. I also used this technique to pressure my DSL provider to give me a better DSLAM port as I could also prove that the port was going down A LOT and I could prove it from the log records and my emails. I even has the NAS send an email copy directly to the network tech. Got a new port immediately after they installed a completely new DSLAM chassis in the local CO. There are benefits with this device you just don't know about until you have one.

    Glenn.

    ReplyDelete
    Replies
    1. I'm having trouble right now with DDNS. I use changeip.com since dyndns kicked me off a while back. Just today my ip address changed and it didn't get recorded at changeip and I was off the air for a while.

      I have enough information to fix it, but just started looking at it. I may well put that code on the NAS since it's extremely light weight and would take care of the entire house.

      That'll probably be another blog post at some point.

      Delete
  4. Look closely at the DDNS capability of the QNAP software. You might be pleasantly surprised.

    ReplyDelete
    Replies
    1. I did, and the one I use is not supported by them. At some point, I may change the provider, and I'll choose one they support. But, I've got my new script written to overcome problems with the way it was done before and I should be fine from now on.

      Delete