I spent a few hours getting a Pi 3B running properly for this Grafana project. I actually think it is going to worth the effort. Grafana seems to work well and I have all the code on my own machine.
What I had to do was first set up a usb drive for the machine so I didn't depend on a silly SD card. I went into this with my teeth clamped down because previously it had been a pain to implement <link>. This time, it was a piece of cake.
What the folk did at Raspberry Pi was take the idea of a usb drive seriously. You enable a bit in the configuration, read the SD card you already have running into a file, write it out to the usb drive, plug it in and boot the Pi. OK, it's a little bit more than that, but not much.
I followed the instructions for setting the bit that allows boot from usb drive in the instructions on the Pi site <link> and found out the bit was already set. Then I remembered doing it when I was experimenting before. Then I took out the SD card, stuck it in my laptop and copied an image to a file. Next, I plugged in the usb ssd I already had from last time and copied the image over to it.
I walked across the room and plugged the usb drive into the PI, plugged in the power and went back to the laptop. Putty connected to the Pi on the first try. Notice I didn't say anything about plugging in an SD card?
Nope, the machine is running just fine without an SD card at all. You do realize that this makes the Pi into a full blown computer that you can actually rely on! Yep, this may justify making an enclosure for the combination so it looks pretty on the shelf. Right now it looks a little forlorn over there:
Looks a little strange hanging by its wires doesn't it? Notice the unopened Echo Dot boxes right by it? That's for another project I hope to get to some day.
I created a special user in the database that can only read data from certain tables to protect against the scary "SQL injection" attacks by hackers so my database is OK. I forwarded a port to the machine from my router so it can be seen from outside when I finally trust it enough to do so; I may be ready to show this thing off in a few days.
Then I spent a little time adding a few new items to the Grafana dashboard so I had more things I could check on just by glancing at the display. It looks like this right now:
I have the appliance power usage and battery level of the room temperature sensors charted so I can follow them over time. There's other things I can chart for fun and some other indicators that would be nice to see from time to time, but I think I'll use this arrangement for a while to see how it feels.
This entire project has been a success. I guess it was time for my readers to nudge me again about using some of the public tools. Remember this all started when I wrote about Xively a little while ago <link>. Maybe I'll look at Home Assistant next.
Or maybe I'll just make cool charts for a week or two.
I live in the Arizona Desert, Southwestern USA. It gets hot here, and my power bills got out of hand. This is a journal of my various efforts to bring this problem under control using the cheapest technology I could find. Saving money shouldn't cost a fortune.
Showing posts with label Charting Data. Show all posts
Showing posts with label Charting Data. Show all posts
Friday, February 9, 2018
Thursday, February 8, 2018
My Freezer: Right in the middle of installing Grafana
So, I'm installing Grafana on my laptop and pull up a chart that looks like this for my monitored appliances:
What the heck? the freezer in the house is getting hot! About a minute later, an email comes in that I actually check since I'm setting right there and it tells me that the house freezer is over 50 F.
Yes, my freezer sends me email. I set this up about 4 months ago when the door didn't get closed completely and was open for HOURS before I noticed it. It didn't cost me much to replace the stuff that had thawed out, I cooked it instead, but having it happen got me to thinking about alarming that kind of thing. I have a process called 'healthcheck.py' that I run as a daemon (all the time) that monitors the other processes and many of the devices that are reporting around the house. Things like the XBee coordinator process, the thermostats and the septic tank float are watched for problems and an email is sent every hour that they don't perform up to snuff. I just added code to this process to watch the temperature in the appliances to make sure they were OK.
I got up, walked over to the freezer, put the frozen french fries back on the shelf correctly, and closed the door. Problem solved and crises averted. All the scoffers out there that poo-poo'd me about putting a temperature sensor in the appliances ... No, none of the other folk that are automating and monitoring their houses had disparaging comments, it was neighbors and friends that just let the house react to things and then complain wondered why I bothered with that much work.
Really ?
What was really unexpected though was the aftermath of letting the freezer warm up like that. Take a look at this chart of the aftermath:
I suppressed the other appliances and expanded this to show a little normal operation on both sides. The big deal is how long it took for the freezer to return to normal. Because I left the door open for about two and a half hours the temperature went all the way from below zero up to 63 degrees. Then it took over 24 hours for the thing to get back to normal.
Wow, I gotta pay more attention to that door. What the heck was happening with the power usage during this time?
Here's the power chart for the same period, and I see that shortly after I closed the door the freezer went into a defrost cycle. That's a good thing because there had to be a bunch of ice all over the evaporator coil. The compressor was on trying cool a freezer with an open door and humidity from the house was freezing all over the coil lowering its efficiency. Then the ice maker took off to fill the empty ice bucket that I had emptied of sludge. The compressor ran for the entire 24 hour period getting the internal temperature back down where it should be, but the defrost cycles and ice maker ate some of the energy during this time.
So, the power bill will be a little higher this month, but notice that the freezer only uses around 150 watts. It won't cost me much at all. Once I started watching the appliances, it really impressed me how little power they use. A little Honda generator could power the appliances around my house for an extended power failure. That will almost certainly become a project in the future.
There's a couple of other things I want to point out to readers. First, the reason I left the door open was that I was playing with Grafana and got distracted. Second, the graph I was playing with actually told me something useful: the stupid freezer was getting hot. Third, I was immediately able to research the results of my screw up. What went on with the temperature recovery in the freezer and how that affected my power usage.
That is so stinkin' COOL. (pun intended)
So, I really recommend saving data for some period. I've never gone back more than a few months looking for some change or problem, but as shown here, a couple of weeks can really be valuable. I personally carry years of data, but that doesn't mean I have to; it also slows down my database queries. So, balance your own needs to what you want to do. Over time, you'll work it out.
Also, Grafana allowed me to look at different time frames and data really quickly. It took me just a few minutes to prowl through the data to understand what was going on. I could have done the exact same thing with other charting tools I've tried, but it would have taken me way longer. Man I'm glad they implemented MySQL in this tool.
I probably should go clean the freezer now, but at least it's working.
What the heck? the freezer in the house is getting hot! About a minute later, an email comes in that I actually check since I'm setting right there and it tells me that the house freezer is over 50 F.
Yes, my freezer sends me email. I set this up about 4 months ago when the door didn't get closed completely and was open for HOURS before I noticed it. It didn't cost me much to replace the stuff that had thawed out, I cooked it instead, but having it happen got me to thinking about alarming that kind of thing. I have a process called 'healthcheck.py' that I run as a daemon (all the time) that monitors the other processes and many of the devices that are reporting around the house. Things like the XBee coordinator process, the thermostats and the septic tank float are watched for problems and an email is sent every hour that they don't perform up to snuff. I just added code to this process to watch the temperature in the appliances to make sure they were OK.
I got up, walked over to the freezer, put the frozen french fries back on the shelf correctly, and closed the door. Problem solved and crises averted. All the scoffers out there that poo-poo'd me about putting a temperature sensor in the appliances ... No, none of the other folk that are automating and monitoring their houses had disparaging comments, it was neighbors and friends that just let the house react to things and then complain wondered why I bothered with that much work.
Really ?
What was really unexpected though was the aftermath of letting the freezer warm up like that. Take a look at this chart of the aftermath:
I suppressed the other appliances and expanded this to show a little normal operation on both sides. The big deal is how long it took for the freezer to return to normal. Because I left the door open for about two and a half hours the temperature went all the way from below zero up to 63 degrees. Then it took over 24 hours for the thing to get back to normal.
Wow, I gotta pay more attention to that door. What the heck was happening with the power usage during this time?
Here's the power chart for the same period, and I see that shortly after I closed the door the freezer went into a defrost cycle. That's a good thing because there had to be a bunch of ice all over the evaporator coil. The compressor was on trying cool a freezer with an open door and humidity from the house was freezing all over the coil lowering its efficiency. Then the ice maker took off to fill the empty ice bucket that I had emptied of sludge. The compressor ran for the entire 24 hour period getting the internal temperature back down where it should be, but the defrost cycles and ice maker ate some of the energy during this time.
So, the power bill will be a little higher this month, but notice that the freezer only uses around 150 watts. It won't cost me much at all. Once I started watching the appliances, it really impressed me how little power they use. A little Honda generator could power the appliances around my house for an extended power failure. That will almost certainly become a project in the future.
There's a couple of other things I want to point out to readers. First, the reason I left the door open was that I was playing with Grafana and got distracted. Second, the graph I was playing with actually told me something useful: the stupid freezer was getting hot. Third, I was immediately able to research the results of my screw up. What went on with the temperature recovery in the freezer and how that affected my power usage.
That is so stinkin' COOL. (pun intended)
So, I really recommend saving data for some period. I've never gone back more than a few months looking for some change or problem, but as shown here, a couple of weeks can really be valuable. I personally carry years of data, but that doesn't mean I have to; it also slows down my database queries. So, balance your own needs to what you want to do. Over time, you'll work it out.
Also, Grafana allowed me to look at different time frames and data really quickly. It took me just a few minutes to prowl through the data to understand what was going on. I could have done the exact same thing with other charting tools I've tried, but it would have taken me way longer. Man I'm glad they implemented MySQL in this tool.
I probably should go clean the freezer now, but at least it's working.
Tuesday, February 6, 2018
Grafana: Now to get it working on a Raspberry Pi
Last post I talked about how I got Grafana working on my desktop talking to my database. It actually worked pretty well. Unfortunately, the laptop is NOT where I want it to run; I want it to be running on one of my Raspberry Pi machines. Since I have one of the Pi 3b's and, it isn't doing much of anything, I put it there.
Like almost everything related to linux, this was a pain. It seems that since I last played with that little machine, there have been two major updates to the OS. They went from Jessie to Pixel to Stretch. WTF?? Are the folks at Raspberry Pi running out of things to do?
Fine, I downloaded the minimal image of Stretch and wrote it onto an SD card, made a couple of changes to the boot init files to run headless and booted it up. That worked first try. They really did make some improvements in that area. No gritting your teeth and closing your eyes before rebooting; it just worked. Then I went looking for a Raspberry Pi version of Grafana ... Guess what? Grafana doesn't make one ! Fortunately, there's a person out there that took this job on for the rest of us. The github user 'fg2it' has created a github repository that holds Grafana for the various versions of Pi OS's and it's relatively easy to use them --- after you figure out what the heck he is doing. The repository he created is here <link>, and in the instructions he does a cool trick of updating where Grafana is installed from, then used apt-get to install it. I stumbled over that for a while before I got it working. Really, really clever way to do it.
Aside: I ran into instruction after instruction for installing Grafana on the Pi, and almost all of them referenced back to fg2it's repository but, tried to take the credit for all the work. The slimeballs only referenced fg2it in the various commands. You'll see what I mean when you go looking around. Thank you fg2it.
I installed the 'Stretch' version of Grafana on the Pi I had just updated and made the simple changes necessary and it worked. I could log into Grafana and create charts that I copied from the laptop version I had used previously. Everything worked pretty well. Next, I added some more charts and explored a bit. Here's the way it looks right this minute when I load the graphs up on my laptop:
This is produced by Grafana running on a Raspberry Pi displaying on a Windows laptop and getting stored data from a MySQL database running on a QNAP network appliance. Cool !
Yes, since it's only a web page, it works on a phone as well:
You do have to scroll down on the screen to see all four of the charts, but that's the way it should work on a small device.
There's a ton of features I haven't played with yet, and I'll get to some of those over the next few days, but first I want to protect my database and figure out some way of offering this up on the web. It may get complicated since this machine can't be seen from out there.
Also, this machine only has an SD card on it. I had to leave the usb solid state drive out of the installation to get this much running. That will have to change since I've never had good luck with the SD cards. Three - four months and they crap out leaving you with a days work getting it running again.
Note that I have no dependency on any cloud service at all. It's my database, and all the software runs right here in the house.
Take that Xively !
Like almost everything related to linux, this was a pain. It seems that since I last played with that little machine, there have been two major updates to the OS. They went from Jessie to Pixel to Stretch. WTF?? Are the folks at Raspberry Pi running out of things to do?
Fine, I downloaded the minimal image of Stretch and wrote it onto an SD card, made a couple of changes to the boot init files to run headless and booted it up. That worked first try. They really did make some improvements in that area. No gritting your teeth and closing your eyes before rebooting; it just worked. Then I went looking for a Raspberry Pi version of Grafana ... Guess what? Grafana doesn't make one ! Fortunately, there's a person out there that took this job on for the rest of us. The github user 'fg2it' has created a github repository that holds Grafana for the various versions of Pi OS's and it's relatively easy to use them --- after you figure out what the heck he is doing. The repository he created is here <link>, and in the instructions he does a cool trick of updating where Grafana is installed from, then used apt-get to install it. I stumbled over that for a while before I got it working. Really, really clever way to do it.
Aside: I ran into instruction after instruction for installing Grafana on the Pi, and almost all of them referenced back to fg2it's repository but, tried to take the credit for all the work. The slimeballs only referenced fg2it in the various commands. You'll see what I mean when you go looking around. Thank you fg2it.
I installed the 'Stretch' version of Grafana on the Pi I had just updated and made the simple changes necessary and it worked. I could log into Grafana and create charts that I copied from the laptop version I had used previously. Everything worked pretty well. Next, I added some more charts and explored a bit. Here's the way it looks right this minute when I load the graphs up on my laptop:
This is produced by Grafana running on a Raspberry Pi displaying on a Windows laptop and getting stored data from a MySQL database running on a QNAP network appliance. Cool !
Yes, since it's only a web page, it works on a phone as well:
You do have to scroll down on the screen to see all four of the charts, but that's the way it should work on a small device.
There's a ton of features I haven't played with yet, and I'll get to some of those over the next few days, but first I want to protect my database and figure out some way of offering this up on the web. It may get complicated since this machine can't be seen from out there.
Also, this machine only has an SD card on it. I had to leave the usb solid state drive out of the installation to get this much running. That will have to change since I've never had good luck with the SD cards. Three - four months and they crap out leaving you with a days work getting it running again.
Note that I have no dependency on any cloud service at all. It's my database, and all the software runs right here in the house.
Take that Xively !
Thursday, May 22, 2014
Hooking HighCharts Into My House
If you prowl around this blog very long you'll run into a bunch of charts. I've experimented with several cloud services for storing data and shown examples of my own data using their charting provisions. All of them are cool, but some of them leave a little bit to be desired. For example, Xively uses Rickshaw <link>, and it's a nice package, but the Xively implementation is too darn complex for my taste. If you look at the various cloud providers, they all have their favorites.
Up until now I've been using the Google graph API for my own stuff, but the problem is that Google Graph uses Flash. Cell phones don't like Flash very much. That means my home graphs don't work on my phone. I absolutely can't let that continue any longer, and since my favorite cloud provider, Grovestreams <link>, uses HighCharts <link>, I stole their example and developed my own graph using the same library.
Hey, they stole my idea for the SteelSeries gauges, turn about is only fair play.
But, as usual, it was an exercise in patience. The documentation on HighCharts is voluminous; it goes on forever and ever with tons of examples and links into JFiddle <link> so the ideas and stuff can be experimented with; talk about information overload. It's also tough to figure out how the set the various properties in the right place to get things done, and none of the examples did exactly what I needed. That's why patience was required. I didn't want to get disgusted with it and just give up.
I finally got one working. If you put your pointer inside the chart, you can get the actual reading of any point. If you click, drag across the chart, you can zoom in and see it closer. If you have a touch screen, you can use the 'pinch' and 'expand' to zoom in and out; when your zoomed in, you can slide the chart right and left to see the data. By clicking on the labels at the bottom, you can turn off one of the series to make the remaining one more clear. I could put up a lot of things to examine and choose which one by turning the others off. I'm not sure I like the colors yet, but they're easy to change.
The beauty of this chart is that it's totally javascript. That means it works on the phone, so I can add it to the Android app I have:
OK, the chart looks silly. But look what happens when you turn the phone into portrait mode:
Is that slick or what? Using the touchscreen I can zoom around it and select points to my hearts content.
The data shown is taken from my legacy feed on Xively and was reasonably easy to grab. The only real problems I had getting it going was prowling through the documentation on HighCharts; it was all there, but like finding a needle in a haystack.
If you want to steal my example and modify it for your own use, it's on GitHub, drop a comment here and I'll post the link.
Up until now I've been using the Google graph API for my own stuff, but the problem is that Google Graph uses Flash. Cell phones don't like Flash very much. That means my home graphs don't work on my phone. I absolutely can't let that continue any longer, and since my favorite cloud provider, Grovestreams <link>, uses HighCharts <link>, I stole their example and developed my own graph using the same library.
Hey, they stole my idea for the SteelSeries gauges, turn about is only fair play.
But, as usual, it was an exercise in patience. The documentation on HighCharts is voluminous; it goes on forever and ever with tons of examples and links into JFiddle <link> so the ideas and stuff can be experimented with; talk about information overload. It's also tough to figure out how the set the various properties in the right place to get things done, and none of the examples did exactly what I needed. That's why patience was required. I didn't want to get disgusted with it and just give up.
I finally got one working. If you put your pointer inside the chart, you can get the actual reading of any point. If you click, drag across the chart, you can zoom in and see it closer. If you have a touch screen, you can use the 'pinch' and 'expand' to zoom in and out; when your zoomed in, you can slide the chart right and left to see the data. By clicking on the labels at the bottom, you can turn off one of the series to make the remaining one more clear. I could put up a lot of things to examine and choose which one by turning the others off. I'm not sure I like the colors yet, but they're easy to change.
The beauty of this chart is that it's totally javascript. That means it works on the phone, so I can add it to the Android app I have:
OK, the chart looks silly. But look what happens when you turn the phone into portrait mode:
Is that slick or what? Using the touchscreen I can zoom around it and select points to my hearts content.
The data shown is taken from my legacy feed on Xively and was reasonably easy to grab. The only real problems I had getting it going was prowling through the documentation on HighCharts; it was all there, but like finding a needle in a haystack.
If you want to steal my example and modify it for your own use, it's on GitHub, drop a comment here and I'll post the link.
Tuesday, July 2, 2013
Using the Google Charts API with Xively (Pachube -> Cosm -> Xively)
It's actually getting a little tiring having the old Pachube change names and interfaces every year. However, they have programmers there and they have to be doing something ... right? I decided, based on a request from a follower, to put the code I have been using to collect data that is stored on Xively and hand it off to the Google Graph API. This gives me a nice graph that I can look through to inspect my energy usage.
I've used the Google graphs to find problems with my power usage and distribution for a couple of years now. Nice system. There are other graphing systems out there, but each of them is different from the others and one has to settle on something to work with. I just chose Google, not because it's the best, but because I think it might continue to exist for a while and has reasonable documentation. Oh, it's also free.
At any rate, most of the graphing software takes a table as input. That presents a particular problem for us inexperienced web programmers. What the heck is a table and how do we make one? Once we understand that to some degree, how do we get the data from Xively and put it into a table? This little bit of knowledge is hard to come by. I went through several iterations of trying to get data, format it into something to give to the Google API and see what was displayed. I came up with this:
This is the actual web page I created to display a days worth of data on power usage and outside temperature taken from my sensors. To use it, just copy the code into a file, save it as html (htm, whatever) and then click on it. It'll load my data and give you a graph you can play with. Then, modify it to use your feed id, key, and other stuff to display your own data. It can serve as a template for your own graph as well.
If you want to use it as part of a web page with other information, pictures, etc, just put it into an iframe and include the page. I keep this page on Dropbox so I can use it from anywhere, even my tablet. Some browsers have a little trouble with iframes (chrome) others have trouble with Google charts (Opera, Silk), so your mileage may vary, but it can certainly serve as an example.
This is what you will get with a line like:
<iframe width="100%" height="325" scrolling="No" seamless frameBorder="0"
src="http://dl.dropboxusercontent.com/u/128855213/CosmGraphPower.html"> </iframe>

And yes, the src attribute up there points to the page I have on Dropbox to show this page. Feel free to grab the source directly from there as well.
As I said before, my web programming experience is slim, so there may well be better, more elegant ways to do this. If so, leave a comment for other folks (and me!) to help make our lives simpler. For example, you have no idea what a pain in the butt it was to get this page to work using the tools that Blogger provides. Their editors kept changing things and messing up the display. I finally got it to work, but I haven't tried it on multiple browsers so, it may be messed up. Look at it in FireFox, it seems to be the best with this display.
I've used the Google graphs to find problems with my power usage and distribution for a couple of years now. Nice system. There are other graphing systems out there, but each of them is different from the others and one has to settle on something to work with. I just chose Google, not because it's the best, but because I think it might continue to exist for a while and has reasonable documentation. Oh, it's also free.
At any rate, most of the graphing software takes a table as input. That presents a particular problem for us inexperienced web programmers. What the heck is a table and how do we make one? Once we understand that to some degree, how do we get the data from Xively and put it into a table? This little bit of knowledge is hard to come by. I went through several iterations of trying to get data, format it into something to give to the Google API and see what was displayed. I came up with this:
The Web Page
<html>
<title>Desert Home Day Graph</title>
<head>
</head>
<body>
<!--somehow, SUN thought this was better than the c include statement....sigh -->
<!--These are the google api files that you need for their tables and graphs -->
<script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.3.2/jquery.min.js"></script>
<script type='text/javascript' src='https://www.google.com/jsapi'></script>
<script type='text/javascript'>
google.load('visualization', '1', {'packages':['annotatedtimeline']});
</script>
<script type="text/javascript">
//I stole this function almost verbatim from the Pachube web site.
//It formats the date into a form I like
function adjustdateformat(incoming){
if (typeof(incoming) != "function") { //no scripts allowed in data
var ts_parts = incoming.split('T');
var d_parts = ts_parts[0].split('-');
var t_parts = ts_parts[1].split(':');
var d = new Date(Date.UTC(parseInt(d_parts[0],10), parseInt(d_parts[1],10)-1 ,parseInt(d_parts[2],10), parseInt(t_parts[0],10), parseInt(t_parts[1],10), parseInt(t_parts[2],10) ) );
return(d);
} else {
alert("There was a script in the data !!");
return(0); //for when there is a script
}
}
// this creates the table that will be stuffed with data returned
var data = new google.visualization.DataTable();
// The google data api doesn't have a call to return a column id by name
// so I had to invent this one
function getColumnIdByName(name){
var idx;
var thisname;
var lastOne = data.getNumberOfColumns();
for(idx = 0; idx < lastOne; idx++){
if(name.toString() == data.getColumnLabel(idx)){
return(idx);
}
}
// I really don't like magic values like this, but it'll do for now.
return(99);
}
// This parses into the json data to grab values.
// there's documentation all over the web on the json format
// but it has to be parsed to use it
function climbtree(archivedData){
// this will create the table for the graph
if (typeof(archivedData.datastreams[0].datapoints) == 'undefined'){
alert("No archived data returned");
return;
}
$.each(archivedData.datastreams, function(key, value){
// key 0 has the first datastream and the times for the table.
// you'll always get this if there is data to deliver
if(key == 0){ // only add a date column for the first datastream
//alert("I got key 0 ");
if (data.getNumberOfColumns() == 0){
data.addColumn('datetime', 'Date');
data.addColumn('number', value.tags);
}
$.each(value.datapoints, function(key,value){
//alert("got an at value of" + this.at);
var d = adjustdateformat(value.at);
var level = value.value;
var thisrow = data.addRow();
data.setValue(thisrow, 0, d);
data.setValue(thisrow, 1, parseFloat(level));
})
} else {
//second, third, etc datastreams
//alert("got another key = " + value.tags);
var columnIdx;
if((columnIdx = getColumnIdByName(value.tags)) == 99){
columnIdx = data.addColumn('number', value.tags); //new datastream came in
}
$.each(value.datapoints, function(key,value){
var d = adjustdateformat(value.at); //format its datestamp the same way
var rowIdx = data.getFilteredRows([{column:0,value:d}]); //and find it in the table
var level = parseFloat(value.value);
data.setValue(rowIdx[0], columnIdx, level);
})
}
})
}
</script>
<!-- this actually goes and gets the data. What I'm doing is grabbing the data
for two items: Real Power and Outside Temperature. Since Cosm has a limit on
the number of items they will return, I do it in six hour intervals. So, a day
will loop four times and two days 8, etc. There's also a limit on other stuff,
so the most you can get is around a week. If you want more, you'll have to do
some more work to handle it. Remember though, there's a lot of data here and
it will take some time to get it back from the site. That's why I prefer server
generated charts. The server has the data right there and can access it quickly
when we go get it, it takes time for the chunks to come to us.
-->
<script type="text/javascript">
$(document).ready(function(){
var url='http://api.pachube.com/v2/feeds/9511.json?&key=GtGuoMKJSqv2tzqGvWaITLhlEDUxrYXSixqPSmlyj-s&per_page=1000&datastreams=0,7&duration=6hours&interval=60';
var databack;
var startpoint = '';
var loopcount;
if(queryString["day"] == undefined)
loopcount = 4;
else
loopcount = (queryString["day"] * 4) + 1;
//alert('loopcount = ' + loopcount);
for ( var i=0; i<loopcount; i++ ){
//alert('going for ' + url + startpoint);
$.ajax({
type: 'GET',
url: url + startpoint,
dataType: 'json',
data: {},
async: false, // you have to have this so the data will arrive before display
success: function(dataarchive) {
//alert("the url should have returned");
if (databack = (typeof(dataarchive.datastreams[0].datapoints) != 'undefined')){
var len = dataarchive.datastreams[0].datapoints.length;
//alert("got back " + len + " items");
//alert("from " + dataarchive.datastreams[0].datapoints[0].at +
// "to " + dataarchive.datastreams[0].datapoints[len - 1].at);
startpoint = "&end=" + dataarchive.datastreams[0].datapoints[0].at;
climbtree(dataarchive);
};
}
});
}
//alert('after the get loop');
var chart = new google.visualization.AnnotatedTimeLine(document.getElementById('chartholder'));
// OK, now I've got a chart, but need to rearrange the columns
// this is a matter of taste, so adjust as needed.
var chartView = new google.visualization.DataView(data);
// this sets the order of the columns
chartView.setColumns([0,2,1]);
chart.draw(chartView, {displayAnnotations: true, 'scaleType':'allfixed','scaleColumns': 'allmaximized', 'scaleColumns': [0,1]});
});
</script>
<!--
This little thing allows you to put a parameter in the URL to get
multiple days. Use a ? then the number of days i.e. ?day=2 will get
you two days worth of data.
-->
<script type="text/javascript">
var queryString = new Array();
var parameters = window.location.search.substring(1).split('&');
for (var i=0; i<parameters.length; i++) {
var pos = parameters[i].indexOf('=');
// If there is an equal sign, separate the parameter into the name and value,
// and store it into the queryString array.
if (pos > 0) {
var paramname = parameters[i].substring(0,pos);
var paramval = parameters[i].substring(pos+1);
queryString[paramname] = unescape(paramval.replace(/\+/g,' '));
}
else {
//special value when there is a querystring parameter with no value
queryString[parameters[i]]="[nil]"
}
}
</script>
<!-- and the actual chart declaration; change this for size and stuff -->
<div id='chartholder' style='width:100%; height:310px;'> </div>
</body>
</html>
<title>Desert Home Day Graph</title>
<head>
</head>
<body>
<!--somehow, SUN thought this was better than the c include statement....sigh -->
<!--These are the google api files that you need for their tables and graphs -->
<script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.3.2/jquery.min.js"></script>
<script type='text/javascript' src='https://www.google.com/jsapi'></script>
<script type='text/javascript'>
google.load('visualization', '1', {'packages':['annotatedtimeline']});
</script>
<script type="text/javascript">
//I stole this function almost verbatim from the Pachube web site.
//It formats the date into a form I like
function adjustdateformat(incoming){
if (typeof(incoming) != "function") { //no scripts allowed in data
var ts_parts = incoming.split('T');
var d_parts = ts_parts[0].split('-');
var t_parts = ts_parts[1].split(':');
var d = new Date(Date.UTC(parseInt(d_parts[0],10), parseInt(d_parts[1],10)-1 ,parseInt(d_parts[2],10), parseInt(t_parts[0],10), parseInt(t_parts[1],10), parseInt(t_parts[2],10) ) );
return(d);
} else {
alert("There was a script in the data !!");
return(0); //for when there is a script
}
}
// this creates the table that will be stuffed with data returned
var data = new google.visualization.DataTable();
// The google data api doesn't have a call to return a column id by name
// so I had to invent this one
function getColumnIdByName(name){
var idx;
var thisname;
var lastOne = data.getNumberOfColumns();
for(idx = 0; idx < lastOne; idx++){
if(name.toString() == data.getColumnLabel(idx)){
return(idx);
}
}
// I really don't like magic values like this, but it'll do for now.
return(99);
}
// This parses into the json data to grab values.
// there's documentation all over the web on the json format
// but it has to be parsed to use it
function climbtree(archivedData){
// this will create the table for the graph
if (typeof(archivedData.datastreams[0].datapoints) == 'undefined'){
alert("No archived data returned");
return;
}
$.each(archivedData.datastreams, function(key, value){
// key 0 has the first datastream and the times for the table.
// you'll always get this if there is data to deliver
if(key == 0){ // only add a date column for the first datastream
//alert("I got key 0 ");
if (data.getNumberOfColumns() == 0){
data.addColumn('datetime', 'Date');
data.addColumn('number', value.tags);
}
$.each(value.datapoints, function(key,value){
//alert("got an at value of" + this.at);
var d = adjustdateformat(value.at);
var level = value.value;
var thisrow = data.addRow();
data.setValue(thisrow, 0, d);
data.setValue(thisrow, 1, parseFloat(level));
})
} else {
//second, third, etc datastreams
//alert("got another key = " + value.tags);
var columnIdx;
if((columnIdx = getColumnIdByName(value.tags)) == 99){
columnIdx = data.addColumn('number', value.tags); //new datastream came in
}
$.each(value.datapoints, function(key,value){
var d = adjustdateformat(value.at); //format its datestamp the same way
var rowIdx = data.getFilteredRows([{column:0,value:d}]); //and find it in the table
var level = parseFloat(value.value);
data.setValue(rowIdx[0], columnIdx, level);
})
}
})
}
</script>
<!-- this actually goes and gets the data. What I'm doing is grabbing the data
for two items: Real Power and Outside Temperature. Since Cosm has a limit on
the number of items they will return, I do it in six hour intervals. So, a day
will loop four times and two days 8, etc. There's also a limit on other stuff,
so the most you can get is around a week. If you want more, you'll have to do
some more work to handle it. Remember though, there's a lot of data here and
it will take some time to get it back from the site. That's why I prefer server
generated charts. The server has the data right there and can access it quickly
when we go get it, it takes time for the chunks to come to us.
-->
<script type="text/javascript">
$(document).ready(function(){
var url='http://api.pachube.com/v2/feeds/9511.json?&key=GtGuoMKJSqv2tzqGvWaITLhlEDUxrYXSixqPSmlyj-s&per_page=1000&datastreams=0,7&duration=6hours&interval=60';
var databack;
var startpoint = '';
var loopcount;
if(queryString["day"] == undefined)
loopcount = 4;
else
loopcount = (queryString["day"] * 4) + 1;
//alert('loopcount = ' + loopcount);
for ( var i=0; i<loopcount; i++ ){
//alert('going for ' + url + startpoint);
$.ajax({
type: 'GET',
url: url + startpoint,
dataType: 'json',
data: {},
async: false, // you have to have this so the data will arrive before display
success: function(dataarchive) {
//alert("the url should have returned");
if (databack = (typeof(dataarchive.datastreams[0].datapoints) != 'undefined')){
var len = dataarchive.datastreams[0].datapoints.length;
//alert("got back " + len + " items");
//alert("from " + dataarchive.datastreams[0].datapoints[0].at +
// "to " + dataarchive.datastreams[0].datapoints[len - 1].at);
startpoint = "&end=" + dataarchive.datastreams[0].datapoints[0].at;
climbtree(dataarchive);
};
}
});
}
//alert('after the get loop');
var chart = new google.visualization.AnnotatedTimeLine(document.getElementById('chartholder'));
// OK, now I've got a chart, but need to rearrange the columns
// this is a matter of taste, so adjust as needed.
var chartView = new google.visualization.DataView(data);
// this sets the order of the columns
chartView.setColumns([0,2,1]);
chart.draw(chartView, {displayAnnotations: true, 'scaleType':'allfixed','scaleColumns': 'allmaximized', 'scaleColumns': [0,1]});
});
</script>
<!--
This little thing allows you to put a parameter in the URL to get
multiple days. Use a ? then the number of days i.e. ?day=2 will get
you two days worth of data.
-->
<script type="text/javascript">
var queryString = new Array();
var parameters = window.location.search.substring(1).split('&');
for (var i=0; i<parameters.length; i++) {
var pos = parameters[i].indexOf('=');
// If there is an equal sign, separate the parameter into the name and value,
// and store it into the queryString array.
if (pos > 0) {
var paramname = parameters[i].substring(0,pos);
var paramval = parameters[i].substring(pos+1);
queryString[paramname] = unescape(paramval.replace(/\+/g,' '));
}
else {
//special value when there is a querystring parameter with no value
queryString[parameters[i]]="[nil]"
}
}
</script>
<!-- and the actual chart declaration; change this for size and stuff -->
<div id='chartholder' style='width:100%; height:310px;'> </div>
</body>
</html>
This is the actual web page I created to display a days worth of data on power usage and outside temperature taken from my sensors. To use it, just copy the code into a file, save it as html (htm, whatever) and then click on it. It'll load my data and give you a graph you can play with. Then, modify it to use your feed id, key, and other stuff to display your own data. It can serve as a template for your own graph as well.
If you want to use it as part of a web page with other information, pictures, etc, just put it into an iframe and include the page. I keep this page on Dropbox so I can use it from anywhere, even my tablet. Some browsers have a little trouble with iframes (chrome) others have trouble with Google charts (Opera, Silk), so your mileage may vary, but it can certainly serve as an example.
This is what you will get with a line like:
<iframe width="100%" height="325" scrolling="No" seamless frameBorder="0"
src="http://dl.dropboxusercontent.com/u/128855213/CosmGraphPower.html"> </iframe>

And yes, the src attribute up there points to the page I have on Dropbox to show this page. Feel free to grab the source directly from there as well.
As I said before, my web programming experience is slim, so there may well be better, more elegant ways to do this. If so, leave a comment for other folks (and me!) to help make our lives simpler. For example, you have no idea what a pain in the butt it was to get this page to work using the tools that Blogger provides. Their editors kept changing things and messing up the display. I finally got it to work, but I haven't tried it on multiple browsers so, it may be messed up. Look at it in FireFox, it seems to be the best with this display.
Saturday, June 29, 2013
How to use the SteelSeries Gauges with the Xively API
Over the last months I've gotten compliments and questions on how I created the SteelSeries gauges and how I hooked them to the Xively (pachube -> cosm -> Xively) API. First, let me give credit where credit is due. I didn't create the SteelSeries gauges, Gerrit Grunwald did. He has a great blog here <link> where he describes the various displays he's created. I stumbled across them a while back and hooked them into my Xively data feed to display the last updates so I would have a nice display of current power usage and temperature. They worked really well and look great.
It took a little work to get them going, but it was well worth it. Basically, I query Xively for the last value, then give it to the gauge to display. Here's the current (a minute ago) temperature at my house in the Arizona, USA desert.
There's too much up there to try and explain each item; this blog post would go on forever. However, there's lots of documentation on the web that can explain what I did and the SteelSeries library has enough in it to get you started there.
Good luck and have fun with it. I'll be showing how I use the Google Graph API in an upcoming post as well.
It took a little work to get them going, but it was well worth it. Basically, I query Xively for the last value, then give it to the gauge to display. Here's the current (a minute ago) temperature at my house in the Arizona, USA desert.
Edit: This used to be a live gauge, but I replaced it with a picture.
So, the source to do this looks like this:
<head>
<title>Dave Testing</title>
</head>
<body onload=init()>
<canvas id=gaugeCanvas width=200 height=200>No canvas in your browser...sorry...</canvas>
</body>
<script>
function init()
{
// Initialzing gauge
tempGauge = new steelseries.Radial('gaugeCanvas', {
gaugeType: steelseries.GaugeType.TYPE4,
minValue:0,
maxValue:150,
size: 200,
frameDesign: steelseries.FrameDesign.BRASS,
knobStyle: steelseries.KnobStyle.BRASS,
pointerType: steelseries.PointerType.TYPE6,
lcdDecimals: 0,
section: null,
area: null,
titleString: 'Outside Temp',
unitString: '°F',
threshold: 100,
lcdVisible: true
});
// Start the gauge update
setInterval(function(){
var site = "http://api.pachube.com/v2/feeds/9511.json?&key=GtGuoMKJSqv2tzqGvWaITLhlEDUxrYXSixqPSmlyj-s&&datastreams=7";
//alert('going for ' + site);
$.ajax({
type: 'GET',
url: site,
dataType: 'json',
cache: false,
data: {},
processData: true,
async: false, // you have to have this so the data will arrive before display
success: function(data_archive) {
//alert("the url returned success");
//alert("got back " + data_archive.datastreams[0].current_value);
tempGauge.setValueAnimated(eval(data_archive.datastreams[0].current_value));
},
error: function(a){
//alert("here I am in the error routine");
}
});
//alert('after the get');
}, 10000);
}
</script>
<script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.3.2/jquery.min.js"></script>
<script type=text/javascript src=http://dl.dropbox.com/u/128855213/SteelSeries/tween-min.js></script>
<script type=text/javascript src=http://dl.dropbox.com/u/128855213/SteelSeries/steelseries-min.js></script>
<title>Dave Testing</title>
</head>
<body onload=init()>
<canvas id=gaugeCanvas width=200 height=200>No canvas in your browser...sorry...</canvas>
</body>
<script>
function init()
{
// Initialzing gauge
tempGauge = new steelseries.Radial('gaugeCanvas', {
gaugeType: steelseries.GaugeType.TYPE4,
minValue:0,
maxValue:150,
size: 200,
frameDesign: steelseries.FrameDesign.BRASS,
knobStyle: steelseries.KnobStyle.BRASS,
pointerType: steelseries.PointerType.TYPE6,
lcdDecimals: 0,
section: null,
area: null,
titleString: 'Outside Temp',
unitString: '°F',
threshold: 100,
lcdVisible: true
});
// Start the gauge update
setInterval(function(){
var site = "http://api.pachube.com/v2/feeds/9511.json?&key=GtGuoMKJSqv2tzqGvWaITLhlEDUxrYXSixqPSmlyj-s&&datastreams=7";
//alert('going for ' + site);
$.ajax({
type: 'GET',
url: site,
dataType: 'json',
cache: false,
data: {},
processData: true,
async: false, // you have to have this so the data will arrive before display
success: function(data_archive) {
//alert("the url returned success");
//alert("got back " + data_archive.datastreams[0].current_value);
tempGauge.setValueAnimated(eval(data_archive.datastreams[0].current_value));
},
error: function(a){
//alert("here I am in the error routine");
}
});
//alert('after the get');
}, 10000);
}
</script>
<script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.3.2/jquery.min.js"></script>
<script type=text/javascript src=http://dl.dropbox.com/u/128855213/SteelSeries/tween-min.js></script>
<script type=text/javascript src=http://dl.dropbox.com/u/128855213/SteelSeries/steelseries-min.js></script>
There's too much up there to try and explain each item; this blog post would go on forever. However, there's lots of documentation on the web that can explain what I did and the SteelSeries library has enough in it to get you started there.
Good luck and have fun with it. I'll be showing how I use the Google Graph API in an upcoming post as well.
Saturday, December 15, 2012
Really Nice Gauges
Recently I was shown an extremely nice set of gauges by a fellow on the Arduino forum, graynomad. They were metalic looking and had a great set of parameters to mess with. Well, I couldn't resist, and decided to figure out how to use them and somehow get them on this site.
Well, I sort of got it working. It seems that Google (Blogger) has a problem with iframes, and that's pretty much the only way to include one of these. The problem is that it messes with the scrollbar in Chrome, their own browser. I'm not sure how they show up in IE, except they won't show up at all in IE8 because it doesn't support canvas. For something this complex, the canvas is the only practical solution short of using something like Flash.
So, if you're using Chrome, look over to the right where the scrollbar is, notice it doesn't look right? If you put your mouse over there and click, you'll see that it actually works, but just doesn't look right. I don't have a solution to this yet.
But, let's look at the gauge. Notice it has an LCD display and a moving hand. When my energy usage goes above the red indicator (it's on the dial), the little red LED will light up. These are all parameters that can be set for the gauge. I chose the brass look simply because it fit my taste, there are many, many combinations possible. The gauge is sort of real time. What I do is update it every 10 seconds from my datastore at Cosm which is updated every 60 seconds. So, every minute or so, it will be updated to reflect the current power usage at my house. I plan on working with these gauges quite a bit more; they're just too nice to leave alone.
Well, I sort of got it working. It seems that Google (Blogger) has a problem with iframes, and that's pretty much the only way to include one of these. The problem is that it messes with the scrollbar in Chrome, their own browser. I'm not sure how they show up in IE, except they won't show up at all in IE8 because it doesn't support canvas. For something this complex, the canvas is the only practical solution short of using something like Flash.
So, if you're using Chrome, look over to the right where the scrollbar is, notice it doesn't look right? If you put your mouse over there and click, you'll see that it actually works, but just doesn't look right. I don't have a solution to this yet.
But, let's look at the gauge. Notice it has an LCD display and a moving hand. When my energy usage goes above the red indicator (it's on the dial), the little red LED will light up. These are all parameters that can be set for the gauge. I chose the brass look simply because it fit my taste, there are many, many combinations possible. The gauge is sort of real time. What I do is update it every 10 seconds from my datastore at Cosm which is updated every 60 seconds. So, every minute or so, it will be updated to reflect the current power usage at my house. I plan on working with these gauges quite a bit more; they're just too nice to leave alone.
So, if you want to use them also, or just admire the tremendous amount of work that went into creating them, they're known as the SteelSeries and apparently were initially designed for weather displays. Just go to Google and start looking. If I gave a URL, it would be the wrong one or outdated or something, so check it out yourself.
And, sorry about your scrollbar; it'll come back when you go leave this page.
And, sorry about your scrollbar; it'll come back when you go leave this page.
Monday, July 2, 2012
Charting Things
I was stumbling around the site where I store my data Cosm.com (it used to be Pachube.com) and noticed that they have updated their charting capabilities considerably. This made the data on my Detail Usage Charts page mess up, but also added capabilities that I didn't know I had. I edited this page and was able to display a years worth of data (NICE).
There seems to be some kind of bug though because if I ask for a data using months as a parameter, it takes forever to return the data, whereas if I use 30 days, it returns it quickly. The charts on my detail page use 365 days as a parameter and it only takes a few seconds. I can live with that.
They have another chart that will allow you to get the actual reading for any point, but I haven't figured out how to use it yet. Over time, I bet I will.
There seems to be some kind of bug though because if I ask for a data using months as a parameter, it takes forever to return the data, whereas if I use 30 days, it returns it quickly. The charts on my detail page use 365 days as a parameter and it only takes a few seconds. I can live with that.
They have another chart that will allow you to get the actual reading for any point, but I haven't figured out how to use it yet. Over time, I bet I will.
Friday, October 14, 2011
Playing with Graphs part 2
I did it. With a heck of a lot of help from the support staff at Pachube, I can now read the data for whatever period I want and graph it. I don't have an illustration on the web yet because it takes time to gather the data and people get tired of waiting and just click off somewhere else.
However, this is so cool. I can sample data at one minute intervals for, say an hour, any hour, any day, for the last year (or however long I have had the sensor running) and look at it in detail. Ever wonder what happens when the clouds block the sun from your solar heater sensor? What happens to the pump power demand when the sun comes back. I know all about this now.
However, this is so cool. I can sample data at one minute intervals for, say an hour, any hour, any day, for the last year (or however long I have had the sensor running) and look at it in detail. Ever wonder what happens when the clouds block the sun from your solar heater sensor? What happens to the pump power demand when the sun comes back. I know all about this now.
Saturday, October 8, 2011
Playing with Graphs
I'm just testing the Google visualization library. The chart below should show the last few hours of power usage and temperature changes. This is something I want to do with the data I collect around the house and this is my first javascript code to try it. I currently can't get more than 5 hours of data back from Pachube without it paging on me. This will take some thought.
Subscribe to:
Comments (Atom)




