Wednesday, October 23, 2013

GroveStreams and the SteelSeries Gauges

If you've been following my rantings, you've seen the SteelSeries gauges.  I think they are the very best analog display widgets on the web.  At least right now.  Well, GroveStreams enabled them as a embedded object in their 'Dashboard'.  I couldn't wait to try them out to see if they were as nice as the ones I came up with for other things.  Take a look:


Nice aren't they?  I intentionally chose different appearances and styles for them to illustrate a tiny portion of what is possible.  There are other styles, horizonal, LCDs, well, you can look for yourself.  The really cool thing is you can experiment with the look and type right on the screen to get what you want.  Yes, these are (almost) real time.  I update every minute, so if you wait long enough, they'll change.  The little blue and red ticks on them represent the high and low for the last day.  Think about what they would look like with a weather station.

Then, later when you change your mind, go back to GroveStreams dashboard and change them.  Since the way they embed them is to use the dashboard object, all the places you've used them change as well.

Slick, really slick.

Wednesday, October 16, 2013

Floats and Strings Over a Serial Link (like an XBee)

I've mentioned many times how I use a regular old ascii string as the payload when communicating between XBees whenever possible.  There's several reasons for this, but the biggest is debugging the interaction of the devices.  It's a royal pain in the behind to try and interpret the data traveling over the air as binary using XCTU or a monitor XBee.  I convert integers, floats and such to text, send it, and reconstruct what I need at the other end. This way I can actually read what is traveling between XBees using a monitor XBee or the little sniffer I constructed exactly for this purpose.

However, this has generated questions about how to do such a thing from folks that are just starting out using an Arduino hooked to an XBee because it isn't obvious.  Yes, there are a bazillion examples out there, but not specifically related to passing data between XBees.  Additionally, there are operators in other languages that take care of this kind of thing totally behind the scenes and programmers don't have to worry about it.

Adding insult to injury, the Stream data type in the Arduino IDE has problems when running for a very long time: it will run your board out of memory and cause it to fail.  This isn't a complaint about the Arduino development environment, just a simple fact.  The little Arduino only has 2K of memory to play with and you simply run out if you try to do too much.  It's not like a laptop with 8Gb of memory to play around with, you have to control yourself and your code. So, doing things with as little memory usage as possible using simple tools is the way to make a device that can run for a week without failing.

So, here's a sketch that illustrates the some of the data types including the float.  The integer and long datatypes are relatively easy, but the float confuses some people. The float has generated probably a hundred questions and even more misunderstandings.  This code is waaay over commented and  will compile and run on an Arduino.  It basically takes a float, long and integer, converts them into ascii in a string with other stuff, and then gets it back out into variables to be used further.  The middle part where the string is sent between two devices can be found in other places on this blog. This will compile and run on an Arduino under IDE version 1.0 and up.

The Arduino Sketch
#include <stdio.h>

int intVariable;
long longVariable;
float floatVariable;

// this little function will return the first two digits after the decimal
// point of a float as an int to help with sprintf() (won't work for negative values)
// the .005 is there for rounding.
int frac(float num){
  return( ((num + .005) - (int)num) * 100);
}
// this function prints the characters of a c string one at a time
// without any formatting to confuse or hide things
void printbuffer(char *buffer){
  while(*buffer){
    Serial.write(*buffer++);
  }
}

void setup(){
  Serial.begin(9600);
}

char buff[100]; // we're going to use this to hold our string

void loop(){
  Serial.println("starting ...");
  
  intVariable = 11; // integers are just that, integers, no decimal point possible
  Serial.println(intVariable);
  
  longVariable = 12.45; // longs are really just big integers, they can't have a decimal
  Serial.println(longVariable); // This will show you what happens with a long
  
  floatVariable = 13.45; // floats are a different animal
  Serial.println(floatVariable);
  
  // now I'm putting these in a string.  For this I'm using sprintf() because
  // it makes this kind of thing so much easier.  I use the little frac() routine
  // up above.  This is simply cutting the float into two integers, one being the 
  // part to the left of the decimal and the other being two digits to the right
  // of the decimal.  The long uses a special format specification %ld as in 
  // 'long decimal', and int is just stuffed in there directly
  sprintf(buff, "This whole thing is a string: %d.%2d, %ld, %d\n", 
          int(floatVariable), frac(floatVariable), // the two halves of the float
          longVariable,  // and the long
          intVariable);  // and finally the integer
  
  // the %d means construct it as a decimal number (as in base 10), the '.' is 
  // just a period to be placed in the string, %2d means a 2 digit number with leading 
  // zeros.  Google the printf specification for several million examples.
  
  // Now let's print it one character at a time to illustrate what's in the string
  // without the formatting capabilities of Serial.print() confusing things
  printbuffer(buff);
  
  // Now, buff has a string of characters in it with the ascii 
  // representation of the variables in it.  You can send this
  // string through an XBee, over a wire using one of the serial 
  // techniques, or store it in a file.  It's also pretty good for 
  // serial LCD displays.
  
  // So, let's get the number out of the string.  To do this, you have to know
  // how the string is constructed.  If you choose a method to construct the string
  // that is easy to take apart (like a comma separated string) things are much
  // easier.  However, this string is actually pretty nasty.  So, we'll first find the 
  // colon.
  char *tmp = strchr(buff, ':');
  printbuffer(tmp);
  
  // So, now that we have a pointer into the string that begins at the colon, let's
  // skip the ': ' (colon space) and we'll be right at the number
  tmp += 2;
  printbuffer(tmp);
  
  // OK, now let's get the darn ascii number out of the string and back into a float.
  float recoveredFloat = atof(tmp);
  Serial.print("Float recovered was: ");
  Serial.println(recoveredFloat);
  
  // Now you have your float variable back as a float to do with as you please.
  // So, move over to the ',' that is before the long
  tmp = strchr(tmp, ',');
  printbuffer(tmp);
  // and skip the ', " to get to the number
  tmp += 2;
  printbuffer(tmp);
  // and get it out into a variable
  long recoveredLong = atol(tmp);
  Serial.print("Long recovered was: ");
  Serial.println(recoveredLong);
  // and the whole thing over again for the integer
  tmp = strchr(tmp, ',');
  printbuffer(tmp);
  tmp += 2;
  printbuffer(tmp);
  int recoveredInt = atoi(tmp);
  Serial.print("Int recovered was: ");
  Serial.println(recoveredInt);

  Serial.println("done.");
  while(1){}
}

Yes, I eat up substantial code memory using sprintf(), but it's worth it.  You can reuse the buffer over and over and you only pay the price for sprintf() once, not over and over again like you do with the String datatype.  Notice I didn't get into a long discussion of how floats are stored in memory and how operations on them work.  That's documented in about a million places and I don't want to add confusion by getting into that discussion.  If you need to know, go look it up.

There are three c library routines that are used here: atoi(), atol(), and atof().  These are documented on the web, so take a look at what they do.  One of the keys to understanding this stuff is to do a LOT of looking around for various solutions.  Anything you want to do has been done in some part before and someone probably wrote about it somewhere.

Keep in mind that there are as many ways to do this as there are programmers doing it.  So this is just one way.  I chose this to illustrate it as completely and simply as I could so folks would leave with a few less questions than they came with.

Now, when these questions come up, I can simply point to this page and let folks play with the code until they get the idea.  When people first start out using an Arduino to try and control things, it's a tough enough hurdle just getting the first code to work; I hope this helps some of them.

Thursday, October 10, 2013

GroveStreams Another Data Service

In my continuing search for sites to put my house data on I encountered Grovestreams.com on the Arduino forum.  Well, since I have a cool setup to try these things out I decided to give it a try.  At first I got totally lost in the jargon inherent to cloud services.  These things talk about streams of data, component architecture, registering a service; you know, the kind of language that simply puts people that make things to sleep.  I persevered.

Once I started to understand what they meant by messing up example after example, I was ready to try some code.  I grabbed their python example and updated it to save one of my devices.  Since I'm a power usage freak, I chose my real time measurement of power usage.  It worked.  That was actually really cool, so I added outside temperature.  It worked too.  Then I went a little nuts and added the rest of my house data to it.

Then I was on a roll.  I modified the code again to use the techniques I used on the other cloud services.  I put it under the APSscheduler to do one minute updates, added it to the init table so it would restart if it had trouble and let it go.  It's running right now keeping track of the data I collect around the house for me.  So, what about a chart to look at?  Well, they have a dashboard like some of the other services and I created one to hold some charts; these charts are embeddable:

Pretty slick.  We've seen this before, but it's nice to see a new service like this implement it first instead of years later. They also have other 'widgets', gauges and things. What impressed me about this site is the versatility.  They have capabilities that I won't have time to try out. They can derive data from other data and present it.  It should be possible to build a chart based on my power usage and the billing charges multiplied together and actually show graphically how much money it's costing me to keep cool in the summer.  That's actually a little depressing when you think about it.

They even have alerts that can send SMS and email to you.  I created an alert to tell me when my house power usage goes over 10kWh.  It was annoying when it came in the first time and even more annoying when it came in the second time.  I actually have real time notification that I'm wasting money that comes in on my phone.  I could easily expand this to alert me when the garage door is open after sundown.  That would help a lot keeping the pack rats from stealing my tools and burying them in a mound outside.

Here's the code I'm actually running to update my house data on their site:

The python Sketch Script
#!/usr/bin/python
import time
from time import sleep
from datetime import datetime
import sys
from apscheduler.scheduler import Scheduler
import logging
from simplejson import encoder as jsonEncoder
import httplib
import StringIO
import gzip
import sqlite3
import pprint

# I keep all current values in a data base
DATABASE='/home/pi/database/desert-home'

# These are the secret numbers I use to get into
# the grovestreams service
org = "putyourownsecretthinghere";
api_key = "samewithyoursecretapikey";

#If you want lots of messages and debug information
# set this to true
DEBUG = False

# the site accepts compression, might as well use it
def compressBuf(buf):
zbuf = StringIO.StringIO()
zfile = gzip.GzipFile(mode = 'wb', fileobj = zbuf, compresslevel = 9)
zfile.write(buf)
zfile.close()
return zbuf.getvalue()

def updateGrovestreams():
# This is VERY different from their examples.  I named
# my streams with something I could understand and read
# Probably not the best way to do it, but it works really
# well for me.
component_id = "desert-home-id"
rpowerStream_id = "power_usage"
otempStream_id = "outside_temp"
apowerStream_id = "apparent_power"
voltageStream_id = "voltage"
currentStream_id = "current"
pfactorStream_id = "power_factor"
itempStream_id = "inside_temp"
ptempStream_id = "pool_temp"
pmotorStream_id = "pool_motor"

# This object really helps when displaying
# the rather complex data below
pp = pprint.PrettyPrinter(depth=10)

#get the millis since epoch
# in unix the epoch began back in 1970, look it up
now = datetime.now()
nowEpoch = int(time.mktime(now.timetuple())) * 1000

#assemble feed and convert it to a JSON string
feed = {};
feed['feed'] = {}
feed['feed']['component'] = []
if DEBUG:
pp.pprint(feed)
print
sys.stdout.flush()

comp = {}
comp['stream'] = []
comp['componentId'] = component_id
feed['feed']['component'].append(comp)
if DEBUG:
pp.pprint(feed)
print
sys.stdout.flush()

# Now I'm going to fill in the stream values, open database
# I took a brute force approach to building the dictionary that
# is converted into JSON.  I could have been much more elegant
# in building it, but the folks just starting out would have
# had a tough time understanding it
dbconn = sqlite3.connect(DATABASE)
c = dbconn.cursor()
# So, you make a stream to stuff things into.  It's actually
# a python dictionary that we'll pass to a JSON encoder a ways
# down into the code.  I'll be adding entries to this as I pull
# items out of the database
stream1 = {}  
stream1['streamId'] = rpowerStream_id
stream1['time'] = [] 
stream1['data'] = []
comp['stream'].append(stream1)

current_value = c.execute("select rpower from power").fetchone()[0]  
stream1['time'].append(nowEpoch)
stream1['data'].append(float(current_value))
# this is a cool way to debug this kind of thing.
if DEBUG:
pp.pprint(feed)
print
sys.stdout.flush()
# notice how I get and item out of the database
# and add it to the dictionary.  I'll do this
# several times
stream2 = {}
stream2['streamId'] = otempStream_id
stream2['time'] = []
stream2['data'] = []
comp['stream'].append(stream2)
current_value = c.execute(
"select currenttemp from xbeetemp").fetchone()[0]
stream2['time'].append(nowEpoch)
stream2['data'].append(float(current_value))

stream3 = {}
stream3['streamId'] = apowerStream_id
stream3['time'] = []
stream3['data'] = []
comp['stream'].append(stream3)
current_value = c.execute(
"select apower from power").fetchone()[0]
stream3['time'].append(nowEpoch)
stream3['data'].append(float(current_value))

stream4 = {}
stream4['streamId'] = voltageStream_id
stream4['time'] = []
stream4['data'] = []
comp['stream'].append(stream4)
current_value = c.execute(
"select voltage from power").fetchone()[0]
stream4['time'].append(nowEpoch)
stream4['data'].append(float(current_value))

stream5 = {}
stream5['streamId'] = currentStream_id
stream5['time'] = []
stream5['data'] = []
comp['stream'].append(stream5)
current_value = c.execute(
"select current from power").fetchone()[0]
stream5['time'].append(nowEpoch)
stream5['data'].append(float(current_value))

stream6 = {}
stream6['streamId'] = pfactorStream_id
stream6['time'] = []
stream6['data'] = []
comp['stream'].append(stream6)
current_value = c.execute(
"select pfactor from power").fetchone()[0]
stream6['time'].append(nowEpoch)
stream6['data'].append(float(current_value))

stream7 = {}
stream7['streamId'] = itempStream_id
stream7['time'] = []
stream7['data'] = []
comp['stream'].append(stream7)
current_value = c.execute(
"select avg(\"temp-reading\") from thermostats").fetchone()[0]
stream7['time'].append(nowEpoch)
stream7['data'].append(float(current_value))

stream8 = {}
stream8['streamId'] = ptempStream_id
stream8['time'] = []
stream8['data'] = []
comp['stream'].append(stream8)
current_value = c.execute(
"select ptemp from pool").fetchone()[0]
stream8['time'].append(nowEpoch)
stream8['data'].append(float(current_value))

stream9 = {}
stream9['streamId'] = pmotorStream_id
stream9['time'] = []
stream9['data'] = []
comp['stream'].append(stream9)
tmp = c.execute("select motor from pool").fetchone()[0];
if (tmp == 'High'): # a little special handling for the pool motor
motor = 2
elif (tmp == 'Low'):
motor = 1
else:
motor = 0
stream9['time'].append(nowEpoch)
stream9['data'].append(int(motor))

# all the values are filled in, close the database
dbconn.close() # close the data base
# This will print the entire database I just constructed
# so you can see what is going on
if DEBUG:
pp.pprint(feed)
print
sys.stdout.flush()
# exit() # I put this in for debugging.  It exits before
# the JSON string is constructed and sent off to grovestreams
# Of course you want to keep it commented until needed
#
# And this is where the JSON string is built
encoder = jsonEncoder.JSONEncoder()
json = encoder.encode(feed);
# and this will print it so you can see what is happening
if DEBUG:
print json # for debugging
print
sys.stdout.flush()

#Upload the feed
try:
print "Updating GroveStream ", time.strftime("%A, %B %d at %H:%M:%S")
sys.stdout.flush()
conn = httplib.HTTPConnection('www.grovestreams.com')

url = '/api/feed?&org=%s&api_key=%s' % (org, api_key)

compress = True
if compress:
body = compressBuf(json)
headers = {"Content-type": "application/json", "Content-Encoding" : "gzip"}
else:
body = json
headers = {"Content-type": "application/json", "charset":"UTF-8"}

conn.request("PUT", url, body, headers)

response = conn.getresponse()
status = response.status

if status != 200 and status != 201:
try:
if (response.reason != None):
print('reason: ' + response.reason + ' body: ' + response.read())
sys.stdout.flush()
else:
print('body: ' + response.read())
sys.stdout.flush()
except Exception:
print('HTTP Fail Status: %d' % (status) )
sys.stdout.flush()

except Exception as e:
print('HTTP Send Error: ' + str(e))
sys.stdout.flush()
   
finally:
if conn != None:
conn.close()

# I just discovered the statement below.
# someday I'll have go figure out what it really does.
if __name__ == '__main__':
print "started at ", time.strftime("%A, %B, %d at %H:%M:%S")
sys.stdout.flush()
logging.basicConfig()

#------------------Stuff I schedule to happen -----
scheditem = Scheduler()
scheditem.start()
# every minute update the data store on Xively
scheditem.add_interval_job(updateGrovestreams, seconds=60)
#
# A couple of people asked me why I put this statement in
# since I have it scheduled to happen every 60 seconds already
# Well, when you're debugging something it sucks to have to
# wait 60 seconds to see if you fixed it, so I do it 
# first, then let the scheduler take care of the rest.
#
updateGrovestreams()
while True:
time.sleep(20) #This doesn't matter much since it is schedule driven
 

Yes, in my usual form, it's got way too many comments.  It's also very inelegant; I'm not a big fan of strange statements that run really well but take an hour to understand so I just brute forced the URL and body creation of the request sent.  Notice that there isn't a library with obscure calls and methods and stuff.  This is all python code that I simply stole from them and added my particular items to.  Makes it nice that I don't have to learn how to use some special library.  I also put in a variable to control debugging so you can see what is happening.  It gets a little complex at a couple of points.

I didn't stop there.  I took their arduino example and modified it to update some items as well.  It worked first try.  The example had a little trouble since Grovestreams used Strings as a way of making the code more easily read and it started losing memory.  The poor little arduino has so little memory that I knew it would die over time, so I sent them a note describing the problem.  Guess what?  They responded!

Yes, in this 21st century world of minimal text messages, they actually answered me and came up with a new example that works better.  Unfortunately, I didn't save that one as an illustration for you, you'll just have to go to their site and get it yourself.  However, I did save the first version I tried:

The Arduino Sketch
/*

 Arduino GroveStreams Stream Feed via Ethernet

 The GroveStreams client sketch is designed for the Arduino and Ethernet.
 A full "how to" guide for this sketh can be found at https://www.grovestreams.com/developers/getting_started_arduino_temp.html
 This sketch updates several stream feeds with an analog input reading,
 from a temperature probe, via the GroveStreams API: https://www.grovestreams.com/developers/apibatchfeed.html
 The Arduino uses DHCP and DNS for a simpler network setup.
 The sketch also includes a Watchdog / Reset function to make sure the
 Arduino stays connected and/or regains connectivity after a network outage.
 Use the Serial Monitor on the Arduino IDE to see verbose network feedback
 and GroveStreams connectivity status.

 GroveStreams Setup:

 * Sign Up for Free User Account - https://www.grovestreams.com
 * Create a GroveStreams organization and select the Arduino blueprint
 * Enter a unique MAC Address for this network in this sketch under "Local Network Settings" 
 *    (Newer shields have the mac address on a sticker on the shield. Use that.)
 *    (A MAC address can also be generated within a GroveStreams organization: tools - Generate MAC Address)
 * Enter the GroveStreams org uid under "GroveStreams Settings" in this sketch 
 *    (Can be retrieved from a GroveStreams organization: tools - View Organization UID)
 * Enter the GroveStreams api key under "GroveStreams Settings" in this sketch  
 *    (Can be retrieved from a GroveStreams organization: click the Api Keys toolbar button, 
 *     select your Api Key, and click View Secret Key)

 Arduino Requirements:

 * Arduino with Ethernet Shield or Arduino Ethernet
 * Arduino 1.0 IDE

 Network Requirements:

 * Ethernet port on Router    
 * DHCP enabled on Router
 * Unique MAC Address for Arduino

 Additional Credits:
 Example sketches from Arduino team, Ethernet by David A. Mellis

 */

#include <SPI.h>
#include <Ethernet.h>
#include <MemoryFree.h>


// Local Network Settings
byte mac[] = {0x90, 0xA2, 0xDA, 0x00, 0x33, 0x33};        // Change this!!! Must be unique on local network. 
                                               // Look for a sticker on the back of your Ethernet shield.

// GroveStreams Settings
String gsApiKey = "You\'re going to need your own number here";   //Change This!!!
String gsOrg = "another number you need to put in";      //Change This!!!
String gsComponentName = "Temperature";                     //Optionally change. Set this to give your component a name when it initially registers.

char gsDomain[] = "grovestreams.com";   //Don't change. The Grove Streams domain. 
String gsComponentTemplateId = "temp";  //Don't change. Tells GS what template to use when the feed initially arrives and a new component needs to be created.
                                        // The blueprint is expecting "temp".

//GroveStreams Stream IDs. Stream IDs tell GroveStreams which component streams the values will be assigned to.
//Don't change these unless you edit your GroveStreams component definition and change the stream ID to match this.
String gsStreamId1 = "s1";   //Temp C - Random Stream. 
String gsStreamId2 = "s2";   //Temp F - Random Stream. Don't change.
String gsStreamId3 = "s3";   //Temp C - Interval Stream (20 second intervals). Don't change.
String gsStreamId4 = "s4";   //Temp F - Interval Stream (20 second Intervals). Don't change.

const int updateFrequency = 20 * 1000; // GroveStreams update frequency in milliseconds (the GS blueprint is expecting 20s)
const int temperaturePin = 0;          // Then Temperature pin number.    

// Variable Setup
String myIPAddress;  //Set below from DHCP. Needed by GroveStreams to verify that a device is not uploading more than once every 10s.
String myMac;        //Set below from the above mac variable. The readable Mac is used by GS to determine which component the feeds are uploading into.

long lastConnectionTime = 0;    //Don't change. Used to determine if the Ethernet needs to be reconnected. 
boolean lastConnected = false;  //Don't change. 
int failedCounter = 0;          //Don't change. 

// Initialize Arduino Ethernet Client
EthernetClient client;


void setup()
{
  // Start Serial for debugging on the Serial Monitor
  Serial.begin(9600);

  // Start Ethernet on Arduino
  startEthernet();
  randomSeed(analogRead(1));
}

void loop()
{

  // Print Update Response to Serial Monitor
  if (client.available())
  {
    char c = client.read();
    Serial.print(c);
  }

  // Disconnect from GroveStreams
  if (!client.connected() && lastConnected)
  {
    Serial.println("...disconnected");
    Serial.println();
    showMem();

    client.stop();
  }

  // Update sensor data to GroveStreams
  if(!client.connected() && (millis() - lastConnectionTime > updateFrequency))
  {
    String tempC = getTemperatureC();
    Serial.print (tempC);
    String tempF = getTemperatureF();
    Serial.print (tempF);
    updateGroveStreams(tempC, tempF);
  }

  // Check if Arduino Ethernet needs to be restarted
  if (failedCounter > 3 ) {
    
    //Too many failures. Restart Ethernet.
    startEthernet();
  }

  lastConnected = client.connected();
}

void updateGroveStreams(String tempC, String tempF)
{
  Serial.println("\nin update routine");
  if (client.connect(gsDomain, 80))
  {         

    //Assemble the url used to pass the temperature readings to GroveStreams
    // The Arduino String class contains many memory bugs and char arrays should be used instead, but
    // to make this example simple to understand we have chosen to use the String class. 
    // No none memory issues have been seen with this example to date.
    //We are passing temperature readings into two types of GroveStreams streams, Random and Interval streams.
    String url = "PUT /api/feed?&compTmplId=" + gsComponentTemplateId + "&compId=" + myMac + "&compName=" + gsComponentName;
    url += "&org=" + gsOrg + "&api_key=" + gsApiKey;

    url += "&" + gsStreamId1 + "=" + tempC;  //Temp C - Random Stream
    url += "&" + gsStreamId2 + "=" + tempF;  //Temp F - Random Stream
    url += "&" + gsStreamId3 + "=" + tempC;  //Temp C - Interval Stream (20 second intervals)
    url += "&" + gsStreamId4 + "=" + tempF;  //Temp F - Interval Stream (20 second intervals)

    url += " HTTP/1.1";
    Serial.print(url);
    client.println(url);  //Send the url with temp readings in one println(..) to decrease the chance of dropped packets
    client.println("Host: " + String(gsDomain));
    client.println("Connection: close");
    client.println("X-Forwarded-For: "+ myIPAddress); //Include this line if you have more than one device uploading behind 
                                                      // your outward facing router (avoids the GS 10 second upload rule)
    client.println("Content-Type: application/json");
    client.println();

    if (client.available())
    {
      //Read the response and display in the the console
      char c = client.read();
      Serial.print(c);
    }

    lastConnectionTime = millis();

    if (client.connected())
    {     
      failedCounter = 0;
    }
    else
    {
      //Connection failed. Increase failed counter
      failedCounter++;

      Serial.println("Connection to GroveStreams failed ("+String(failedCounter, DEC)+")");   
      Serial.println();
    }

  }
  else
  {
     //Connection failed. Increase failed counter
    failedCounter++;

    Serial.println("Connection to GroveStreams Failed ("+String(failedCounter, DEC)+")");   
    Serial.println();

    lastConnectionTime = millis(); 
  }
}

void startEthernet()
{
  //Start or restart the Ethernet connection.
  client.stop();

  Serial.println("Connecting Arduino to network...");
  Serial.println();  

  //Wait for the connection to finish stopping
  delay(1000);

  //Connect to the network and obtain an IP address using DHCP
  if (Ethernet.begin(mac) == 0)
  {
    Serial.println("DHCP Failed, reset Arduino to try again");
    Serial.println();
  }
  else
  {

    Serial.println("Arduino connected to network using DHCP");
    Serial.println();

    //Wait to ensure the connection finished
    delay(1000);

    //Set the mac and ip variables so that they can be used during sensor uploads later
    myMac =getMacReadable();
    Serial.println("MAC: " + myMac);

    myIPAddress = getIpReadable(Ethernet.localIP());
    Serial.println("IP address: " + myIPAddress);
  }

}

String getMacReadable() 
{
  //Convert the mac address to a readable string
  char macstr[20];
  snprintf(macstr, 100, "%02x:%02x:%02x:%02x:%02x:%02x", mac[0], mac[1], mac[2], mac[3], mac[4], mac[5]);
  return String(macstr);
}

String getIpReadable(IPAddress p)
{
  //Convert the ip address to a readable string
  String ip;
  for (int i =0; i < 3; i++)
  {
    ip += String(p[i], DEC);
    ip += ".";
  }
  ip +=String(p[3], DEC);
  return ip;
}

String getTemperatureF() 
{
  //Get the temperature analog reading and convert it to a string
  float voltage, degreesC, degreesF; 

  //voltage = (analogRead(temperaturePin) * 0.004882814);
  //degreesF = degreesC * (9.0/5.0) + 32.0;
  degreesF = float(random(70, 120));

  char temp[20] = {0}; //Initialize buffer to nulls
  dtostrf(degreesF, 12, 3, temp); //Convert float to string

  String stemp = temp;
  stemp.trim();  //Trim off head and tail spaces
  return stemp; 
}

String getTemperatureC() 
{
  //Get the temperature analog reading and convert it to a string
  float voltage, degreesC, degreesF; 

  //voltage = (analogRead(temperaturePin) * 0.004882814);
  //degreesC = (voltage - 0.5) * 100.0;
  degreesC = float(random(20, 40));

  char temp[20] = {0}; //Initialize buffer to nulls
  dtostrf(degreesC, 12, 3, temp); //Convert float to string

  String stemp = temp;
  stemp.trim();  //Trim off head and tail spaces
  return stemp; 
}

void showMem(){
  char Dbuf [100];
  
  strcpy_P(Dbuf,PSTR("Mem = "));
  Serial.print(Dbuf);
  Serial.println(freeMemory());
}



This worked really well, but like I said above, it will die over time because the Streams library will run the arduino out of memory.  I left the code in it that I used to check memory usage and also the random number generation I used to test it.

Notice that the call to send the data is different?  Of course you did.  The call is part of their batch update API that is much simpler to use than the JSON call I used in the python example.  This makes getting the data up there much easier for the arduino folks.  They're even working on improvements to make the API more easily understood.  The python folk seem to like infinitely complex data structures, so I'm leaving that example as it is.

They even have a full blown tutorial on hooking the arduino up to their service.  Unlike a lot of the tutorials out there, this one can actually be read.  You don't even have to click on page after page like the darn instructables we've all learned to love.

I really like this site.  Like I said, I haven't gotten past the very surface of its capabilities, and probably never will, but it was nice to see a site where the folks running it actually care if people can use it.

Sunday, October 6, 2013

Xively, I've just about given up on it.

I was reviewing a couple of my pages after changing my House Controller to a Raspberry Pi, and noticed that one of my charts was several hours behind the current readings. Then I noticed that the chart that I present on my House Controller was presenting yesterdays readings. After about three hours of chasing problems, it turned out that the Xively server is caching pages.

Yes, I'm trying to get the latest data I uploaded and the server is returning me the data from hours or even days ago because it sees the request hasn't changed and simply returns what it has in its cache. No, I wasn't changing the request I sent; it was a request for the latest data. Why would I change a request that asked for the latest?

 What I had to do to overcome the problem was to get the date and time and ask for data that ended with the current time instead. This made the request change each time I sent it and the server had to read the data to fulfill the request. This explains a lot of the problems mentioned on StackOverflow that I read while trying to get my own stuff to work. Additionally, I wanted to step through Xively's process of 'Deploy' and couldn't.  Seems the device itself has to have code on it to activate itself. There's no support for this in their library and I really didn't want to spend hours going through the debugging process that I would incur to try and meet their API. They could have simply put in a single unit deployment process on their web site so that little guys like me could step through it.

Their rates are huge, a grand being the lowest level with transaction costs on top of that; a far cry from the 10 bucks or so a month that pachube had back in the good old days before some faceless corporation bought them out. So, let's list a few things that I've noticed. They preserved the legacy feeds that we've been using, but provide no reasonable way of moving them to the new system. They cache data so we're very likely to get data back that is days old. They have no way to deploy a device in their new system without the device itself doing the work. They took away their forum for exchanging ideas, instead they expect us to post on StackOverflow which is specifically designed for problems. They even removed the forum so that the old ideas and techniques have disappeared. Yet, they post on Twitter that they are attending shows, giving speeches, and publishing articles on the prevalence of the Internet of Things.

Does anyone actually like this service since the latest change? Am I the only one that has seen a steady decline since the original developers presented this cool new tool that us home automation and control freaks went nuts over?

Sad.