Wednesday, October 16, 2013

Floats and Strings Over a Serial Link (like an XBee)

I've mentioned many times how I use a regular old ascii string as the payload when communicating between XBees whenever possible.  There's several reasons for this, but the biggest is debugging the interaction of the devices.  It's a royal pain in the behind to try and interpret the data traveling over the air as binary using XCTU or a monitor XBee.  I convert integers, floats and such to text, send it, and reconstruct what I need at the other end. This way I can actually read what is traveling between XBees using a monitor XBee or the little sniffer I constructed exactly for this purpose.

However, this has generated questions about how to do such a thing from folks that are just starting out using an Arduino hooked to an XBee because it isn't obvious.  Yes, there are a bazillion examples out there, but not specifically related to passing data between XBees.  Additionally, there are operators in other languages that take care of this kind of thing totally behind the scenes and programmers don't have to worry about it.

Adding insult to injury, the Stream data type in the Arduino IDE has problems when running for a very long time: it will run your board out of memory and cause it to fail.  This isn't a complaint about the Arduino development environment, just a simple fact.  The little Arduino only has 2K of memory to play with and you simply run out if you try to do too much.  It's not like a laptop with 8Gb of memory to play around with, you have to control yourself and your code. So, doing things with as little memory usage as possible using simple tools is the way to make a device that can run for a week without failing.

So, here's a sketch that illustrates the some of the data types including the float.  The integer and long datatypes are relatively easy, but the float confuses some people. The float has generated probably a hundred questions and even more misunderstandings.  This code is waaay over commented and  will compile and run on an Arduino.  It basically takes a float, long and integer, converts them into ascii in a string with other stuff, and then gets it back out into variables to be used further.  The middle part where the string is sent between two devices can be found in other places on this blog. This will compile and run on an Arduino under IDE version 1.0 and up.

The Arduino Sketch
#include <stdio.h>

int intVariable;
long longVariable;
float floatVariable;

// this little function will return the first two digits after the decimal
// point of a float as an int to help with sprintf() (won't work for negative values)
// the .005 is there for rounding.
int frac(float num){
  return( ((num + .005) - (int)num) * 100);
}
// this function prints the characters of a c string one at a time
// without any formatting to confuse or hide things
void printbuffer(char *buffer){
  while(*buffer){
    Serial.write(*buffer++);
  }
}

void setup(){
  Serial.begin(9600);
}

char buff[100]; // we're going to use this to hold our string

void loop(){
  Serial.println("starting ...");
  
  intVariable = 11; // integers are just that, integers, no decimal point possible
  Serial.println(intVariable);
  
  longVariable = 12.45; // longs are really just big integers, they can't have a decimal
  Serial.println(longVariable); // This will show you what happens with a long
  
  floatVariable = 13.45; // floats are a different animal
  Serial.println(floatVariable);
  
  // now I'm putting these in a string.  For this I'm using sprintf() because
  // it makes this kind of thing so much easier.  I use the little frac() routine
  // up above.  This is simply cutting the float into two integers, one being the 
  // part to the left of the decimal and the other being two digits to the right
  // of the decimal.  The long uses a special format specification %ld as in 
  // 'long decimal', and int is just stuffed in there directly
  sprintf(buff, "This whole thing is a string: %d.%2d, %ld, %d\n", 
          int(floatVariable), frac(floatVariable), // the two halves of the float
          longVariable,  // and the long
          intVariable);  // and finally the integer
  
  // the %d means construct it as a decimal number (as in base 10), the '.' is 
  // just a period to be placed in the string, %2d means a 2 digit number with leading 
  // zeros.  Google the printf specification for several million examples.
  
  // Now let's print it one character at a time to illustrate what's in the string
  // without the formatting capabilities of Serial.print() confusing things
  printbuffer(buff);
  
  // Now, buff has a string of characters in it with the ascii 
  // representation of the variables in it.  You can send this
  // string through an XBee, over a wire using one of the serial 
  // techniques, or store it in a file.  It's also pretty good for 
  // serial LCD displays.
  
  // So, let's get the number out of the string.  To do this, you have to know
  // how the string is constructed.  If you choose a method to construct the string
  // that is easy to take apart (like a comma separated string) things are much
  // easier.  However, this string is actually pretty nasty.  So, we'll first find the 
  // colon.
  char *tmp = strchr(buff, ':');
  printbuffer(tmp);
  
  // So, now that we have a pointer into the string that begins at the colon, let's
  // skip the ': ' (colon space) and we'll be right at the number
  tmp += 2;
  printbuffer(tmp);
  
  // OK, now let's get the darn ascii number out of the string and back into a float.
  float recoveredFloat = atof(tmp);
  Serial.print("Float recovered was: ");
  Serial.println(recoveredFloat);
  
  // Now you have your float variable back as a float to do with as you please.
  // So, move over to the ',' that is before the long
  tmp = strchr(tmp, ',');
  printbuffer(tmp);
  // and skip the ', " to get to the number
  tmp += 2;
  printbuffer(tmp);
  // and get it out into a variable
  long recoveredLong = atol(tmp);
  Serial.print("Long recovered was: ");
  Serial.println(recoveredLong);
  // and the whole thing over again for the integer
  tmp = strchr(tmp, ',');
  printbuffer(tmp);
  tmp += 2;
  printbuffer(tmp);
  int recoveredInt = atoi(tmp);
  Serial.print("Int recovered was: ");
  Serial.println(recoveredInt);

  Serial.println("done.");
  while(1){}
}

Yes, I eat up substantial code memory using sprintf(), but it's worth it.  You can reuse the buffer over and over and you only pay the price for sprintf() once, not over and over again like you do with the String datatype.  Notice I didn't get into a long discussion of how floats are stored in memory and how operations on them work.  That's documented in about a million places and I don't want to add confusion by getting into that discussion.  If you need to know, go look it up.

There are three c library routines that are used here: atoi(), atol(), and atof().  These are documented on the web, so take a look at what they do.  One of the keys to understanding this stuff is to do a LOT of looking around for various solutions.  Anything you want to do has been done in some part before and someone probably wrote about it somewhere.

Keep in mind that there are as many ways to do this as there are programmers doing it.  So this is just one way.  I chose this to illustrate it as completely and simply as I could so folks would leave with a few less questions than they came with.

Now, when these questions come up, I can simply point to this page and let folks play with the code until they get the idea.  When people first start out using an Arduino to try and control things, it's a tough enough hurdle just getting the first code to work; I hope this helps some of them.

12 comments:

  1. Thanks for saving me from needing to reguarly use a soft reset in my Arduino <-> Openhab project. I have to implement this and then the arduino will hopefully run for more than ~10h without reset.

    ReplyDelete
  2. I'm glad this helped. I've been using this technique for quite a while to crowd as much as I can into an Arduino. You can also use a little routine I have on the blog in a couple of places to monitor memory usage to be sure you have it under control.

    ReplyDelete
  3. Dave:

    This link no longer works:
    outdoor temperature sensor

    Cheers.
    Glenn.

    ReplyDelete
    Replies
    1. Wher'e the link at? I didn't find it on this page. Am I missing something obvous?

      Delete
  4. Hi Dave. I am sending float number from one Xbee to another using sprintf() method as you did above.It works perfect when I am trying to send small numbers. However, it is not sending big numbers like 86536,46 properly. I guess it is because of the buffer size. But when I give big number to buffer like char buffer[100000] it says overflow in array dimension. So can you suggest me what to do in this situation?

    ReplyDelete
    Replies
    1. Big numbers shouldn't be a problem unless you're sending a lot of them, or a number really, really huge. The reason you're having trouble with the big buffer is because there just isn't that much memory on an arduino.

      We're talking a couple of K of memory to work inside and a hundred thousand byte buffer would be a problem.

      What the heck are you trying to do?

      Delete
    2. I think I have solved sending big numbers but new problem appeared unfortunately. When I am trying to send float type number I can only send the one which has one, two, three and four digits after decimal. Like 86745.321 or 94567.3478. But when I am trying to send, lets say 86345.56734 it puts different number to the buffer and sends it instead. So do you know what is the problem?

      Delete
    3. Looks like you exceeded the capability of the floating point library for the Arduino. I'm not sure of that, but it fits the symptoms. The numbers you're messing with look like you're reading a GPS chip. If you are, those numbers originate from the chip as part of a string. They're already in ascii, why not just use that?

      If you're reading the GPS strings and grabbing the numbers, then converting them to floating point, then converting them to strings to send, just cut out the middle step and use the ascii values directly. It's easy to parse the fields out of the GPS strings.

      Delete
    4. Yes it seems like all of these happen because of the memory constraint of the arduino. Yes you are right GPS chip return coordinates as ascii. But the format of these strings are unclear. That's why I have got DataTransfer method which parse this data and returns double instead. Hence, I have got some problems with sending big number and also if the number of the digit after decimal exceeds 4.


      double Datatransfer(char *data_buf,char num)//convert the data to the float type
      { //*data_buf:the data array
      double temp=0.0; //the number of the right of a decimal point
      unsigned char i,j;

      if(data_buf[0]=='-')
      {
      i=1;
      //process the data array
      while(data_buf[i]!='.')
      temp=temp*10+(data_buf[i++]-0x30);
      for(j=0;j<num;j++)
      temp=temp*10+(data_buf[++i]-0x30);
      //convert the int type to the float type
      for(j=0;j<num;j++)
      temp=temp/10;
      //convert to the negative numbe
      temp=0-temp;
      }
      else//for the positive number
      {
      i=0;
      while(data_buf[i]!='.')
      temp=temp*10+(data_buf[i++]-0x30);
      for(j=0;j<num;j++)
      temp=temp*10+(data_buf[++i]-0x30);
      for(j=0;j<num;j++)
      temp=temp/10 ;
      }
      return temp;
      }

      Delete
    5. I've decoded the output of a GPS chip, and it isn't too hard. Basically all it is is taking the NMEA strings, choosing the one you want, and then separating the fields from it. There's hundreds of posts on how to do this on the web. There's even a small library that uses the software serial library and allows the use of the hardware serial port for debugging and monitoring. The TinyGPS library is described here:

      http://arduiniana.org/libraries/TinyGPS/

      And, the software serial library is included as part of the recent Arduino IDE. The library returns integer values which are much easier to convert to string data to be forwarded through an XBee. The machine at the other end can then convert it to floating point for calculations.

      There are other methods that can be used especially since the GPS sentences are comma separated. Comma separated fields are not too hard to parse with an Arduino and there are example out there for this as well. I use the strtok() routine to parse this kind of data a lot.

      Delete
  5. Hi Dave,

    Do you know if there is a way to get the RSSI value in the format of a float or double? Currently I can only get values as integers but i'm looking for a bit more resolution.

    Cheers!
    Tom

    ReplyDelete
    Replies
    1. I've never found anything other than an integer using the DB command. They do talk about getting the rssi from a hardware pin (6) that they set to the value, but I've never used it.

      Delete