Don’t Waste Memory : Read From Streams

5 March, 2010 (09:01) | Uncategorized | By: joseanes

Just because we now build the most basic desktop with 16 GB of RAM, it doesn’t mean you want to read that 2 GB file into memory before processing it.
Two recent situations that re-iterate this idea:

JAVA JAXB
The other day we discovered a performance issue on one of the software applications we where building during the Quality Assurance phase. With small, unit testing input files the application was flying. With large files the application performance dropped quickly, worse than linearly. The input files where XML files being processed by JAXB.

We initially thought the culprit must have been some inefficiency inside JAXB but we soon discovered we were wrong on that initial assumption. Very simple log4j debug statements before and after a point where we converted a File to a String (prior to sending to JAXB) was the bulk of the time spent – a big waste of time since JAXB is great at processing File streams directly, without any need to convert to JAXB.

Lesson learned: whenever possible go from File or Stream to whatever processes your data. Skip loading into memory.

C Microcontroller Based HTTP Client / Server
On another project we were working on , we quickly discovered that using memory was not an option. These microcontrollers have 1 or 2 KBytes of RAM. That means that all of your variables must consume less than about one or two dozens of lines of text worth of data. That is tiny. Yet, you can serve a web server out of them, but only if you avoid RAM.

The secret is not to build the string that you intend to send back to the user in memory before sending it. Rather, you want to programmatically build each component and spit it out the Stream (network) as soon as it is available. That way you are not limiting the amount of data you can send.

Lesson learned: Treat your memory as a scarce commodity, and you will soon find that you can do wonders with very little resources.

  • Share/Bookmark

Pachube

18 February, 2010 (17:49) | Arduino, Data Center Energy | By: joseanes

Pachube is a data aggregator for time series data (things you can chart).  You dump data on it on a periodic basis and later on you can retrieve it.  Kind of a data hub.  Very useful on applications that are geographically and network dispersed (firewalled)  that you only want to access from around the world.  Could be a great idea for a Data Center Energy Monitoring Solution where you have a low density of elements (sensors) per location, but a widely dispersed and highly firewalled environment.  Also great for locations where you want to put instrumentation, but you do not want to put a full featured workstation to collect the data.  They are on an wide open beta.  Check them out.  And they do favor Arduino folks, and that is how I found the site.
I decided to try them out.  I may use them as a repository of sensor data on small, less sensitive projects.  For more security sensitive projects, I might use other alternatives.
My initial experimentation goal was to extend upon the libraries they offer for the Arduino and the Arduino Ethernet Shield.  I produced a library that I quite like:  20100218-pachube
This is the experiment I did:
My goal is to create a Pachube manual update client on the arduino that pushes EEML XML format to Pachube directly, without a PC in the middle.  I want to do this because the XML format is a lot better than the CSV one:  it allows me to tag the data from within the code with labels and units.  It is also cleaner to understand.  I also want to use the same EEML XML format for my own purposes.  So the same Arduino + Ethernet Shield that sends the data, I want it to provide the data itself as a web server for my internal applications as well as to pachube.
Eventually I will equip this machine with an XBEE to serve as a gateway to other applications, much like what Funnel IO is trying to do but eliminating the need for a PC in the middle.  (No sense in having a small device if it has to be connected to a full featured PC).
If someone has a better library, please let me know where to find it.  Here is a link to the code I am using to accomplish this purpose.  So far it is working well enough.  Some interesting points:
  • I had to print directly to the stream (ethernet in or out), because creating an EEML XML string was too much for the small memory of the Arduino.  Direct writing as a stream solves that.
  • I still want to avoid the reset every time it uploads to pachube (took that from the pachube CSV example).

The code: 20100218-pachube

  • Share/Bookmark

XBee / ZigBee

9 February, 2010 (09:46) | Uncategorized | By: joseanes

Digi has a mesh network device that, I must say, has impressed me.  The ZigBee / XBee product line can quickly convert a microprocessor / micro-controller(like a PIC or an Arduino) into a powerful network application.  They communicate with the micro-controller like a serial port (think COM ports in your computer), but transmit wirelessly in the 900Mhz / 2.4GHz frequencies.  The configuration is minimal (unless you want advanced features), and they have a range from a few hundred feet to a few miles, depending on the model you buy.

Given those properties, I decided I had to give it a try for the Arduino Data Center Energy Sensor Platform experiments.

Setup was incredibly easy.  I just had to:

  1. Download the management tool from Digi.  Installed.
  2. Connected the Xbee USB Explorer (Sparkfun $25)
  3. Inserted first XBee.
  4. Updated firmware.
  5. Set the low destination bits to FFFF – broadcast.
  6. Did 3-5 for the 2nd XBee.
  7. Plugged in the 1st XBee on the Arduino XBee Shield (Sparkfun $25) that I had for the sensor transmitter (see the Arduino posting).
  8. Leave the 2nd XBee on the explorer, and watch the serial input traffic come in.

It couldn’t be simpler.

  • Share/Bookmark

Data Center Energy Sensor Platform – Arduino Based

9 February, 2010 (09:39) | Uncategorized | By: joseanes

I have had the opportunity to work on Data Center Energy management projects.  They are great:  you see the payout as soon as you start instrumenting your environment, and the value grows every time you add sensors or data analysis and presentation software.

However, the biggest push back you get from current data-center owners is the cost of deploying the sensors:  It can be overwhelming at first.  Most of the time the cost is not on the devices itself, but on the cost of deploying them.  I have made integrations with ControlByWeb (a $200 USD device that provides temperature readings in HTTP / XML format).  ControlByWeb works great, but it has two drawbacks:  network wired (waste a switch port) and wired (waste a Power Distribution Unit electrical outlet).  It works for some customers, but not all.  Then there are other solutions like Synapsense - wireless sensors, but a bit on the expensive side – and a bit cumbersome and closed.  Great for another set of customers as well, but not for all.

What I was looking for was an open source, easy entry Datacenter Energy Management platform.  I think I have found a match – inspired by Libelium (Squidbee) and others I have found on the web: A micro-controller based wireless sensor platform.  (Note:  If I had to do a big scale project, I would probably contact Libelium or similar companies for mass production of the device).  Essentially, the solution must have:

  • Wireless capability of at least a few hundred feet (better if mesh).
  • Low power consumption (we do not want to waste power trying to reduce power consumption).
  • Easy to deploy: Something small that you can throw, Velcro or duct-tape inside every rack of your Data Center.
  • Easy access to power (plug in any USB port of any server, no need to interface with the server).
  • Very extensible – to a wide arrange of sensor equipment.  (Energy, Temperature, Humidity – at least).
  • Capability of multiple sensors per box.
  • Firmware upgradeable.
  • Open-Source.
  • Open standard output.
  • Inexpensive – Read:  less than $100 for a single sensor.

The Arduino micro-controller captured my attention some time ago.  It is programmable in a C-like language, has a serial / USB interface, plenty of digital and analog inputs, and there is plenty of enthusiasts out there willing to help in building projects with it.  Most importantly, many people have already made most of the effort on the projects that interest me: you just have to add a few details and it provides value.  For the wireless part, I decided to use ZigBee / XBee (see the implementation notes on that one) due to the quick implementation timeframe.

Once I received the components the sensor platform was working as expected in less than an hour (and that includes the effort to learn how it worked).

What I got:

  • Arduino USB board (Sparkfun: $30, Seedstudio Clone: $19)
  • XBee stamp (1mW, wire antennae, Sparkfun $23)
  • XBee shield (interface) for Arduino (Sparkfun $25 – Clones available for $10-15)
  • 10kohm thermsistor $2
  • 10kohm resistor
  • USB cable, some jumper cables, small breadboard
  • Code and simple schematic from Arduino playground.

General procedure:

  1. Get the Arduino IDE, install on your system, connect your Arduino via Serial or USB as per instructions.
  2. Program and set up the sensor as you would like it to work and make sure it outputs data to a serial port.  By default it comes with a very high speed on the serial port, but I decided to downgrade it to 9600 bps for easy interfacing tot he Xbee (although I think Xbees can work at higher speeds).  The Arduino playground example is a good starting point.  Actual code used on the project attached.  I connected the test voltage to pin Analog 0 as the code suggests.
  3. Check that you get output on the Serial output of your Arduino IDE.  (as attached picture).
  4. Prepare your XBee as described on the Xbee posting (with Destination address set to FFFF – broadcast).
  5. Insert your Xbee on the Xbee shield, and insert the shield on the Arduino board (you will have to temporarily remove the Analog 0 input and reconnect on the bypass plug on the shield.
  6. Your USB / Serial console will stop reporting measurements.  Your Xbee is capturing the serial output and sending it wirelessly.  Check on the other Xbee by watching the serial port on which that one is connected.

That is it!  You got a wireless sensor that feeds out of the widely available serial port.

Future of this test:

  • Better, digital sensors.
  • Humidity.
  • Energy (current), split-cores.
  • Getting it to be in the sub $50 USD range.
  • Experimenting with PIC micro-controllers.
  • Building a better receiver.
  • Recording the data to a database for reporting.

Code:

#include <math.h>
//Schematic:
// [Ground] ---- [10k-Resister] -------|------- [Thermistor] ---- [+5v]
//                                     |
//                                Analog Pin 0
double Thermistor(int RawADC) {
 // Inputs ADC Value from Thermistor and outputs Temperature in Celsius
 //  requires: include

 // Utilizes the Steinhart-Hart Thermistor Equation:
 //    Temperature in Kelvin = 1 / {A + B[ln(R)] + C[ln(R)]^3}
 //    where A = 0.001129148, B = 0.000234125 and C = 8.76741E-08
 long Resistance;  double Temp;  // Dual-Purpose variable to save space.
 Resistance=((10240000/RawADC) - 10000);  // Assuming a 10k Thermistor.  Calculation is actually: Resistance = (1024/ADC)
 Temp = log(Resistance); // Saving the Log(resistance) so not to calculate it 4 times later. // "Temp" means "Temporary" on this line.
 Temp = 1 / (0.001129148 + (0.000234125 * Temp) + (0.0000000876741 * Temp * Temp * Temp));   // Now it means both "Temporary" and "Temperature"
 Temp = Temp - 273.15;  // Convert Kelvin to Celsius                                         // Now it only means "Temperature"

 // BEGIN- Remove these lines for the function not to display anything
  Serial.print("ADC: "); Serial.print(RawADC); Serial.print("/1024");  // Print out RAW ADC Number
  Serial.print(", Volts: "); printDouble(((RawADC*4.860)/1024.0),3);   // 4.860 volts is what my USB Port outputs.
  Serial.print(", Resistance: "); Serial.print(Resistance); Serial.print("ohms");
 // END- Remove these lines for the function not to display anything

 // Uncomment this line for the function to return Fahrenheit instead.
 //Temp = (Temp * 9.0)/ 5.0 + 32.0; // Convert to Fahrenheit
 return Temp;  // Return the Temperature
}

void printDouble(double val, byte precision) {
  // prints val with number of decimal places determine by precision
  // precision is a number from 0 to 6 indicating the desired decimal places
  // example: printDouble(3.1415, 2); // prints 3.14 (two decimal places)
  Serial.print (int(val));  //prints the int part
  if( precision > 0) {
    Serial.print("."); // print the decimal point
    unsigned long frac, mult = 1;
    byte padding = precision -1;
    while(precision--) mult *=10;
    if(val >= 0) frac = (val - int(val)) * mult; else frac = (int(val) - val) * mult;
    unsigned long frac1 = frac;
    while(frac1 /= 10) padding--;
    while(padding--) Serial.print("0");
    Serial.print(frac,DEC) ;
  }
}

void setup() {
  // Use 9600 and FFFF for low destination address on the Xbees.
 Serial.begin(9600);
}

#define ThermistorPIN 0   // Analog Pin 0
double temp;
void loop() {
 temp=Thermistor(analogRead(ThermistorPIN));           // read ADC and convert it to Celsius
 Serial.print(", Celsius: "); printDouble(temp,3);     // display Celsius
 temp = (temp * 9.0)/ 5.0 + 32.0;                      // converts to Fahrenheit
 Serial.print(", Fahrenheit: "); printDouble(temp,3);  // display Fahrenheit
 Serial.println("");                                   // End of Line
 delay(10000);                                           // Delay a bit... for fun, and to not Serial.print faster than the serial connection can output
}
  • Share/Bookmark

Cisco VPN for Windows 7 x64

9 December, 2009 (15:21) | Uncategorized | By: joseanes

I was dissapointed when I found out that my Cisco 871W VPN wasn’t compatible with my new Windows 7 x64 laptop.  Fortunately there is a solution from the same vpnc that I use from my Ubuntu laptops.  The open source VPNC Front End, available on SourceForge.  Works like a charm.

  • Share/Bookmark

Embedded Jetty SSL / HTTPS

24 October, 2009 (06:31) | Java | By: joseanes

We love Jetty.  It is a very lightweight Java application server / container.  So light, you can just embed it inside your applications and forget about having to deal with web.xml and services.xml and all other files.  You can just distribute a very small executable .jar to your clients and forget about all of the other dependencies.

Yesterday I tried to Google how to use it with SSL and I didn’t found a good example on how to do it in embedded mode.  I will share the example here:

Server server = new Server();
SslSocketConnector connector = new SslSocketConnector();
connector.setPort(httpsServerPort);
connector.setKeyPassword(keystorePassword);
if (keystoreFile != "") {
	connector.setKeystore(keystoreFile);
}
server.setConnectors(new Connector[] { connector });
Context root = new Context(server, "/", Context.SESSIONS);
root.addServlet(new ServletHolder(new BaaisServlet()), "/");
root.addServlet(new ServletHolder(new BaaisServlet()), "/index.html");
server.addHandler(root);
server.start();

I find it extremely simple. Give it a try.  For keystore generation examples, visit the Jetty site.

  • Share/Bookmark

hexBinary WSDL: Axis2 Example to Upload Binary Files

19 October, 2009 (07:30) | Java | By: joseanes

We wanted to upload binary files, but lacked a few concrete examples on the web.  We use a combination of Eclipse tools to create the WSDL and Axis2 WSDL2Java to create the Skeleton / Stubs.  The WSDL creation tools allowed us to easily choose the type of the field as hexBinary (field called “binary” on the example, with an associated String field called “filename”, so that we knew how to call it when it arrived its destination).

The trick is converting (encoding) the binary into something easy to transfer as text on SOAP (hex).  Fortunately there are classes that simplify the encoding and decoding, as you will see below.

We found out that transfering a file is as easy as this:

WsStub stub = new WsStub(url);
PutFile f = new PutFile();
String filepath = "C:\\test\\test.txt";
byte[] bytes = org.apache.xml.security.utils.JavaUtils
			    .getBytesFromFile(filepath);
HexBinary b = new HexBinary(bytes);
f.setFilename("myFile.txt");
f.setBinary(b);
StatusResponse r = stub.putFile(f);

The server side, it will be as simple as this:

byte[] bytes = putFile.localBinary.getBytes();
org.apache.xml.security.utils.JavaUtils.writeBytesToFilename(localPath
		    + putFile.localFilename, bytes);


Some hints you might find useful:

Once you try it is fairly simple.   And it is fairly compatible with many SOAP implementations.

  • Share/Bookmark

Axis2 WSDL2Java Skeleton Properties File

22 September, 2009 (12:04) | Java | By: joseanes

The more I do it, the more I like doing web applications with embedded Axis.  Not too much documentation about them out there.  One of the things that is not easy to find is a way to load a properties file with some configuration parameters.

I did used WSDL2Java (through the eclipse plugin).  Fairly easy to get yourself started.

Do an init method on your class, and load the properties there.  You have to be careful about loading the path, and that is the tricky part.  I got mine to work by doing something like this:


public void init(ServiceContext serviceContext) {
  Properties props = new Properties();
  try {
    props.load(this.getClass().getResourceAsStream("myApp.properties"));
    targetURL = props.getProperty("targetURL");
    logger.debug(" init targetURL: " + targetURL);
  } catch (FileNotFoundException e) {
    e.printStackTrace();
  } catch (IOException e) {
    e.printStackTrace();
  }
}

For more information about how to create a properties file go to this example or to the Properties class definition.

Context

The next question you may ask is how do I circumvent the stateless nature of Axis and keep the variables I have modified for the next session to use.  You will have to use the concept of ConfigurationContext (applies to the whole axis application), as opposed to the ServiceContext (applies only to this session) that is passed to the init() method.  Fortunately, the ServiceContect knows its corresponding ConfigurationContext.  An example of something you may want to add in your init() function:


ConfigurationContext configurationContext = serviceContext
    .getConfigurationContext();
if (configurationContext.getProperty("numberOfCallsProcessed") != null) {
  numberOfCallsProcessed = (Integer) serviceContext
	    .getProperty("numberOfCallsProcessed");
} else {
  numberOfCallsProcessed = 0;
}
numberOfCallsProcessed++;
configurationContext.setProperty("numberOfCallsProcessed",
    numberOfCallsProcessed);
logger.debug(" variable - Number of Calls Processed: "
	   + numberOfCallsProcessed);

For more information on context there is a great article by developer.com.

  • Share/Bookmark

Email:Q/A Advice on Learning a New Programming Language

22 September, 2009 (04:46) | Uncategorized | By: joseanes

Contrary to popular perception, not all IT / computer people program.  Many have done it at some point in their lives but didn’t develop the skills further.  Recently we received an e-mail requesting advice on how to pick up some programming skills.  It read like this:

Recently, I really feel like picking up programming skills.  So I wonder if you may recommend some books or online learning resources, as well as an efficient learning approach.
Thanks!
- Future Programmer

Some key parts of this simple e-mail we got:

  • Previous knowledge.  Has this person programmed before?  If not, he/she may be up for a steep learning curve.
  • How deep to go?
  • How much time to spend?
  • For what goal?  Some people want to learn how to program for fun, others have a job description / position in mind.
  • What kind of activity will the programs do?  What kind of data?

The answer will depend on many of these questions.  There are many ways of advising someone and give some starting points.  Language choice is one of them (Perl / Python or Java or C#).  Approach (21 day book vs. Google it vs. instructor led training).

This is the answer one of our programmers  gave.  On it, the programmer made the assumption that this person was serious about it and that it would be for career purposes.  In those cases attempting a solid, widely used and in high demand language (like Java) could be a good choice:

Dear Future Programmer:

I don’t know what is your skill level in C++/Java so it is hard to recommend a book.  Most information is found online and that has been my main source of knowledge for the last few years. I can recommend you pick up Java instead of C++ not only because it is easier, but it is also in higher demand. If you have some basic knowledge of programming in any language (data types[int, double, array, etc], scope [global variable vs local variable, etc], control structures [if, for, while], objects [fields, methods, constructors, inheritance, etc]) and basic knowledge or databases (what is a DB, how to run simple queries) you may want to focus on learning a technology that is in demand and practice using Java.

For instance:

  • Learn about XML and how to parse it using available parsers (nobody writes parsers from scratch anymore).
  • Web Services: What is a web service, what is an application server, how to create a service and run it on an application server (keywords here: Tomcat, Axis, JAX-WS, among others)
  • Web Development (PHP, FLASH, Java Script) try to create a couple of good web pages, maybe something that access a database and updates it.

This is an excellent website to get introduced to current web technologies: www.w3schools.com. For Java related topics: http://java.sun.com/docs/books/tutorial/

For everything else you can pretty much google it and you will find it.

If you are not familiar with the concepts I mentioned at the beginning you can find online tutorials to pick up those concepts first.
I think most of the good technical books al sold by o’reilly:
http://oreilly.com/ if you want to go deeper into one of the technologies that
I mentioned or any other one.

If you decide to pick any of these I will be happy to help.

-Master Programmer

What would have been your answer?

  • Share/Bookmark

log4j 4 Tomcat

21 September, 2009 (13:00) | Uncategorized | By: Jose Anes

The problem:  I am used to specify a log file name on java applications and expect it to be on the same directory where the application is.  This may not be the case on Web Applications under tomcat (or under other applications servers).  The path is not relative to your application, but relative to where tomcat was started (which could very well be $TOMCAT_HOME/bin).  Not fun to discover a bunch of log4j log files under the bin directory of your application server.

The Apache Tomcat 6.0 documentation explains how to do logging for the server itself.  It is useful, but not exactly what I was looking for.  I want it to be at the application level.  It does indicates that you want your own:

  • log4j jar file inside your WEB-INF/lib
  • log4j.properties on your WEB-INF/classes directory.

Those are very good starting points, and just doing so can get debug based logging straight into the $TOMCAT_HOME/logs/catalina.out .

However, I did wanted them to go to files.  And this is where the things can get more interesting.  Specifying a hardcoded path on the log4j.properties file can do the trick (as suggested by dave), but that is not a very elegant solution, in my opinion.  I do not even know if my application will run under Unix or Windows.

StackOverflow provided me the solution:

log4j.rootCategory=DEBUG,errorfile
log4j.appender.errorfile.File=${custom.logging.root}/LogFilename.log

There are more elegant choices if you want to make it container independent by setting a System property, but the simple one above did the trick for me.

  • Share/Bookmark