David's Dev Notes

Using Date and Calendar – At your own risk!

2011-05-26T06:40:00.001-07:00

You can find lots of issues discussed on the ‘net about the problems of the Calendar/Date classes in Java. Here are a few that recently reared their ugly head in our code:

1. This one is actually not in either of these but in trying to output our Date. Beware of the mm instead of MM when using SimpleDateFormat. Yep, one of my team members used the following code:

SimpleDateFormat sdf = new SimpleDateFormat(“yyyymmdd”);

Surprisingly everyone in our tests had a birthday of the first of the month! I think that the right solution would be that mm is wrong and mi is minutes and mo is month. This would solve the confusion.

2. Did you know that there is a difference whether you use Calendar.HOUR or Calendar.HOUR_OF_DAY? Ok, i admit it this one is documented in the javdocs, but i would still think that it is a bit of surprise that in the afternoon, if you do

cal.set(Calendar.HOUR,0);

that you will get 12:00 PM!

3. Why does toString() of Calendar not put out the date in some format?

4. Beware that DAY_OF_WEEK is not zero based while MONTH is.

Using StAX to extract records from an XML file

2011-04-28T04:15:00.000-07:00

When processing large XML files, it is impossible to build a DOM of the whole file as you are sure to run out of memory. Java 6 includes StAX (it was around earlier of courser) and it is the best solution to deal with large XML files to build a scalable XML file processor. However, it still may be desirable to build a Document Model when processing an individual record due to decisions in code that may need to be made based on information lower in the node. Due to this requirement, we use StAX to extract each record from our gargantuan XML files and then pass to a different class that builds a DOM for the processing the record. Here is the very simple code with explanation of what we did.

public class StaxExtracter {

    XMLEventReader reader;

    boolean headerRead = false;

 

    public void openStream(InputStream s) throws XMLStreamException {

        XMLInputFactory inputFactory = XMLInputFactory.newInstance();

        reader = inputFactory.createXMLEventReader(s);

    }

 

    public void close() throws XMLStreamException {

        reader.close();

    }

 

    //function to allow us to read part of a document that does not include the header

    public void setHeaderRead(boolean val) {

        headerRead = val;

    }

The above is some boilerplate code needed for our class. The heavy work is the XMLEventReader, created in the openStream function on line 7.

Following this we have two main functions. The first is used to read the open XML tag of the document and any important attributes in this tag, and the second is to read an actual record. Since the class is written to read recursively all sub elements of a record, it is important that we not read the open XML tag of our document as a regular record, since that will mean we wind up reading our entire document as one record! However, since there may be need to work on part of a file, i allowed an override on line 15 above so we can skip reading the header. The open XML tag is obviously not a complete XML tag so we do some heavier work here, returning the attributes as a map for use by our application, in addition to the raw line which can be used upstream in our application.

Here is the actual code:

/**

     * Since header is not a complete element we need to do special treatment so we will read the header attributes

     * into a hashmap to save all the attributes besides returning the raw data

     *

     * @return the rawData of document start and header

     * @throws XMLStreamException

     */

    public String getHeaderAndAttributesAsMap(Map<String, String> attributes) throws XMLStreamException {

        boolean documentStart = false;

        StringBuffer rawData = new StringBuffer();

        while (reader.hasNext() && !headerRead) {

            XMLEvent e = reader.nextEvent();

            if (e.isStartDocument()) {

                documentStart = true;

            } else if (e.isStartElement()) {

                headerRead = true;

                StartElement startElement = e.asStartElement();

                for (Iterator i = startElement.getAttributes(); i.hasNext();) {

                    Attribute attr = (Attribute) i.next();

                    attributes.put(attr.getName().getLocalPart(), attr.getValue());

                }

            }

            //skip whitespace before document

            if (documentStart) {

                rawData.append(e.toString());

            }

        }

        if (!headerRead)

            return null;

        else {

            return rawData.toString();

        }

    }

Now, on to the main function for reading each element of our XML file:

public enum RecordStatus {BODY,NO_DATA,CLOSE_BODY_TAG,END_DOCUMENT}

 

RecordStatus getNextRecord(StringBuffer recordData) throws XMLStreamException {

    if (!headerRead) {

        throw new RuntimeException("need to read header line before calling this function");

    }

    RecordStatus rc = RecordStatus.NO_DATA;

    String recordName = null;

    boolean finishedReadingElement = false;

    boolean startOfRecord = false;

    while (reader.hasNext() && !finishedReadingElement) {

        XMLEvent e = reader.nextEvent();

        if (e.isStartElement()) {

            if (!startOfRecord) {

                startOfRecord = true;

                recordName = e.asStartElement().getName().getLocalPart();

            }

        } else if (e.isEndElement()) {

            EndElement endElement = e.asEndElement();

            if (endElement.getName().getLocalPart().equals(recordName)) {

                finishedReadingElement = true;

                rc = RecordStatus.BODY;

            }  else if (!startOfRecord) {

                //todo  we could make this better by e adding validation here to save the START_BODY_TAG

                rc = RecordStatus.CLOSE_BODY_TAG;

                finishedReadingElement = true;

            }

        } else if (e.isEndDocument()) {

            rc = RecordStatus.END_DOCUMENT;

        }

        //skip whitespace before element

        if (startOfRecord || rc == RecordStatus.CLOSE_BODY_TAG) {

            if (e.isCharacters())

                //need to re-escape characters so SAX parser wont choke on it

                recordData.append(StringEscapeUtils.escapeXml(e.toString()));

            else

                recordData.append(e.toString());

        }

    }

    return rc;

}

We use an enum to give us maximal information about what was read while return the actual raw data in a StringBuffer that is allocated outside our function.

Of course, you could do ALL the parsing in your StAX implementation as well, as described here and other places, but this code allows us to continue to use the more elegant DOM for our actual work.

Java in Hollywood -- Using the Actor Model in Java

2011-02-14T04:04:00.000-08:00

We have all been hearing how the programming paradigm has started to change, in that Moore’s law will no longer bail us out to speed up our applications and instead we we need to program better to use our multiple processors. Many have been pushing the Actor Model as the simplest way to get to solid multithreading without locks, etc. And for that reason, we are getting pushed to use Scala and other languages instead of Java. But you can also use these frameworks in your existing Java application so i figured i would give it a spin using Akka, unfortunately postponing yet again my first application in Scala.
For a starter, my app just needs to run a job asynchronously with no response, but this would be enough to familiarize myself with Akka and see how to use it.

So, after downloading Akka, the following dependencies are needed to get started:

Now, we can write a small class. Here is the code to send asynchronous messages and not get a response back.

public class MyActor1 extends UntypedActor {

 

    public void onReceive(Object message) {

        Integer numMessage = (Integer) message;

        System.out.println(Thread.currentThread().getName() + "\tincoming message " + numMessage);

        for (int i=0;i<1000000000;i++) {           

        }

        System.out.println(Thread.currentThread().getName() + "\tFinished Message " + numMessage);

        System.out.println(Thread.currentThread().getName() + "\tStopping on thread " + numMessage);

        getContext().stop();

    }

 

    public static void main(String[] args) throws InterruptedException {

 

        for (int i=0;i<50;i++) {

            Actors.actorOf(MyActor1.class).start().sendOneWay(i);

        }

       Thread.sleep(50000);

    }

}

Line 16 shows the code to launch the thread as we “start’ the class and then send the message. In this case we sendOneWay but other methods include sendRequestReply, and sendRequestReplyFuture, as you can see in the documentation.

When you run this application you will see that it launched 16 threads and uses them for the 50 jobs. I found it nice that it was expressive – sendOneWay, as well as to access the context to end the activity of this object.

The next step was learning how to customize the behavior of my thread pool, which led to this change to the application:

public class MyActor extends UntypedActor {

 

    public MyActor(MessageDispatcher dispatcher) {

        getContext().setDispatcher(dispatcher);

 

    }

    public void onReceive(Object message) {

        Integer numMessage = (Integer) message;

        System.out.println(Thread.currentThread().getName() + "\tincoming message " + numMessage);

        for (int i=0;i<1000000000;i++) {           

        }

        System.out.println(Thread.currentThread().getName() + "\tFinished Message " + numMessage);

        System.out.println(Thread.currentThread().getName() + "\tStopping on thread ");

        getContext().stop();

    }

 

    public static void main(String[] args) throws InterruptedException {

       final MessageDispatcher d = Dispatchers.newExecutorBasedEventDrivenDispatcher("dispatcher").withNewThreadPoolWithLinkedBlockingQueueWithCapacity(50)

      .setCorePoolSize(5)

      .setMaxPoolSize(6)

      .setKeepAliveTimeInMillis(60000)

      .setRejectionPolicy(new ThreadPoolExecutor.CallerRunsPolicy())

      .build();

 

        for (int i=0;i<50;i++) {

            ActorRef actor = Actors.actorOf(new UntypedActorFactory() {

                public UntypedActor create() {

                    return new MyActor(d);

            }

            });

            actor.start().sendOneWay(i);

        }

 

       Thread.sleep(50000);

    }

}

As you see, on line 18 we create the ThreadPool, in Akka terminology called a MessageDispatcher. There are a few different kinds as mentioned here. But this change means that we now need a constructor to receive this dispatcher for use. (Don’t worry Spring integration is not a problem to use DI for the actors).

And voila, we see in the printouts that indeed this is running in multiple threads. Of course, for the need of one way with no reply, there really is no need to use Akka since the code is pretty straightforward using plain old Spring + Java as follows, where we define our ThreadPool bean in the application context:

<bean id="taskExecutor" class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor" destroy-method="destroy">

        <property name="corePoolSize" value="5" />

        <property name="maxPoolSize" value="10" />

        <property name="queueCapacity" value="25" />

</bean>

Once we have this, use of the code to do the same things just involves: taskExecutor.execute(bean). So, in this specific instance, the overhead of adding the scala and akka jars would not be justified, but clearly for more significant multithreaded needs, this could be a powerful tool to simplify code and make it very readable.

Attachments in Axis 1.4

2010-12-27T08:17:00.000-08:00

In our "legacy" application, we have a series of classes implemented to connect to a certain 3rd party web service implementation. As this was implemented a number of years ago, it was done using axis 1.4. There are many web service calls, but the differences in their implementation is not that great, as all have core fields that are in common. Thus, the implementation we used was of the template method pattern. The use of this pattern was to encapsulate ALL the connectivity and security issues (client certificates) of using the web services in our base class. Each new web service is a trivial amount of work. All we need to do is to run wsdl2java to generate the classes, inherit our new implementation from the template base class, which then forces us to implement a number of abstract methods that include the URL of the service, as well as functions to return the request and response classes of the web service call. The template method also requests each service to add its unique data specific to that request.

So, even after our knowledgeable web services implementer moved on, this did not impede our ability to add new services, since all that plumbing was untouched code that need not be thought about. All that changed recently when we had a new web service call to add that uses attachments to send XML as an attachment. (This is third party software but I think i understand the logic of why this was done). Suddenly, the work was not as simple as it was before, and we had no axis 1.4 expert anymore and little place to turn anymore, as the world has moved forward past axis 1.4. But with such an elegant implementation it seemed worth trying to make this work within axis 1.4 and after much Googling (and even "Bing"ing), and finally some trial and error of coding, we came to an elegant solution to keep our template method pattern intact and yet still add this functionality. The main (only?) documentation i found about using attachments in axis 1.4 was here

Here is the code added to the template base class:

DataHandler attachment = getAttachment();

if (attachment == null) {

    response = call.invoke(new Object[]{request});

} else {

    AttachmentPart ap= new AttachmentPart(attachment);

    QName qn = new QName(namespace, "DataHandler");

    call.registerTypeMapping(attachment.getClass(), //Add serializer for attachment.

            qn, JAFDataHandlerSerializerFactory.class, JAFDataHandlerDeserializerFactory.class);

    call.addAttachmentPart(ap);

    

    call.addParameter("source", qn, ParameterMode.IN);

    response = call.invoke(new Object[]{request,attachment});

}

The first line calls a new method getAttachment which is not abstract (or final). This means that we can add this functionality without having to implement it in all the implementing classes. The default implementation of the base class returns null and the new class we have will obviously override this and return the attachment of type javax.activation.DataHandler.

If we have an attachment, the steps we need to take are:

On Line 5 we wrap this attachment in an Axis class AttachmentPart
On Line 7-8, we need to register this attachments using the serializer/serializer JAFDataHandler that will handle the serializing and deserializing of the attachment. Note that the name “DataHandler” is just what i made up.
Line 9 adds the attachment to the call
Line 11 adds the attachment as a parameter to the call
Line 12 includes our request class and the attachment in the actual call.

One other change we needed to make was that since until now we always had only one parameter, we did not need to register the first parameter, but now this became necessary in order to add the second.

And in these few lines since our use of the template method pattern enabled us to keep our complex implementation intact and still accomodate this new functionality.

Beware of Static Initializers (even in 3rd party code!)

2010-08-25T04:41:00.001-07:00

As i was just releasing a small patch to our application that uses POI, i noticed a bunch of errors at startup. Of course, I immediately thought i changed more than I thought and began combing thru the code to see why i was getting this error. The stack trace was pointing to an error in the code of POI, which had a index out of range on something that clearly was not related to anything i had changed in our code in a LONG time. But why was this error happening and why now?

After searching the web and coming up empty on this error, I began looking more carefully and realized that the data structure at hand is one that is statically initialized on demand. What that means is that it is one of those static functions that when called, checks to see if the given data structure is built and if not, builds it then. Well, that’s nice but in our multithreaded application, we had two threads simultaneously trying to build an Excel spreadsheet so one of them got thrown out due to this static initialization that was probably in progress.

The code looks as follows:

public static List getBuiltinFormats()
    {
        if ( builtinFormats == null )
        {
            populateBuiltinFormats();
        }
        return builtinFormats;
    }

So we have a static function using the builtInFormats. The actual function populateBuiltinFormats is synchronized but that just means that it cannot be invoked simultaneously, but if is partially populated then the second thread will think it is complete!

Since this is third party code, the simplest solution is at startup of my main to force the initialization of this static data structure, so i invoke HSSFDataFormat.getBuiltinFormats(); and voila, the issue has been solved.

As a Spring evangelist, I would point out that if you use Spring for functionality like this, and then in an init function that Spring uses, you will have the data populated. You could also just have it statically created and not conditionally. Otherwise, you would need to build the internal data structure to a temporary variable and only assign it when complete. Of course, that will mean a second check in the function populateBuiltinFormats to see that if by the time you got in, the data is already built, or maybe only allow this function to run once.

Tips For Maintainable Code

2010-08-17T04:10:00.001-07:00

Due to a need of late to maintain a lot of code (where of course not all of it has been written by me), I have been able to give a lot of thought to which tips I have read about in the past that strike me as the most important to have maintainable code. In many scenarios, companies simply rewrite their applications every few years and thus this issue may not be relevant but in my case where the lack of manpower makes this impossible, (and the application works so well there is no business justification for it), it has led me to think carefully how to set up a project and practices to have in place so that an application is maintainable and customizable in a reasonable way. Much of what i will be listing is based on the book Clean Code which I already blogged about previously here.

1. Keep functions short and simple – and if you need to add in more functionality, refactor it into a new function that clearly defines in its name what it does. In the book Clean Code, the insistence is that all functions do exactly one thing.

2. Building on number 1, use small functions as a way to comment “funny” things you need to do. Sometimes, we have small things we needed to do – set some parameter to some funny value or the like and the person who makes this change may or may not comment this, since at the time it was so obvious and it may not even make the comment for the check in, since many files were checked in simultaneously, but if you get in the habit of creating a small function, such “nullOutParameterSoThatRequestHandlesCorrectly”, this comment is forever part of the code and will not get lost. It also means that when someone comes along later and sees it he can be sure it is there on purpose and not something wonder – it seems to me that removing this line is ok but i am not sure…

3. Comprehensive Unit Tests. This is a great way to document your code since we can see the kind of input you expect to receive for a given function and how you expect to handle it. As has been written about extensively on the ‘net, when we see these unit tests, we have greater confidence to make changes since we have a baseline to evaluate our changes. One other advantage of the unit tests, is that if you added a new class with dependencies, and you are using Dependency Injection, like Spring, even if the code never made it to production we have a working example to build off for later use. This actually happened to me, where code was written and not released, but lacking an example of how to set up this class in my Spring application context led to my lack of confidence in using this class.

4. Regression Tests. While Unit Tests are imperative, So are regression Tests. These are automated end to end tests so that we have a baseline of how the application works, in dailsychaining multiple functions together, what data was passed from beginning to end and what was received. I have used TestNG to make up regression tests due to the amazing features of paramterized testing, enabling to set up use cases that run multiple times with different inputs. (I hope to write more about this in an upcoming blog)

5. use the Source Control and DONT save pruned code. Remove it. Don’t worry about it since you can get it from the source control. It just creates confusion.

Remember, that YOU may be the one left to maintain the code so its in your best interest to get it right when developing it…

Testable Code Vs. OO code

2010-06-07T04:41:00.000-07:00

Can we ever have a situation where the desire to have testable code is not 100% compatible with our Object Oriented Design? As i was working on some new code, i came across this issue and was happy to find that this is not a new issue as we see here. Even if you are not willing to accept the entire thesis, at the very least, we may want to use the thought of testing our code to drive some of our design decisions, since a robust, easy to run test suite is so important in software development, especially as projects grow in size and length and we can hardly remember what our code does anymore. An example of this in my project was my decision to implement the template method pattern where the template methods implemented by the child classes provide data back the parent class rather than invoke helper methods and provide greater functionality. This means that it is easier to test each new implementation and is less time consuming.

An additional aspect of this issue can be seen based on the book Clean Code, discussed in a previous post. While the focus there was on readability of the code, testability seems to overlap quite a bit with readability. The simpler we make our classes and our functions, the simpler for testing and finding the problems when test fail. Another aspect discussed at length there is the issue of parameters to function (or better yet, not having them). It is recommended instead to have members the contain the necessary data rather than pass it from function to function. (Beware, as this means your classes will not be thread-safe and will require a new instance for each use). From what i recall (not having access to this book right now), the main issue was readability, but here too, another fringe benefit is testability, especially if you are using TestNG (or JUnit 4). The reason for this, you create this class and populate the member data correctly and then can run tests repeatedly with little typing other than the method names. Writing and running the tests is now easier since you need not figure out all the parameters of a given function. Also, running multiple tests with different data (using TestNG DataProvider) is easier to provide.

There are two other ways that testability impacts design in my current project:

The first is in using interfaces even when i have no plans to implement another implementation of a member class. It is much easier to mock behavior and isolate a test if we use interfaces and then plug in a simpler implementation of our interface for test purposes. Of course, use of Spring here and the Spring application context makes this trivial

The second aspect is in determining the scope of functions in order to enable testing of private functions as discussed here. I agree wholeheartedly with Bob Lee in his post that your code will be better if private functions are tested individually and you do not leave all your testing to the public API.

I realize that much of what i write here is well known but i see that many of my peers are still not recognizing how important repeatable tests are. It is hard to see down the line when you are creating a new project from scratch. But, there are two ultra important reasons that we need to have a repeatable test suite. The first is documentation. As written about in Clean Code, Javadocs are not reliable. It happens too often in projects that programmers do not make sure to update javadocs as things change, and this is true for comments as well. The best documentation in addition to well written code with good naming (as discussed in my previous post about the book Clean Code), is the tests of the API, that show exactly how a function was meant to be used by the original author. It enables us to also what results it was meant to return and all error cases. The more tests on a function the more information provided. The second reason is maintainability. As our software has gotten larger/bloated, while staff has been pared down, a rewrite of our software is not truly possible. The confidence to make changes stems from good, repeatable tests.

A second part of the confidence to make changes in a larger project comes from a good regression suite, which i hope to discuss in an upcoming post.

Using a non default Calendar in the Google API

2010-04-14T04:20:00.000-07:00

All the examples in the Google Calendar API documentation involve using a default calendar. But i was looking at a project where i would have a shared calendar among users and this calendar would not be the default one. The simple code below demonstrates how to do this in Java. There are three parameter you will need to inject into this class (in my case via Spring) -- the username, password, and calendarname. Using these three you have what you need to work with the non default calendar.

public class CalendarUtil {

  private final static Logger logger = Logger.getLogger(CalendarUtil.class.getName());

  private String username;

  private String password;

  String calendarName;

  private URL calendarUrl;

 

  private CalendarService service;

 

  // The base URL for a user's calendar metafeed (needs a username appended).

  private static final String METAFEED_URL_BASE =

          "http://www.google.com/calendar/feeds/";

 

  private static final String CALENDAR_URL_SUFFIX = "/private/full";

 

  public void init() {

    service = new CalendarService("DR-Appt-App-1");

    try {

      service.setUserCredentials(username, password);

    } catch (AuthenticationException e) {

      throw new RuntimeException("Could not authenticate to Google Calendar with username: " + username, e);

    }

    String calendarId = null;

    try {

      calendarId = getCalendarId();

    } catch (IOException e) {

      throw new RuntimeException("Error caught trying to connect to calendar: ", e);

    } catch (ServiceException e) {

      throw new RuntimeException("Error caught trying to connect to calendar: ", e);

    }

    try {

      calendarUrl = new URL(METAFEED_URL_BASE + calendarId + CALENDAR_URL_SUFFIX);

    } catch (MalformedURLException e) {

      throw new RuntimeException("Bad URL: " + calendarUrl.toString(), e);

    }

    if (logger.isDebugEnabled()) {

      logger.debug("Url is " + calendarUrl.toString());

    }

  }

 

 

  //get id of correct calendar

  private String getCalendarId() throws IOException, ServiceException {

    String calendarId = null;

    URL metafeedUrl = new URL(METAFEED_URL_BASE + username);

    if (logger.isDebugEnabled()) {

      logger.debug("Url is " + metafeedUrl.toString());

    }

    CalendarFeed resultFeed = service.getFeed(metafeedUrl, CalendarFeed.class);

    for (int i = 0; i < resultFeed.getEntries().size(); i++) {

      CalendarEntry entry = resultFeed.getEntries().get(i);

      if (calendarName.equalsIgnoreCase(entry.getTitle().getPlainText())) {

        calendarId = new File(entry.getId()).getName();

        break;

      }

    }

    return calendarId;

  }



I have left out of the above the getters and setters to make it more concise.
Using the above class, you now can access the non primary calendar simply using the syntax:

calendarUtil.getService().insert(…)
or any other service function.

Not a Maven yet….

2010-02-22T04:12:00.001-08:00

Notwithstanding my post about using Gradle as a build tool, when i needed to start a new project i decided to use maven due to its amazing integration with an IDE (in my case Intellij), as well as its unbiquitousness.

So, as i have written previously, it is really nice to use Maven as you get started, since with no writing of any config files you have the build process working for you. You have a clean, compile, jar, etc with no work and no opening of an XML file. But that is all theoretical. As i attempted to get going with my project i encountered two really serious issues that took a while to figure out, as well as one issue that needs better resolution.

I will start with the more minor issue. When we use standard libraries that we need to add to our project, then things are pretty straightforward, in fact even better than not using maven. All you need to do is go to http://mvnrepository.com and find the item you want and then it gives you the lines to add to your pom, and voila, you have added your dependency to your project. Of course, in one case, this was not so simple. I wanted to use log4j, so i said, ok, I will just use the latest version. That turned out to be a mistake since all of sudden the compile was failing with libraries i had never heard of unable to be downloaded and installed. As it turns out, after a bit of googling, i learned that i needed to stay away from this latest version. But that is not even the issue i was referring to above. The issue i ran into was when i needed to use non standard libraries that are not in the mvnrepository site. This became difficult as i did not have an easy time figuring out what repository to use and what to add to my POM. In my case, i wanted to use Spring 3 and as far as i can tell, there is no browser for the Spring 3 release repository and thus no instructions for what my artifact ids would look like. (Hint to SpringSource – mention in the javadocs which jar is needed for a package). I wound up guessing based on one item i found somewhere and since i had downloaded the spring 3 distribution i was able to guess how the others would look and solved that issue. The second set of non standard artifacts were the google calendar ones. Here, i had to manually install them in my repository to get it going. Since it was a lot of jars, it was tedious, thought here is a nice link to make it easier, just be sure to use version 2 and not 1 for those things are now in version 2.

Now on the two major issues that i encountered. The first one was that my project would not compile successfully even though there was very little code in it other than use of the Google Calendar API. Finally after not getting anywhere for a while, i realized the issue must be Maven since it did compile in my IDE. As it turned, it was a documented Jira issue, but as the guys in the comments wrote there, who would have thought that default of maven today would not be java 5 at least?

The second one related to testing. Even though my most recent work has been using TestNG, i thought maybe i would give JUnit a try. When my JUnit tests were not being recognized, i figured out that this is because, once again, the default is junit 1.3. After a bit of searching i found what i need to do to add Junit4 to be the test runner. But, since i found the documentation of JUnit 4 lacking, i had decided to switch back but figured i would leave both dependencies in the POM. Well, that turned out to be a mistake too, as i eventually found here

But with these problems behind us, we trudge forward assuming that all should now be smooth sailing, until we need some other complex library….

Asynchronous Groovy

2010-01-27T02:36:00.000-08:00

In my previous post, i showed how simple asynchronous code in Spring 3 is. I wanted to see what it takes to write the same code in Groovy. Well, it is quite straightforward, using the GPARS library. The docs there show it to be quite powerful, but to be loyal to our simple example, here is the code:

Asynchronizer.withAsynchronizer(4) {ExecutorService service ->

      (0..9).each { num->

        service.submit({

        Thread.sleep(2000);

        println("Running for ${num} thread ${Thread.currentThread()}")}as Runnable)

      }

}

See how you specify the number of threads in the parameter to the closure withAsynchronizer.
There are lots more examples at the GPARS site.

Asynchronous Code with Spring 3 – Simple as an annotation

2010-01-13T06:09:00.000-08:00

After reviewing some of the features of Spring 3, i decided to test out the asychronous features via annotation. And below is all the code i needed in order to have a function executed asynchronously in my application:

        1: public class Asyncher {

        2:     @Async

        3:     public void myMethod(int i) {

        4:         try {

        5:             Thread.sleep(2000);

        6:         } catch (InterruptedException e) {

        7:             e.printStackTrace();

        8:         }

        9:         System.out.println("Running for " + i);

       10:     }

       11:  

       12:     public static void main(String[] args) throws InterruptedException {

       13:         ApplicationContext ctx = new ClassPathXmlApplicationContext("appContext.xml");

       14:         Asyncher a = ctx.getBean("asyncher", Asyncher.class);

       15:         for (int i=0; i<10;i++) {

       16:                 a.myMethod(i);

       17:         }      

       18:  

       19:         ThreadPoolTaskExecutor t = ctx.getBean("myExecutor",ThreadPoolTaskExecutor.class);

       20:         while (t.getActiveCount() > 0) {

       21:             System.out.println("Number of tasks left: " + t.getActiveCount());

       22:             Thread.sleep(2000);

       23:         }

       24:         t.shutdown();

       25:     }

       26: }

Hard to believe, but all i needed to do was add the Async annotation on line 2 to my function and it works. of course we need a few lines in our application Context to make this work, but here they are:

<?xml version="1.0" encoding="UTF-8"?>

<beans xmlns="http://www.springframework.org/schema/beans"

       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:task="http://www.springframework.org/schema/task"

       xmlns:context="http://www.springframework.org/schema/context"

       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/task http://www.springframework.org/schema/task/spring-task-3.0.xsd http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd">

 

    <task:annotation-driven executor="myExecutor" />

    <task:executor id="myExecutor" pool-size="2"/>

    <bean id="asyncher" class="Asyncher"/>

 

</beans>

This same method can be used to create scheduled jobs as well, by changing or adding on line 7 the scheduler attribute and then adding in a scheduler bean

<task:scheduler id="myScheduler" pool-size="10"/>

You can read about it in the docs here

Creating a Zip file from a Stream

2009-12-28T06:44:00.000-08:00

Given an InputStream generated for me, i wanted to zip it up and email it. From the java almanac i saw that i could easily do this if i write the stream to disk first. But this seemed like an unnecesary step for me. All i needed to do was to attach this outputstream to my email code and get it done. So, after a bit of playing, I got it working. Here is the code:

private byte[] createZipFile(InputStream in, String fileName) throws IOException {

        ByteArrayOutputStream zipOutputStream = new ByteArrayOutputStream();

        ZipOutputStream zippedFile = new ZipOutputStream(zipOutputStream);

        byte[] buf = new byte[1024];

        zippedFile.putNextEntry(new ZipEntry(myFileName));

        // Transfer bytes from the file to the ZIP file

        int len;

        BufferedInputStream bs = new BufferedInputStream(bds[0].getInputStream());

        while ((len = bs.read(buf)) > 0) {

            zippedFile.write(buf, 0, len);

        }

        zippedFile.closeEntry();

        bs.close();        

        zippedFile.close();

        return zipOutputStream.toByteArray();

}

By wrapping the ZipOutputStream around a ByteArrayOutputStream, i am able to access the byte array and use that to attach to my email and send out.

Gradle, my new build tool?

2009-11-30T00:39:00.001-08:00

Last week, i had the pleasure of attending the JavaEdge conference. The keynote speaker was Ted Neward, and you could read his thoughts about the conference here. I enjoyed his lecture very much whose theme was that new programming languages in the VM is the future and we should all get used to it and get comfortable with it.

One of the sessions that i attended was “Your Next Successful Build”, given by Baruch Sadogursky, who used a slick tool instead of Powerpoint for his presentation, though it was a bit dizzying. I was happy that for the first time, i heard someone say some of my heretical thoughts – that while Maven has a lot of positive, it has many warts, mostly for the reasons of lack of documentation as well as dependency management. As a side note, i was amazed to see that statistics show that 44% of companies are using Maven, which probably means they are spending countless hours debugging build issues! It was also refreshing to hear him mention the pros of Ant as well as something he wants to be able to continue to use in his new build environment. But the most exciting part was to hear from Baruch that there is a tool out there that answers the call and it is Gradle. On the one hand, we like the standardization of the file layout of maven, but Gradle adds in better documentation (200 pages!!!), and the ability to use a scripting language, Groovy, which also means you automatically have full access to Ant via Groovy. And of course, he says that dependency management has been fixed as compared to Maven 2. I am hoping to try it out in my next project.

Why did Hibernate do that?!

2009-11-23T05:57:00.001-08:00

As i have mentioned in the past, i am not a fan of Hibernate and other ORM tools that generate the SQL for me. This week i have yet another reason for not being willing to consider changing my opinion.

Unbeknownst to me, in our Hibernate application when we were using an “update” to update the one or two columns we change during the regular running of our application, we discovered that actually the update was updating all fields as well as cascading to additional tables joined to this object, which included a CLOB that was getting and update called on it and causing a lot of overhead.

After being warned by our DBA’s to fix ASAP, our next step was to reproduce the problem in our environment so we quickly turned on the “hibernate.show_sql” property to see our SQL. And before we even got to the issue we were looking to solve, we discovered another perplexing Hibernate-ism. We found that when we were doing an initial load or find by id, we were seeing an update happening to the record at the same time as the find/load!

After some fishing around the internet, we understood why this was happening. It was because one of out getters was changing the value from null to something more meaningful, as we see here. When we altered the code to not have this happen, then that update was prevented.

Now onto the problem we were meant to deal with (isn’t always the case that this happens!). After not seeing the results we were hoping for through setting some of the parameters mentioned in various forums (dynamic-update and select-before-update) in an attempt to have our Hibernate “update” not cascade unnecessarily through all the objects and update them all, we settled on a very iBatis like solution. We created the update ourselves so that we know what we are updating, i.e:

String hql = "update myTable set field1 = :field1 where id = :id";
Session session = SessionFactoryUtils.getSession(getSessionFactory(), false);
query = session.createQuery(hql);
query.setString("field1", myObj.getField1());
query.setLong("id",myObj.getId());
query.executeUpdate();

Of course, once you are doing this, you have lost the “advantage” of Hibernate.

Java Web Start – putting my application out there

2009-11-16T03:26:00.000-08:00

I recently decided to write a small Java Web Start application using Groovy and Swingbuilder (and HttpBuilder). As i had no experience with Java Web Start, there was a bit of a learning curve. I knew that i would need signed jars in order for my application to be able to write to the local file system. However, a true signing certificate can be very expensive so i realized i would best serve my needs with a self signed certificate that i would keep renewing every 6 months. Using Ant, i have been able to have my app easily deployed with new signed jars.

It is important to mention that there is one simple solution that I did not use for my application, which is to use Griffon, as it takes care of all this stuff behind the scenes for you. However, since I had a simple application and had begun before i knew about it, I decided not to go this route and do it myself, which also ensures i have a better understanding of what is going on. Griffon will still be useful to you for reference. I especially found it useful to look over a sample jnlp file, the file used to list dependencies and configuration information

For signing the jars, i created a self signed certificate. However, every 6 months, i need to renew this certificate, which is pretty straightforward:

keytool -selfcert -keystore keystore -alias myAlias

After re-creating the certificate, i use ANT to sign and deploy. As Ant has a built in task for this, it is trivial:

<target name="signJars" description="sign the jars" depends="makeJar">

        <mkdir dir="signed"/>

        <mkdir dir="signed/lib"/>        

        <signjar destDir="signed"

                 alias="mastertorah" keystore="keystore"

                 storepass="changeit"

                 keypass="mastertorah"

                 >

            <path>

                <fileset dir="dist" includes="**/*.jar"/>

            </path>            

        </signjar>

        <signjar destDir="signed"

                         alias="mastertorah" keystore="keystore"

                         storepass="changeit"

                         keypass="mastertorah"

                         >

                    <path>

                        <fileset dir="dist" includes="**/*.jar"/>

                    </path>            

        </signjar>

    </target>

In case you are wondering, line 3 above creates a second folder since my signed dependency jars sit separately from the main code; i just left this repeated signing command out of the snippet above.

As you see, line 4 uses the built in ant task signjar and it references my keystore. So, now i have a simple build process to easily redeploy my application.

For users reporting problems after deployment, one thing i have found helpful is to tell them to enable debugging, which is set in the control panel on Windows machines (in the java option there). This way they can send you a log of what went wrong.

I am left with one question. Why should i use SwingBuilder when i can drag and drop widgets using an IDE and get the format of my layout more simply.

Setting Version in your manifest using ANT

2009-10-28T06:07:00.001-07:00

While the Gant folks will tell you that is not natural to use XML for your makefile as it is nicer to use a real scripting language (and perhaps today’s post will enforce this for you!), it is doable and in my opinion readable. The below is an excerpt of our ant buildfile that we use for upping the manifest version in an interactive way with the the person doing the build.

We added the ant-contrib task in order to enable us to have “if” statements in the XML, so we define the task as follows:

        1: <taskdef resource="net/sf/antcontrib/antlib.xml">

   <classpath>

      <pathelement location="${3rdParty.home}/ant-contrib/ant-contrib-1.0b3.jar" />

   </classpath>

</taskdef>

This will be used below to drive the logic if the user wants to keep the same version of the manifest or change it.

There are two macros that we have defined, loadmanifest – which reads and parses the manifest and createmanifest – which invokes loadmanifest and then interactively decides whether to write a new manifest or not.

Here is loadmanifest:

<macrodef name="loadManifest">

  <attribute name="manifest" />

  <sequential>

    <echo message="about to load file: @{manifest}"/>

        <loadfile property="@{manifest}.build.version" srcFile="@{manifest}">

            <filterchain>

  <filterreader classname="org.apache.tools.ant.filters.LineContains">

                        <param type="contains" value="Implementation-Version:" />

                </filterreader>

                <tokenfilter>

  <replacestring from="Implementation-Version: " to="" />

                </tokenfilter>

                <striplinebreaks />

          </filterchain>
          </loadfile>
        </sequential>
    </macrodef>

On line 5 we use the loadfile task to read the version number into a property that we will display to the user. We then use the very powerful filterchain to get to the correct line and the correct section of the line with the version number. For lack of a better way, we strip away the text part of the line (Implementation-Version:) on line 11, and are left with just the version number to be stored in the loadfile property @{manifest}.build.version.

Using this property we can now query the user to determine if the version number suits him or not, and then if necessary put out a new manifest file. This is the macro below:

<macrodef name="createManifest">
        <attribute name="manifest" />
        <attribute name="jarDescription" />
        <sequential>
            <loadManifest manifest="@{manifest}" />
            <echo message="Current Version: ${@{manifest}.build.version}" />
            <input message="Enter new version?" validargs="y,n" addproperty="@{jarDescription}.do.newversion" />
            <if>
                <equals arg1="${@{jarDescription}.do.newversion}" arg2="y" />
                <then>
                    <input message="Please enter new version number" addproperty="@{jarDescription}.new.build.version" />
                    <manifest file="@{manifest}">
                        <section name="${@{jarDescription}.jar.description}">
                            <attribute name="Implementation-Title" value="${@{jarDescription}.jar.description}" />
                            <attribute name="Implementation-Version" value="${@{jarDescription}.new.build.version}" />
                        </section>
                    </manifest>
                </then>
                <else>
                    <property name="@{jarDescription}.new.build.version" value="${@{manifest}.build.version}" />
                </else>
            </if>
        </sequential>
    </macrodef>

Other than “if”, the rest are all standard ant tasks that you can look up in the manual on the ant site.

The correct thing to do at this point would be to create a parallel GANT site. Unfortunately, i dont have one ready at the moment so this will have to wait until i begin to play with GANT

SVN Build automation with ant

2009-10-19T03:13:00.001-07:00

As i indicated in my last post, we use Ant to create our new scripts. In response to some requests, i am sharing here our “commonbuild.xml” file that shows how we do the key parts of this build in a reusable way. Before you get started you will need to make sure you have late version of ant (1.7 please) and you will need to add some libraries to your ant lib folder. These are the subclipse libraries as well as commons-net if you want to send emails at the end of your build.

Here is is a snapshot of all new libraries added:

The first 4 are needed for mail functions and the last 4 are added from svn. I am using svn 1.5 due to the fact that i have not yet updated my Intellij past version 7 so only 1.5 is supported.

In addition you need to install collabnet subversion client (windows) for this to work.

For Svn Purposes, we have set up the following task:

  1: <taskdef resource="org/tigris/subversion/svnant/svnantlib.xml">
  2:         <classpath>
  3:             <fileset dir="${3rdParty.home}/svnant-1.2.0-RC1/lib" includes="**/*.jar" />
  4:         </classpath>
  5: </taskdef>

Using it is pretty straightforward. Below is a macro that we use for commit and tag:

  1: <macrodef name="commitAndTag">
  2:         <attribute name="commitMessage" />
  3:         <attribute name="tagName" />
  4:         <sequential>
  5:             <echo message="checking in dist folder and manifest versions" />
  6:             <svn>
  7:                 <commit message="@{commitMessage}">
  8:                     <fileset dir="${trunk.dir}">
  9:                         <include name="dist/**" />
 10:                         <include name="**/build/**.txt" />
 11:                     </fileset>
 12:                 </commit>
 13:             </svn>
 14:             <svn>
 15:                 <copy srcUrl="${svnUrl}/${svn.projectName}/trunk" destUrl="${svnUrl}/${svn.projectName}tags/@{tagName}" message="@{tagName} tag" />
 16:             </svn>
 17:         </sequential>
 18:     </macrodef>

And to invoke this macro is simple:

  <commitAndTag commitMessage="${release.version}" tagName="${release.version}" />

In the next post I hope to show the macro used for editing the manifest file and upping the version number.

Time to become a Maven?

2009-10-15T02:14:00.001-07:00

I was afraid to post about the fact that we have not moved over all my builds to maven, but i saw that the iBatis team only recently moved. And anyway, last year at a Java conference, the presenter about Maven talked about how only recently had the Maven folks gotten their act together to the point where he was not pulling out his hair over new jars that appeared and changed dependencies happening without his knowing. (I really hate things that happen without my knowing!)

I will start by saying that all our current projects are happily using Ant. We have built some pretty nice scripts to create a full build that include using svn (subclipse) to get the latest version of our code, up the version in the manifest, commit, create a tag, and copy the file to the distribution location. Additionally, the start of a project by our small teams have usually been through a project started in an IDE, (Intellij), and after we have something running, we then take our project and create an ant build, using our existing scripts. So, using maven does not easily fit this model since at this point we already have an existing structure. But everybody is doing it nowadays, so we cant easily dismiss it.

So, we have done a few projects starting with Maven. And of course, it meant we had to break our methodology described above. We used Maven to create the original infrastructure for the project as well as the project file for Intellij. And, as the project would progress and added more dependencies, it seems that the simplest way to move forward was not to add the dependencies to the project in Intellij, but to fix the POM and then have maven regenerate the project file. I believe this is fixed in the latest version of Intellij where it knows how to sync with the POM.

So, what are my conclusions?

First of all, the documentation needs improvement! But, notwithstanding that, I think it makes sense to use Maven to create new projects and this will mean that you will have a simple structure where it is obvious where to put all your additional resources, config files. The alternative would be that we need to keep worrying about these “copy” lines in my ant file.

But the power of ant to do what you want, and use legacy processes (like our use of JWSDP 1.6) would mean that we still want ant in our build process. Luckily this is possible, as I saw here.

Of course, Ant will remain a key tool for us as i use it to run apps and more easily build classpaths and pass params to it than alternatives that I know of. And as a big believer in “If it ain’t broke don’t fix it”, i dont see us moving our old applications to Maven.

Test Driving IBATIS 3 (and Spring3)

2009-09-09T03:50:00.001-07:00

As a big fan and heavy user of iBATIS SqlMaps, i have wanted to give iBATIS version 3 a look for some time, but due to lack of time and limited documentation, I had not gotten around to it. But the beta version is now here, and with it a very detailed and well written document, and I finally had some time between projects, so I figured i could delay no longer. I immediately headed to check the latest version of Spring (3.0M4 as of this post) but i see that there is no built in support for it yet, though there is some JIRA activity on the iBATIS and Spring side trying to make this happen.

The goal of my first encounter with iBATIS 3 was to get a small project up and running, where the iBATIS session would be provided to me by the Spring container. This would give me the datasource and transactions power of Spring even though i would not have the full DataAccessException hierarchy and Runtime Exception changes.

As the iBATIS folks recommended, i am using maven to create the project (a tool i have a lot of reservations about), and am followiing their folder structure. In order to do this with Spring 3 M4, i needed to add the following repository to my pom.xml

        1: <repository>   2:     <id>spring-releases</id>   3:     <name>Spring Maven RELEASE Repository</name>   4:     <url>http://s3.amazonaws.com/maven.springframework.org/release</url>   5: </repository>

The end result is that my project looks like this:

As recommended in the documentation, I added a new folder within my package structure called “data” where all my configuration files will be stored. These files, are my Spring Application Context (applicationContext.xml), the SqlMapConfig file for Ibatis configuration (minus datasource information), and the Mapper XML file. There is one new file there, an interface with the same name as the SqlMap file. This is a new feature of iBATIS3 that enables type safety. Each mapping will have a function defined in this interface. I may also choose to implement this interface for my dao functions, so the interface could be useful.

Let’s start with the application Context. It seems to me that Spring has come a long way to solve the length of our application context, by enabling the use of annotation in our java code, as well as the shorthand notation in our XML file. So, here is my very simple application Context that contains 2 beans, the datasource and a call to a static method used by iBATIS to read the SqlMapConfig.xml. This file still contains the mapping files as well as global settings, but obviously no datasource settings. Of course, since i will be using JavaConfig, i need to configure the annotation configuration as well. So, here it is

   1: <?xml version="1.0" encoding="UTF-8"?>   2: <beans xmlns="http://www.springframework.org/schema/beans"   3:        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"   4:        xmlns:context="http://www.springframework.org/schema/context"   5:        xmlns:aop="http://www.springframework.org/schema/aop"   6:         xmlns:p="http://www.springframework.org/schema/p"   7:        xsi:schemaLocation="   8:     http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.5.xsd   9:     http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-2.5.xsd">  10:    11:    12:     <context:property-placeholder location="classpath:properties/config.properties"/>  13:     <context:annotation-config/>  14:     <context:component-scan base-package="com.dr.ibatis3app"/>  15:    16:    17:     <bean id="dataSource"  18:           class="org.springframework.jdbc.datasource.DriverManagerDataSource"  19:                 p:driverClassName="${jdbc.driver}"  20:                 p:url="${jdbc.url}"  21:                 p:username="${jdbc.username}"  22:                 p:password="${jdbc.password}"/>  23:    24:    25:     <bean id="ibatisResourceReader" class="org.apache.ibatis.io.Resources"  26:           factory-method="getResourceAsReader">  27:             <constructor-arg value="com/dr/ibatis3app/data/SqlMapConfig.xml"/>  28:     </bean>  29:    30: </beans>

The rest of the beans needed for my project are defined below in the JavaConfig file. The reason is obvious when you see the code:

   1: @Configuration   2: public class AppConfig {   3:     @Resource   4:     DataSource dataSource;   5:     6:     @Resource(name="ibatisResourceReader")   7:     Reader sqlMapConfigReader;   8:        9:     @Bean  10:     public SqlSessionFactory sqlSessionFactoryBuilder() throws IOException {  11:         System.out.println("Calling SqlSessionFactoryBuilder build method");  12:         SqlSessionFactoryBuilder builder = new SqlSessionFactoryBuilder();  13:         return builder.build(sqlMapConfigReader);  14:     }  15:    16:     @Bean  17:     @Scope("prototype")  18:     public SqlSession sqlSession() throws DataAccessException {  19:         try {  20:             return sqlSessionFactoryBuilder().openSession(dataSource.getConnection());  21:         } catch (SQLException e) {  22:             throw new DataSourceLookupFailureException("Unable to open session");  23:         } catch (IOException e) {  24:             throw new DataSourceLookupFailureException("Error opening sqlconfig");  25:         }  26:     }  27: }

Rather than using messy method calls in the XML file, i can have a few simple lines of Java that handle the instantiation of the SqlSession, yet the resource file name is injected from the XML file so this is easily changed. The SqlSession is defined as scope prototype as it is not thread safe.

The mapper file is straightforward, with the main change being the required namespace to match the exact location of the interface and XML files.

   1: package com.dr.ibatis3app.data;   2:     3: public interface CardMapper {   4:     public String getPrimAcctNum(int acctId);   5: }   6:     7: //XML file is below   8: <?xml version="1.0" encoding="UTF-8" ?>   9: <!DOCTYPE mapper  10:         PUBLIC "-//ibatis.apache.org//DTD Mapper 3.0//EN"  11:         "http://ibatis.apache.org/dtd/ibatis-3-mapper.dtd">  12: <mapper namespace="com.dr.ibatis3app.data.CardMapper">  13:     <select id="getPrimAcctNum" parameterType="int" resultType="String">  14:         SELECT prim_acct_num FROM acct WHERE acct_id = #{id}  15:     </select>  16: </mapper>

And now we use the code:

   1: public static void main(String[] args) {   2:        3:     ClassPathXmlApplicationContext ctx = new ClassPathXmlApplicationContext("com/dr/ibatis3app/data/applicationContext.xml");   4:     SqlSession session = ctx.getBean("sqlSession", SqlSession.class);   5:     try {   6:         CardMapper mapper = session.getMapper(CardMapper.class);   7:         String  cardNum = mapper.getPrimAcctNum(4185576);   8:         System.out.println("CARDNUM IS " + cardNum);   9:     } finally {  10:         session.close();  11:     }  12: }

And that’s it. We have a working application, and can start to kick the tires a bit.

Grails and the Domain Model, or why I prefer IBatis

2009-08-19T01:32:00.001-07:00

I have spent many hours playing with Grails, to see how much effort it would be to take one of our existing apps and port it to Grails, since, as i have written previously, i prefer not to have to deal too much with GUI, so if Grails gives me a lot automatically, then I would be very happy using it.

At the same time, i have been using Grails to write a customized version of Quicken for my own personal budgetary needs (unfortunately, it has not progressed as well as i would want due to lack of time).

I will start with the second project first. As advertised, Grails is an absolutely amazing framework to take an idea and make it functional really fast. Just set up your domain classes and you automatically get a whole web site up and running, from which you can customize the columns and edits. And in the case of my personal app, the domain model is a few tables without any complex joins.

However, when trying to set up a Grails application on an existing database, this creates a lot of problems. I will start by saying that I am not a big fan of Hibernate either for a similar reason. In most reasonably sized companies, you have a pretty complex database (or set of databases) and in my opinion, it is easier to write queries than to try to “teach” the entire schema to your application. There are solutions like creating views, but then inserts will still be a problem. With Hibernate though, i found it possible with some effort to get around most of the issues. But in Grails, if i want the advantages of the framework, i cannot veer too far from the default domain model. After a lot of frustration trying to make the domain model work with our database, i found the best way to work was to keep tweaking it using the HSQLDB and seeing the scripts generated until they matched. In the end, i opened two issues that prevented us from moving forward, and both have been addressed, 3616, and 4158. So, I guess now i am ready to resume.

But in the end, i still believe that the model IBatis provides is more appropriate for a company application. It means that the developers wears two hats – one of the SQL developer, writing and tweaking queries until they are just what you want and then putting on the programmer hat to use those queries that are mapped within an XML file. I see that IBatis version 3 is out, though every time i have checked i did not see too much documentation about what is different. If you are new to IBatis, I can tell that in version 2, the documentation is great, and it definitely is easy to get up and running with it, especially if you use Spring together with IBatis. And there are many nice advanced features that can take you quite far.

Technorati Tags: Grails,IBatis

Clean Code – Programmers are Authors!

2009-06-30T06:32:00.001-07:00

I recently read the book Clean Code, by Robert C, Martin, but I waited to comment on it until i saw how it fundamentally changed how I write code. I see there are quite a few blogs discussing it but i wanted to give my take on it, and specifically, in the short term, as I hope that other aspects will affect me more in the long run as well. (like improving my unit testing!)

The first powerful principle, laid down at the beginning is that “We are authors” and he develops this point two key ways that I thought i would share:

1. We read a lot of code.

Yes, its true. Whether like me, you are now reading others’ code as you have inherited their code, or you need to re-read your own code that you wrote 6 months ago and cannot remember, we are always reading our code. When we write complex code, long functions, etc, even our code takes way too long to remember what it does

2. Our code is our only reliable way to know what the code does. Comments are unreliable. Yes, its true – i see it in our application. I see that someone from our team changed code and left the now irrelevant comment in place. So, what does that mean? That the only effective and reliable way to document code is to write it in a way that it is readable. (yes, there are needs for comments on a class, but on specific lines of code, no)

So, how do we do that? Well, go ahead and read the book! But i will start and say – keep the functions short, keep the names of the functions descriptive so that the name of the function itself makes it clear what you will do in the function (and of course, don’t do too much in the function)

Irregular Expressions

2009-06-16T03:39:00.001-07:00

A friend sent me the following code that did not work as he expected

        1: public static void main(String[] args) {       2:         3:     String str = " status=\"The word deleted is in this sentence and no carriage return";       4:     String str1 = " status=\"The word deleted is in this sentence and carriage return\n";       5:         6:     assert str.matches(".*deleted.*");       7:     assert str1.matches(".*deleted.*") : "carriage return threw off the regex";       8: }

If you run this code you will see that carriage return is not considered “.*” by default. Of course, a little looking around helped us realize that this was documented,

The regular expression . matches any character except a line terminator unless the DOTALL flag is specified

In other words, we need to use the ugly Pattern class in order to make this work, as follows:

   1: public static void main(String[] args) {       2:         3:         String str = "The word deleted is in this sentence and no carriage return";       4:         String str1 = "The word deleted is in this sentence and carriage return\n";       5:         6:         Pattern p = Pattern.compile(".*deleted.*", Pattern.DOTALL);       7:         Matcher m = p.matcher(str);       8:         assert m.matches();       9:         m = p.matcher(str1);      10:         assert m.matches();      11:        12:     }

The key to making this work is the parameter Pattern.DOTALL used in the compile.

And how about in Groovy? It works of course!

   1: String str = " status=\"The word deleted is in this sentence and no carriage return";       2: String str1 = " status=\"The word deleted is in this sentence and carriage return\n";       3: assert str =~ /.*deleted.*/       4: assert str1 =~ /.*deleted.*/

As written about extensively, one of the really well done features of Groovy is definitely the regular expression syntax but here we see that the implementation was also improved.

Batch loading to the database – Part 2 – SimpleJdbcInsert

2009-06-16T03:05:00.001-07:00

As mentioned in my last post, we will now have a a method foe insertion of a delimited file into a database that does not involve our DBA’s.

In Spring 2.5, the SimpleJdbcInsert class was added making this task especially easy, and the added bonus here is we see the elegance of Groovy when combined with this class.

Here is the code.

        1: class DBInsert {      2:        3: DataSource ds;      4: SimpleJdbcInsert jdbcInserter;      5: int numRowsPerBatchInsert = 500;      6: int numRowsProcessed = 0;      7: List fieldNames = ['Column1', 'Column2', 'Column3']      8: List<Map<String, Object>> rowsToInsert = [];      9: java.sql.Date today;     10: final static String SchemaName = "myschema";     11:       12: public void init() {     13:   jdbcInserter = new SimpleJdbcInsert(ds).withSchemaName(SchemaName).withTableName("myTable");     14:   today = new java.sql.Date(cal.getTimeInMillis())     15: }     16:       17: /**     18:  * We assume that the row is delimited by commas.     19:  * This can be refactored to take some other delimiter as second parameter     20:  */     21: public void insertRow(String s) {     22:   if (numRowsProcessed++ > 0) {     23:     def rowAsList = s.split(",")     24:       25:     def iterator = rowAsList.iterator()     26:     Map<String, Object> insertMap = ['eff_dte': today]     27:     fieldNames.collect {     28:       insertMap[it] = iterator.next().trim()     29:     }     30:     rowsToInsert += insertMap;     31:     if (numRowsProcessed % numRowsPerBatchInsert == 0) {     32:       completeInsert();     33:     }     34:   }     35: }     36:       37: public void completeInsert() {     38:   jdbcInserter.executeBatch((Map<String, Object>[]) rowsToInsert.toArray());     39:   rowsToInsert.clear();     40: }     41: }

Right off the bat, one nice thing we have in groovy is the removal of the need for getters and setters, automatically making the file smaller. But beside that, lines 26-29 have the concise, yet very readable Groovy closure that takes the column names and creates a map between them and values of the specific line. And this little class gives us what we need with barely any parsing and NO sql.

Batch loading to the database – Part 1 – External Tables

2009-06-08T01:35:00.001-07:00

At our company, we have a large amount of data that we load each day to our database. Some of this data can be a few million lines. Since our database is Oracle (11g), there are two very elegant and efficient ways to get this done. The first is via Oracle sqlloader application, which has many options and configurations to optimize the load. However, we have seen that unless you relax indices on the tables you are loading to, the process can still be quite taxing on the database.

Another quite elegant method I recently learned about (new as of 10g), is the use of external tables. The data sits as a file on the disk of the Oracle database but is accessible using normal SQL. You can then manipulate and transform this data using an “insert select” to put the data in the correct table which is especially nice if the data requires some transformation before it is in it final format. The simplest example of this is using a CSV file as you see in this example. However, it gets better. You can even user a pure XML File, which may even have headers and footers that you choose to ignore (as we had), all easily set in the CREATE TABLE DML.

The following example shows how we did this.

        1: CREATE TABLE titan.external_table1
    (field1                        VARCHAR2(20),
    field2                         VARCHAR2(20),
    field3                         VARCHAR2(20),
    field4                         VARCHAR2(1),
    description                    VARCHAR2(2000))
  ORGANIZATION EXTERNAL (
   DEFAULT DIRECTORY  EXT_TAB_DIR
    ACCESS PARAMETERS(records delimited by "</XMLBEGINTAG>"
badfile EXT_TAB_DIR:'EXCEPTION.bad'
logfile EXT_TAB_DIR:'EXCEPTION.log'
fields
(
filler char(2000) terminated by "<XMLBEGINTAG>",
field1 char(20) enclosed by "<FIELD1>" and "</FIELD1>",
field2 char(20) enclosed by "<FIELD2>" and "</FIELD2>",
field3 char(20) enclosed by "<FIELD3>" AND "</FIELD3>",
field4 char(1) enclosed by "<FIELD4>" AND "</FIELD4>",
description char(2000) enclosed by "<DESCRIPTION>" AND "</DESCRIPTION>"
)
)

The syntax you see on Line 7 is what informs Oracle that this is going to be an external table line 8 indicates the name of the oracle directory (see the link above for more about that). Line 14 tells us that everything before the first tag in the file (an XML header) should be considered filler and ignored. And then we define the begin and end tag of each field.

As i started work on another project (not at work, and thus not using Oracle), i saw that the external table feature is even in HSQLDB (the default Grails database). Using this feature, i was able find a simple way to load an encoded CSV file to my database and have all the text appear correctly in my grails application when displaying the data.

The syntax is straightforward and well documented, and here it is for my example:

set TABLE CC_TRANSACTION SOURCE "src/outFile.txt;fs=|;ignore_first=true;encoding=Windows-1255

The line above indicates where the file is, what the delimiter is (|) and the encoding. This table now appears as a regular table in the schema and be used to join or manipulate data for insert elsewhere.

Not all companies are likely to allow access to the Oracle Server filesystem. So, in the next post I hope to discuss an alternative that would work with any database. It also shows the beauty of the groovy syntax together with the ease of use of Spring.

FtpClient – the advantage of open source code

2009-06-02T01:18:00.001-07:00

Yesterday, it was brought to my attention that one of our apps, whose job is to retrieve files via ftp and process them has not been processing files over the past few days (ok, a bit longer and i should have NOT ignored those emails). A look at the stack trace showed the problem was not in our code (even though we HAD just released new code around the time the problem started, whew!).

After investigation, we found the problem is that our customer seems to have changed their configuration of their ftp server. If you ever manually ftp somewhere, you see a line that usually looks something like this:

230 User anonymous logged in.
Remote system type is UNIX.

Well, unfortunately, when we logged in to this customer, the message was a bit different:

230 User anonymous logged in.
Remote system type is XXXXXXXXXXXX.

As it turns out, this line about the remote system type is very important to our ftp library – Jakarta Commons Net’s FtpClient. A look at the source code based on what the stack trace showed that this string determines the parser to be used and if no matches are found, then it throws a ParserInitializationException. The exact error is:

ParserInitializationException: Unknown parser type:

This seems to be the intent of the designer, even though in my opinion if a regular ftp client can access the site, the FtpClient should be able to as well. A bit of looking around on the net did not help, so we set upon a simple and elegant solution.

We subclassed this class, the DefaultFTPFileEntryParserFactory class and added the following code:

public class MyFTPFileEntryParserFactory extends DefaultFTPFileEntryParserFactory
{
  public FTPFileEntryParser createFileEntryParser(String key)
  {
      if (key.equals("XXXXXXXXXXXX")) {
          key = FTPClientConfig.SYST_UNIX;
      }
      return super.createFileEntryParser(key);
  }
}

Now, the only issue left was to tell the application to use our parser instead. There are two ways to do this. One is to use listFiles and pass the parser, or in our case it was easier when creating the FtpClient (in Spring of course), to add the setter for the ParserFactory to use our class.

This is a great example of the convenience of using Spring here. All that happens is that you add your new class to the class path and update the application Context!

I will add that this is not the first time we have been bitten by this library. Last February 29 we were hit by this issue.

Observer Pattern using Closures

2009-05-20T06:39:00.001-07:00

I have been working on a small Swing application. Its purpose is a batch download of various items that you select in the GUI. I only discovered Griffon after i had already begun and since it is just one screen, i decided to stay away from it for now. So, once i had the UI built (quite painful as it was a first foray into Swing for me, but that is Java's fault and not Groovy. Perhaps more on that in a separate post) and it was time to launch a thread and get updates from the thread to put in the text box in my application. In Java 6 there is a new class, SwingWorker to deal with this in an elegant way (see here for a demo of how to use it). However, with no inner classes in Groovy, this did not seem like it would work. But, in essence the closure has the power of the inner class since it has access to all the members of the class, so this seemed like way to go. And here it is

So, my worker class has the following:

private List closureList = []

public void addObserver(Closure c) {
closureList += c
}

public void removeObserver(Closure c) {
closureList -= c;
}

private void notifyObservers(String fileName, DownloadStatus status) {
for (c in closureList) {
   c.call(itemName status)
}
}

and the GUI class has following closure defined.

def guiUpdater = {String filename, DownloadStatus status ->
swing.edt {
   if (status == DownloadStatus.COMPLETE) {
      resultsTextArea.append("Download of ${filename} completed\n")
   }
}
}

Notice how in my closure i still need to use the swing event dispatch closure for the updates.

Let's get started

2009-05-19T00:50:00.000-07:00

Even though I put a little blurb about the blog on the side, I figured I would still get started with a blog introduction. As a java programmer over the past few years, I have started dabbling with Groovy (and Grails as well, though less due to my lesser focus on GUI) and have looked for any possible reason to write tools in groovy. That is the reason I added it to the name of the blog as I figure at this point in time, my posts will probably gravitate more to the groovy world, so why not use the double meaning in the name.

A colleague of mine has commented that documentation of groovy is lacking. I think the problem is that many of the examples are simplistic, and for us java programmers what we are looking to learn is "what is the groovy way of doing things" and the ideas are not easily expanded to a more complex problem. This is further exacerbated by it being a scripting language, so the code complete will usually not provide the answers. Instead you will be looking around to try to figure out "what are the parameters i get in this closure?" or similar issues.

I find that i actually burn a lot of time on this, so developing in groovy has been slow, as my goal not to see Java code with a .groovy extension. In the end, i am usually quite satisfied with the results and am impressed with the readibility of this more concise code. And of course the next time i need to something it goes MUCH faster and it quite elegant!

So, i figured i would try to give back and help out with some of what i have done, providing the perspective of a relative newcomer to Groovy.

Hope you enjoy.