Monday, December 27, 2010

Attachments in Axis 1.4

In our "legacy" application, we have a series of classes implemented to connect to a certain 3rd party web service implementation. As this was implemented a number of years ago, it was done using axis 1.4. There are many web service calls, but the differences in their implementation is not that great, as all have core fields that are in common. Thus, the implementation we used was of the template method pattern. The use of this pattern was to encapsulate ALL the connectivity and security issues (client certificates) of using the web services in our base class. Each new web service is a trivial amount of work. All we need to do is to run wsdl2java to generate the classes, inherit our new implementation from the template base class, which then forces us to implement a number of abstract methods that include the URL of the service, as well as functions to return the request and response classes of the web service call. The template method also requests each service to add its unique data specific to that request.

So, even after our knowledgeable web services implementer moved on, this did not impede our ability to add new services, since all that plumbing was untouched code that need not be thought about. All that changed recently when we had a new web service call to add that uses attachments to send XML as an attachment. (This is third party software but I think i understand the logic of why this was done). Suddenly, the work was not as simple as it was before, and we had no axis 1.4 expert anymore and little place to turn anymore, as the world has moved forward past axis 1.4. But with such an elegant implementation it seemed worth trying to make this work within axis 1.4 and after much Googling (and even "Bing"ing), and finally some trial and error of coding, we came to an elegant solution to keep our template method pattern intact and yet still add this functionality. The main (only?) documentation i found about using attachments in axis 1.4 was here

Here is the code added to the template base class:



   1: DataHandler attachment = getAttachment();



   2: if (attachment == null) {



   3:     response = call.invoke(new Object[]{request});



   4: } else {



   5:     AttachmentPart ap= new AttachmentPart(attachment);



   6:     QName qn = new QName(namespace, "DataHandler");



   7:     call.registerTypeMapping(attachment.getClass(), //Add serializer for attachment.



   8:             qn, JAFDataHandlerSerializerFactory.class, JAFDataHandlerDeserializerFactory.class);



   9:     call.addAttachmentPart(ap);



  10:     



  11:     call.addParameter("source", qn, ParameterMode.IN);



  12:     response = call.invoke(new Object[]{request,attachment});



  13: }



The first line calls a new method getAttachment which is not abstract (or final). This means that we can add this functionality without having to implement it in all the implementing classes. The default implementation of the base class returns null and the new class we have will obviously override this and return the attachment of type javax.activation.DataHandler.

If we have an attachment, the steps we need to take are:
  • On Line 5 we wrap this attachment in an Axis class AttachmentPart
  • On Line 7-8, we need to register this attachments using the serializer/serializer JAFDataHandler that will handle the serializing and deserializing of the attachment. Note that the name “DataHandler” is just what i made up.
  • Line 9 adds the attachment to the call
  • Line 11 adds the attachment as a parameter to the call
  • Line 12 includes our request class and the attachment in the actual call.
One other change we needed to make was that since until now we always had only one parameter, we did not need to register the first parameter, but now this became necessary in order to add the second.

And in these few lines since our use of the template method pattern enabled us to keep our complex implementation intact and still accomodate this new functionality.

Wednesday, August 25, 2010

Beware of Static Initializers (even in 3rd party code!)

As i was just releasing a small patch to our application that uses POI, i noticed a bunch of errors at startup. Of course, I immediately thought i changed more than I thought and began combing thru the code to see why i was getting this error. The stack trace was pointing to an error in the code of POI, which had a index out of range on something that clearly was not related to anything i had changed in our code in a LONG time. But why was this error happening and why now?

After searching the web and coming up empty on this error, I began looking more carefully and realized that the data structure at hand is one that is statically initialized on demand. What that means is that it is one of those static functions that when called, checks to see if the given data structure is built and if not, builds it then. Well, that’s nice but in our multithreaded application, we had two threads simultaneously trying to build an Excel spreadsheet so one of them got thrown out due to this static initialization that was probably in progress.

The code looks as follows:

public static List getBuiltinFormats()
    {
        if ( builtinFormats == null )
        {
            populateBuiltinFormats();
        }
        return builtinFormats;
    }

So we have a static function using the builtInFormats. The actual function populateBuiltinFormats is synchronized but that just means that it cannot be invoked simultaneously, but if is partially populated then the second thread will think it is complete!

Since this is third party code, the simplest solution is at startup of my main to force the initialization of this static data structure, so i invoke HSSFDataFormat.getBuiltinFormats(); and voila, the issue has been solved.

As a Spring evangelist, I would point out that if you use Spring for functionality like this, and then in an init function that Spring uses, you will have the data populated. You could also just have it statically created and not conditionally. Otherwise, you would need to build the internal data structure to a temporary variable and only assign it when complete. Of course, that will mean a second check in the function populateBuiltinFormats to see that if by the time you got in, the data is already built, or maybe only allow this function to run once.

Tuesday, August 17, 2010

Tips For Maintainable Code

Due to a need of late to maintain a lot of code (where of course not all of it has been written by me), I have been able to give a lot of thought to which tips I have read about in the past that strike me as the most important to have maintainable code. In many scenarios, companies simply rewrite their applications every few years and thus this issue may not be relevant but in my case where the lack of manpower makes this impossible, (and the application works so well there is no business justification for it), it has led me to think carefully how to set up a project and practices to have in place so that an application is maintainable and customizable in a reasonable way. Much of what i will be listing is based on the book Clean Code which I already blogged about previously here.

1. Keep functions short and simple – and if you need to add in more functionality, refactor it into a new function that clearly defines in its name what it does. In the book Clean Code, the insistence is that all functions do exactly one thing.

2. Building on number 1, use small functions as a way to comment “funny” things you need to do. Sometimes, we have small things we needed to do – set some parameter to some funny value or the like and the person who makes this change may or may not comment this, since at the time it was so obvious and it may not even make the comment for the check in, since many files were checked in simultaneously, but if you get in the habit of creating a small function, such “nullOutParameterSoThatRequestHandlesCorrectly”, this comment is forever part of the code and will not get lost. It also means that when someone comes along later and sees it he can be sure it is there on purpose and not something wonder – it seems to me that removing this line is ok but i am not sure…

3. Comprehensive Unit Tests. This is a great way to document your code since we can see the kind of input you expect to receive for a given function and how you expect to handle it. As has been written about extensively on the ‘net, when we see these unit tests, we have greater confidence to make changes since we have a baseline to evaluate our changes. One other advantage of the unit tests, is that if you added a new class with dependencies, and you are using Dependency Injection, like Spring, even if the code never made it to production we have a working example to build off for later use. This actually happened to me, where code was written and not released, but lacking an example of how to set up this class in my Spring application context led to my lack of confidence in using this class.

4. Regression Tests. While Unit Tests are imperative, So are regression Tests. These are automated end to end tests so that we have a baseline of how the application works, in dailsychaining multiple functions together, what data was passed from beginning to end and what was received. I have used TestNG to make up regression tests due to the amazing features of paramterized testing, enabling to set up use cases that run multiple times with different inputs. (I hope to write more about this in an upcoming blog)

5. use the Source Control and DONT save pruned code. Remove it. Don’t worry about it since you can get it from the source control. It just creates confusion.

Remember, that YOU may be the one left to maintain the code so its in your best interest to get it right when developing it…

Monday, June 7, 2010

Testable Code Vs. OO code

Can we ever have a situation where the desire to have testable code is not 100% compatible with our Object Oriented Design? As i was working on some new code, i came across this issue and was happy to find that this is not a new issue as we see here. Even if you are not willing to accept the entire thesis, at the very least, we may want to use the thought of testing our code to drive some of our design decisions, since a robust, easy to run test suite is so important in software development, especially as projects grow in size and length and we can hardly remember what our code does anymore. An example of this in my project was my decision to implement the template method pattern where the template methods implemented by the child classes provide data back the parent class rather than invoke helper methods and provide greater functionality. This means that it is easier to test each new implementation and is less time consuming.

An additional aspect of this issue can be seen based on the book Clean Code, discussed in a previous post. While the focus there was on readability of the code, testability seems to overlap quite a bit with readability. The simpler we make our classes and our functions, the simpler for testing and finding the problems when test fail. Another aspect discussed at length there is the issue of parameters to function (or better yet, not having them). It is recommended instead to have members the contain the necessary data rather than pass it from function to function. (Beware, as this means your classes will not be thread-safe and will require a new instance for each use). From what i recall (not having access to this book right now), the main issue was readability, but here too, another fringe benefit is testability, especially if you are using TestNG (or JUnit 4). The reason for this, you create this class and populate the member data correctly and then can run tests repeatedly with little typing other than the method names. Writing and running the tests is now easier since you need not figure out all the parameters of a given function. Also, running multiple tests with different data (using TestNG DataProvider) is easier to provide.

There are two other ways that testability impacts design in my current project:
The first is in using interfaces even when i have no plans to implement another implementation of a member class. It is much easier to mock behavior and isolate a test if we use interfaces and then plug in a simpler implementation of our interface for test purposes. Of course, use of Spring here and the Spring application context makes this trivial

The second aspect is in determining the scope of functions in order to enable testing of private functions as discussed here. I agree wholeheartedly with Bob Lee in his post that your code will be better if private functions are tested individually and you do not leave all your testing to the public API.

I realize that much of what i write here is well known but i see that many of my peers are still not recognizing how important repeatable tests are. It is hard to see down the line when you are creating a new project from scratch. But, there are two ultra important reasons that we need to have a repeatable test suite. The first is documentation. As written about in Clean Code, Javadocs are not reliable. It happens too often in projects that programmers do not make sure to update javadocs as things change, and this is true for comments as well. The best documentation in addition to well written code with good naming (as discussed in my previous post about the book Clean Code), is the tests of the API, that show exactly how a function was meant to be used by the original author. It enables us to also what results it was meant to return and all error cases. The more tests on a function the more information provided. The second reason is maintainability. As our software has gotten larger/bloated, while staff has been pared down, a rewrite of our software is not truly possible. The confidence to make changes stems from good, repeatable tests.

A second part of the confidence to make changes in a larger project comes from a good regression suite, which i hope to discuss in an upcoming post.


Wednesday, April 14, 2010

Using a non default Calendar in the Google API

All the examples in the Google Calendar API documentation involve using a default calendar. But i was looking at a project where i would have a shared calendar among users and this calendar would not be the default one. The simple code below demonstrates how to do this in Java. There are three parameter you will need to inject into this class (in my case via Spring) -- the username, password, and calendarname. Using these three you have what you need to work with the non default calendar.


   1: public class CalendarUtil {



   2:   private final static Logger logger = Logger.getLogger(CalendarUtil.class.getName());



   3:   private String username;



   4:   private String password;



   5:   String calendarName;



   6:   private URL calendarUrl;



   7:  



   8:   private CalendarService service;



   9:  



  10:   // The base URL for a user's calendar metafeed (needs a username appended).



  11:   private static final String METAFEED_URL_BASE =



  12:           "http://www.google.com/calendar/feeds/";



  13:  



  14:   private static final String CALENDAR_URL_SUFFIX = "/private/full";



  15:  



  16:   public void init() {



  17:     service = new CalendarService("DR-Appt-App-1");



  18:     try {



  19:       service.setUserCredentials(username, password);



  20:     } catch (AuthenticationException e) {



  21:       throw new RuntimeException("Could not authenticate to Google Calendar with username: " + username, e);



  22:     }



  23:     String calendarId = null;



  24:     try {



  25:       calendarId = getCalendarId();



  26:     } catch (IOException e) {



  27:       throw new RuntimeException("Error caught trying to connect to calendar: ", e);



  28:     } catch (ServiceException e) {



  29:       throw new RuntimeException("Error caught trying to connect to calendar: ", e);



  30:     }



  31:     try {



  32:       calendarUrl = new URL(METAFEED_URL_BASE + calendarId + CALENDAR_URL_SUFFIX);



  33:     } catch (MalformedURLException e) {



  34:       throw new RuntimeException("Bad URL: " + calendarUrl.toString(), e);



  35:     }



  36:     if (logger.isDebugEnabled()) {



  37:       logger.debug("Url is " + calendarUrl.toString());



  38:     }



  39:   }



  40:  



  41:  



  42:   //get id of correct calendar



  43:   private String getCalendarId() throws IOException, ServiceException {



  44:     String calendarId = null;



  45:     URL metafeedUrl = new URL(METAFEED_URL_BASE + username);



  46:     if (logger.isDebugEnabled()) {



  47:       logger.debug("Url is " + metafeedUrl.toString());



  48:     }



  49:     CalendarFeed resultFeed = service.getFeed(metafeedUrl, CalendarFeed.class);



  50:     for (int i = 0; i < resultFeed.getEntries().size(); i++) {



  51:       CalendarEntry entry = resultFeed.getEntries().get(i);



  52:       if (calendarName.equalsIgnoreCase(entry.getTitle().getPlainText())) {



  53:         calendarId = new File(entry.getId()).getName();



  54:         break;



  55:       }



  56:     }



  57:     return calendarId;



  58:   }



  59:  


I have left out of the above the getters and setters to make it more concise.
Using the above class, you now can access the non primary calendar simply using the syntax:
calendarUtil.getService().insert(…)
or any other service function.



Monday, February 22, 2010

Not a Maven yet….

Notwithstanding my post about using Gradle as a build tool, when i needed to start a new project i decided to use maven due to its amazing integration with an IDE (in my case Intellij), as well as its unbiquitousness.

So, as i have written previously, it is really nice to use Maven as you get started, since with no writing of any config files you have the build process working for you. You have a clean, compile, jar, etc with no work and no opening of an XML file. But that is all theoretical. As i attempted to get going with my project i encountered two really serious issues that took a while to figure out, as well as one issue that needs better resolution.

I will start with the more minor issue. When we use standard libraries that we need to add to our project, then things are pretty straightforward, in fact even better than not using maven. All you need to do is go to http://mvnrepository.com and find the item you want and then it gives you the lines to add to your pom, and voila, you have added your dependency to your project. Of course, in one case, this was not so simple. I wanted to use log4j, so i said, ok, I will just use the latest version. That turned out to be a mistake since all of sudden the compile was failing with libraries i had never heard of unable to be downloaded and installed. As it turns out, after a bit of googling, i learned that i needed to stay away from this latest version. But that is not even the issue i was referring to above. The issue i ran into was when i needed to use non standard libraries that are not in the mvnrepository site. This became difficult as i did not have an easy time figuring out what repository to use and what to add to my POM. In my case, i wanted to use Spring 3 and as far as i can tell, there is no browser for the Spring 3 release repository and thus no instructions for what my artifact ids would look like. (Hint to SpringSource – mention in the javadocs which jar is needed for a package). I wound up guessing based on one item i found somewhere and since i had downloaded the spring 3 distribution i was able to guess how the others would look and solved that issue. The second set of non standard artifacts were the google calendar ones. Here, i had to manually install them in my repository to get it going. Since it was a lot of jars, it was tedious, thought here is a nice link to make it easier, just be sure to use version 2 and not 1 for those things are now in version 2.

Now on the two major issues that i encountered. The first one was that my project would not compile successfully even though there was very little code in it other than use of the Google Calendar API. Finally after not getting anywhere for a while, i realized the issue must be Maven since it did compile in my IDE. As it turned, it was a documented Jira issue, but as the guys in the comments wrote there, who would have thought that default of maven today would not be java 5 at least?

The second one related to testing. Even though my most recent work has been using TestNG, i thought maybe i would give JUnit a try. When my JUnit tests were not being recognized, i figured out that this is because, once again, the default is junit 1.3. After a bit of searching i found what i need to do to add Junit4 to be the test runner. But, since i found the documentation of JUnit 4 lacking, i had decided to switch back but figured i would leave both dependencies in the POM. Well, that turned out to be a mistake too, as i eventually found here

But with these problems behind us, we trudge forward assuming that all should now be smooth sailing, until we need some other complex library….

Wednesday, January 27, 2010

Asynchronous Groovy

In my previous post, i showed how simple asynchronous code in Spring 3 is. I wanted to see what it takes to write the same code in Groovy. Well, it is quite straightforward, using the GPARS library. The docs there show it to be quite powerful, but to be loyal to our simple example, here is the code:



   1: Asynchronizer.withAsynchronizer(4) {ExecutorService service ->



   2:       (0..9).each { num->



   3:         service.submit({



   4:         Thread.sleep(2000);



   5:         println("Running for ${num} thread ${Thread.currentThread()}")}as Runnable)



   6:       }



   7: }



See how you specify the number of threads in the parameter to the closure withAsynchronizer.
There are lots more examples at the GPARS site.

Wednesday, January 13, 2010

Asynchronous Code with Spring 3 – Simple as an annotation

After reviewing some of the features of Spring 3, i decided to test out the asychronous features via annotation. And below is all the code i needed in order to have a function executed asynchronously in my application:



   1: public class Asyncher {



   2:     @Async



   3:     public void myMethod(int i) {



   4:         try {



   5:             Thread.sleep(2000);



   6:         } catch (InterruptedException e) {



   7:             e.printStackTrace();



   8:         }



   9:         System.out.println("Running for " + i);



  10:     }



  11:  



  12:     public static void main(String[] args) throws InterruptedException {



  13:         ApplicationContext ctx = new ClassPathXmlApplicationContext("appContext.xml");



  14:         Asyncher a = ctx.getBean("asyncher", Asyncher.class);



  15:         for (int i=0; i<10;i++) {



  16:                 a.myMethod(i);



  17:         }      



  18:  



  19:         ThreadPoolTaskExecutor t = ctx.getBean("myExecutor",ThreadPoolTaskExecutor.class);



  20:         while (t.getActiveCount() > 0) {



  21:             System.out.println("Number of tasks left: " + t.getActiveCount());



  22:             Thread.sleep(2000);



  23:         }



  24:         t.shutdown();



  25:     }



  26: }



Hard to believe, but all i needed to do was add the Async annotation on line 2 to my function and it works. of course we need a few lines in our application Context to make this work, but here they are:


   1: <?xml version="1.0" encoding="UTF-8"?>



   2: <beans xmlns="http://www.springframework.org/schema/beans"



   3:        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:task="http://www.springframework.org/schema/task"



   4:        xmlns:context="http://www.springframework.org/schema/context"



   5:        xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/task http://www.springframework.org/schema/task/spring-task-3.0.xsd http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd">



   6:  



   7:     <task:annotation-driven executor="myExecutor" />



   8:     <task:executor id="myExecutor" pool-size="2"/>



   9:     <bean id="asyncher" class="Asyncher"/>



  10:  



  11: </beans>


This same method can be used to create scheduled jobs as well, by changing or adding on line 7 the scheduler attribute and then adding in a scheduler bean

<task:scheduler id="myScheduler" pool-size="10"/>

You can read about it in the docs here