Tuesday, December 10, 2013

Taking mocking seriously

When I first got involved in using mocks, I viewed them as a necessary evil since I had some dependencies in my class and if don’t use a mock, I will suffer NPE’s in my tests.
However, as I started using them and learning their power, I have seen that I can really use mocking in ways that enhance my unit tests. Here is a concrete example. I am using Spring Data JPA. So, I need to mock the save of my entity when it gets created in the function that I am unit testing. However, once I am mocking I can easily also validate the parameters sent to this function that did the save. Using Mockito, we use the ArgumentCaptor, to do this validation as follows:
MyRepository myRepository = mock(MyRepository.class);
ArgumentCaptor<EntityClass> argument = ArgumentCaptor.forClass(EntityClass.class);
//... invoke function to unit Test that will internally save Entity
verify(myRepository).saveAndFlush(argument.capture());
EntityClass entity = argument.getValue();
assert entity.getName().equals(“name”);
assert entity.getAddress().equals("address");

This now means my unit test has validated that all parameters passed to the function were passed to the database.
Now we see some real added value in the mock, where we can be confident that our copy of the data to the entity we are persisting has not missed any of the parameters.
Another nice feature in mocking is checking how many times a function is invoked. Modifying the the code above:
MyRepository myRepository = mock(MyRepository.class);
ArgumentCaptor<EntityClass> argument = ArgumentCaptor.forClass(EntityClass.class);
//... invoke function to unit Test that will internally save Entity
verify(myRepository,times(1)).saveAndFlush(argument.capture());
EntityClass entity = argument.getValue();
assert entity.getName().equals(name”);
assert entity.getAddress().equals("address");
Happy Mocking!

Thursday, May 26, 2011

Using Date and Calendar – At your own risk!

You can find lots of issues discussed on the ‘net about the problems of the Calendar/Date classes in Java. Here are a few that recently reared their ugly head in our code:

1. This one is actually not in either of these but in trying to output our Date. Beware of the mm instead of MM when using SimpleDateFormat. Yep, one of my team members used the following code:

SimpleDateFormat sdf = new SimpleDateFormat(“yyyymmdd”);

Surprisingly everyone in our tests had a birthday of the first of the month! I think that the right solution would be that mm is wrong and mi is minutes and mo is month. This would solve the confusion.

2. Did you know that there is a difference whether you use Calendar.HOUR or Calendar.HOUR_OF_DAY? Ok, i admit it this one is documented in the javdocs, but i would still think that it is a bit of surprise that in the afternoon, if you do

cal.set(Calendar.HOUR,0);

that you will get 12:00 PM!

3. Why does toString() of Calendar not put out the date in some format?

4. Beware that DAY_OF_WEEK is not zero based while MONTH is.

Thursday, April 28, 2011

Using StAX to extract records from an XML file

When processing large XML files, it is impossible to build a DOM of the whole file as you are sure to run out of memory. Java 6 includes StAX (it was around earlier of courser) and it is the best solution to deal with large XML files to build a scalable XML file processor. However, it still may be desirable to build a Document Model when processing an individual record due to decisions in code that may need to be made based on information lower in the node. Due to this requirement, we use StAX to extract each record from our gargantuan XML files and then pass to a different class that builds a DOM for the processing the record. Here is the very simple code with explanation of what we did.


   1: public class StaxExtracter {



   2:     XMLEventReader reader;



   3:     boolean headerRead = false;



   4:  



   5:     public void openStream(InputStream s) throws XMLStreamException {



   6:         XMLInputFactory inputFactory = XMLInputFactory.newInstance();



   7:         reader = inputFactory.createXMLEventReader(s);



   8:     }



   9:  



  10:     public void close() throws XMLStreamException {



  11:         reader.close();



  12:     }



  13:  



  14:     //function to allow us to read part of a document that does not include the header



  15:     public void setHeaderRead(boolean val) {



  16:         headerRead = val;



  17:     }


The above is some boilerplate code needed for our class. The heavy work is the XMLEventReader, created in the openStream function on line 7.

Following this we have two main functions. The first is used to read the open XML tag of the document and any important attributes in this tag, and the second is to read an actual record. Since the class is written to read recursively all sub elements of a record, it is important that we not read the open XML tag of our document as a regular record, since that will mean we wind up reading our entire document as one record! However, since there may be need to work on part of a file, i allowed an override on line 15 above so we can skip reading the header. The open XML tag is obviously not a complete XML tag so we do some heavier work here, returning the attributes as a map for use by our application, in addition to the raw line which can be used upstream in our application.

Here is the actual code:


   1: /**



   2:      * Since header is not a complete element we need to do special treatment so we will read the header attributes



   3:      * into a hashmap to save all the attributes besides returning the raw data



   4:      *



   5:      * @return the rawData of document start and header



   6:      * @throws XMLStreamException



   7:      */



   8:     public String getHeaderAndAttributesAsMap(Map<String, String> attributes) throws XMLStreamException {



   9:         boolean documentStart = false;



  10:         StringBuffer rawData = new StringBuffer();



  11:         while (reader.hasNext() && !headerRead) {



  12:             XMLEvent e = reader.nextEvent();



  13:             if (e.isStartDocument()) {



  14:                 documentStart = true;



  15:             } else if (e.isStartElement()) {



  16:                 headerRead = true;



  17:                 StartElement startElement = e.asStartElement();



  18:                 for (Iterator i = startElement.getAttributes(); i.hasNext();) {



  19:                     Attribute attr = (Attribute) i.next();



  20:                     attributes.put(attr.getName().getLocalPart(), attr.getValue());



  21:                 }



  22:             }



  23:             //skip whitespace before document



  24:             if (documentStart) {



  25:                 rawData.append(e.toString());



  26:             }



  27:         }



  28:         if (!headerRead)



  29:             return null;



  30:         else {



  31:             return rawData.toString();



  32:         }



  33:     }


Now, on to the main function for reading each element of our XML file:


   1: public enum RecordStatus {BODY,NO_DATA,CLOSE_BODY_TAG,END_DOCUMENT}



   2:  



   3: RecordStatus getNextRecord(StringBuffer recordData) throws XMLStreamException {



   4:     if (!headerRead) {



   5:         throw new RuntimeException("need to read header line before calling this function");



   6:     }



   7:     RecordStatus rc = RecordStatus.NO_DATA;



   8:     String recordName = null;



   9:     boolean finishedReadingElement = false;



  10:     boolean startOfRecord = false;



  11:     while (reader.hasNext() && !finishedReadingElement) {



  12:         XMLEvent e = reader.nextEvent();



  13:         if (e.isStartElement()) {



  14:             if (!startOfRecord) {



  15:                 startOfRecord = true;



  16:                 recordName = e.asStartElement().getName().getLocalPart();



  17:             }



  18:         } else if (e.isEndElement()) {



  19:             EndElement endElement = e.asEndElement();



  20:             if (endElement.getName().getLocalPart().equals(recordName)) {



  21:                 finishedReadingElement = true;



  22:                 rc = RecordStatus.BODY;



  23:             }  else if (!startOfRecord) {



  24:                 //todo  we could make this better by e adding validation here to save the START_BODY_TAG



  25:                 rc = RecordStatus.CLOSE_BODY_TAG;



  26:                 finishedReadingElement = true;



  27:             }



  28:         } else if (e.isEndDocument()) {



  29:             rc = RecordStatus.END_DOCUMENT;



  30:         }



  31:         //skip whitespace before element



  32:         if (startOfRecord || rc == RecordStatus.CLOSE_BODY_TAG) {



  33:             if (e.isCharacters())



  34:                 //need to re-escape characters so SAX parser wont choke on it



  35:                 recordData.append(StringEscapeUtils.escapeXml(e.toString()));



  36:             else



  37:                 recordData.append(e.toString());



  38:         }



  39:     }



  40:     return rc;



  41: }


We use an enum to give us maximal information about what was read while return the actual raw data in a StringBuffer that is allocated outside our function.

Of course, you could do ALL the parsing in your StAX implementation as well, as described here and other places, but this code allows us to continue to use the more elegant DOM for our actual work.

Monday, February 14, 2011

Java in Hollywood -- Using the Actor Model in Java

We have all been hearing how the programming paradigm has started to change, in that Moore’s law will no longer bail us out to speed up our applications and instead we we need to program better to use our multiple processors. Many have been pushing the Actor Model as the simplest way to get to solid multithreading without locks, etc. And for that reason, we are getting pushed to use Scala and other languages instead of Java. But you can also use these frameworks in your existing Java application so i figured i would give it a spin using Akka, unfortunately postponing yet again my first application in Scala.
For a starter, my app just needs to run a job asynchronously with no response, but this would be enough to familiarize myself with Akka and see how to use it.

So, after downloading Akka, the following dependencies are needed to get started:


Now, we can write a small class. Here is the code to send asynchronous messages and not get a response back.


   1: public class MyActor1 extends UntypedActor {



   2:  



   3:     public void onReceive(Object message) {



   4:         Integer numMessage = (Integer) message;



   5:         System.out.println(Thread.currentThread().getName() + "\tincoming message " + numMessage);



   6:         for (int i=0;i<1000000000;i++) {           



   7:         }



   8:         System.out.println(Thread.currentThread().getName() + "\tFinished Message " + numMessage);



   9:         System.out.println(Thread.currentThread().getName() + "\tStopping on thread " + numMessage);



  10:         getContext().stop();



  11:     }



  12:  



  13:     public static void main(String[] args) throws InterruptedException {



  14:  



  15:         for (int i=0;i<50;i++) {



  16:             Actors.actorOf(MyActor1.class).start().sendOneWay(i);



  17:         }



  18:        Thread.sleep(50000);



  19:     }



  20: }


Line 16 shows the code to launch the thread as we “start’ the class and then send the message. In this case we sendOneWay but other methods include sendRequestReply, and sendRequestReplyFuture, as you can see in the documentation.

When you run this application you will see that it launched 16 threads and uses them for the 50 jobs. I found it nice that it was expressive – sendOneWay, as well as to access the context to end the activity of this object.

The next step was learning how to customize the behavior of my thread pool, which led to this change to the application:



   1: public class MyActor extends UntypedActor {



   2:  



   3:     public MyActor(MessageDispatcher dispatcher) {



   4:         getContext().setDispatcher(dispatcher);



   5:  



   6:     }



   7:     public void onReceive(Object message) {



   8:         Integer numMessage = (Integer) message;



   9:         System.out.println(Thread.currentThread().getName() + "\tincoming message " + numMessage);



  10:         for (int i=0;i<1000000000;i++) {           



  11:         }



  12:         System.out.println(Thread.currentThread().getName() + "\tFinished Message " + numMessage);



  13:         System.out.println(Thread.currentThread().getName() + "\tStopping on thread ");



  14:         getContext().stop();



  15:     }



  16:  



  17:     public static void main(String[] args) throws InterruptedException {



  18:        final MessageDispatcher d = Dispatchers.newExecutorBasedEventDrivenDispatcher("dispatcher").withNewThreadPoolWithLinkedBlockingQueueWithCapacity(50)



  19:       .setCorePoolSize(5)



  20:       .setMaxPoolSize(6)



  21:       .setKeepAliveTimeInMillis(60000)



  22:       .setRejectionPolicy(new ThreadPoolExecutor.CallerRunsPolicy())



  23:       .build();



  24:  



  25:         for (int i=0;i<50;i++) {



  26:             ActorRef actor = Actors.actorOf(new UntypedActorFactory() {



  27:                 public UntypedActor create() {



  28:                     return new MyActor(d);



  29:             }



  30:             });



  31:             actor.start().sendOneWay(i);



  32:         }



  33:  



  34:        Thread.sleep(50000);



  35:     }



  36: }



As you see, on line 18 we create the ThreadPool, in Akka terminology called a MessageDispatcher. There are a few different kinds as mentioned here. But this change means that we now need a constructor to receive this dispatcher for use. (Don’t worry Spring integration is not a problem to use DI for the actors).

And voila, we see in the printouts that indeed this is running in multiple threads. Of course, for the need of one way with no reply, there really is no need to use Akka since the code is pretty straightforward using plain old Spring + Java as follows, where we define our ThreadPool bean in the application context:



   1: <bean id="taskExecutor" class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor" destroy-method="destroy">



   2:         <property name="corePoolSize" value="5" />



   3:         <property name="maxPoolSize" value="10" />



   4:         <property name="queueCapacity" value="25" />



   5: </bean>


Once we have this, use of the code to do the same things just involves: taskExecutor.execute(bean). So, in this specific instance, the overhead of adding the scala and akka jars would not be justified, but clearly for more significant multithreaded needs, this could be a powerful tool to simplify code and make it very readable.

Monday, December 27, 2010

Attachments in Axis 1.4

In our "legacy" application, we have a series of classes implemented to connect to a certain 3rd party web service implementation. As this was implemented a number of years ago, it was done using axis 1.4. There are many web service calls, but the differences in their implementation is not that great, as all have core fields that are in common. Thus, the implementation we used was of the template method pattern. The use of this pattern was to encapsulate ALL the connectivity and security issues (client certificates) of using the web services in our base class. Each new web service is a trivial amount of work. All we need to do is to run wsdl2java to generate the classes, inherit our new implementation from the template base class, which then forces us to implement a number of abstract methods that include the URL of the service, as well as functions to return the request and response classes of the web service call. The template method also requests each service to add its unique data specific to that request.

So, even after our knowledgeable web services implementer moved on, this did not impede our ability to add new services, since all that plumbing was untouched code that need not be thought about. All that changed recently when we had a new web service call to add that uses attachments to send XML as an attachment. (This is third party software but I think i understand the logic of why this was done). Suddenly, the work was not as simple as it was before, and we had no axis 1.4 expert anymore and little place to turn anymore, as the world has moved forward past axis 1.4. But with such an elegant implementation it seemed worth trying to make this work within axis 1.4 and after much Googling (and even "Bing"ing), and finally some trial and error of coding, we came to an elegant solution to keep our template method pattern intact and yet still add this functionality. The main (only?) documentation i found about using attachments in axis 1.4 was here

Here is the code added to the template base class:



   1: DataHandler attachment = getAttachment();



   2: if (attachment == null) {



   3:     response = call.invoke(new Object[]{request});



   4: } else {



   5:     AttachmentPart ap= new AttachmentPart(attachment);



   6:     QName qn = new QName(namespace, "DataHandler");



   7:     call.registerTypeMapping(attachment.getClass(), //Add serializer for attachment.



   8:             qn, JAFDataHandlerSerializerFactory.class, JAFDataHandlerDeserializerFactory.class);



   9:     call.addAttachmentPart(ap);



  10:     



  11:     call.addParameter("source", qn, ParameterMode.IN);



  12:     response = call.invoke(new Object[]{request,attachment});



  13: }



The first line calls a new method getAttachment which is not abstract (or final). This means that we can add this functionality without having to implement it in all the implementing classes. The default implementation of the base class returns null and the new class we have will obviously override this and return the attachment of type javax.activation.DataHandler.

If we have an attachment, the steps we need to take are:
  • On Line 5 we wrap this attachment in an Axis class AttachmentPart
  • On Line 7-8, we need to register this attachments using the serializer/serializer JAFDataHandler that will handle the serializing and deserializing of the attachment. Note that the name “DataHandler” is just what i made up.
  • Line 9 adds the attachment to the call
  • Line 11 adds the attachment as a parameter to the call
  • Line 12 includes our request class and the attachment in the actual call.
One other change we needed to make was that since until now we always had only one parameter, we did not need to register the first parameter, but now this became necessary in order to add the second.

And in these few lines since our use of the template method pattern enabled us to keep our complex implementation intact and still accomodate this new functionality.

Wednesday, August 25, 2010

Beware of Static Initializers (even in 3rd party code!)

As i was just releasing a small patch to our application that uses POI, i noticed a bunch of errors at startup. Of course, I immediately thought i changed more than I thought and began combing thru the code to see why i was getting this error. The stack trace was pointing to an error in the code of POI, which had a index out of range on something that clearly was not related to anything i had changed in our code in a LONG time. But why was this error happening and why now?

After searching the web and coming up empty on this error, I began looking more carefully and realized that the data structure at hand is one that is statically initialized on demand. What that means is that it is one of those static functions that when called, checks to see if the given data structure is built and if not, builds it then. Well, that’s nice but in our multithreaded application, we had two threads simultaneously trying to build an Excel spreadsheet so one of them got thrown out due to this static initialization that was probably in progress.

The code looks as follows:

public static List getBuiltinFormats()
    {
        if ( builtinFormats == null )
        {
            populateBuiltinFormats();
        }
        return builtinFormats;
    }

So we have a static function using the builtInFormats. The actual function populateBuiltinFormats is synchronized but that just means that it cannot be invoked simultaneously, but if is partially populated then the second thread will think it is complete!

Since this is third party code, the simplest solution is at startup of my main to force the initialization of this static data structure, so i invoke HSSFDataFormat.getBuiltinFormats(); and voila, the issue has been solved.

As a Spring evangelist, I would point out that if you use Spring for functionality like this, and then in an init function that Spring uses, you will have the data populated. You could also just have it statically created and not conditionally. Otherwise, you would need to build the internal data structure to a temporary variable and only assign it when complete. Of course, that will mean a second check in the function populateBuiltinFormats to see that if by the time you got in, the data is already built, or maybe only allow this function to run once.

Tuesday, August 17, 2010

Tips For Maintainable Code

Due to a need of late to maintain a lot of code (where of course not all of it has been written by me), I have been able to give a lot of thought to which tips I have read about in the past that strike me as the most important to have maintainable code. In many scenarios, companies simply rewrite their applications every few years and thus this issue may not be relevant but in my case where the lack of manpower makes this impossible, (and the application works so well there is no business justification for it), it has led me to think carefully how to set up a project and practices to have in place so that an application is maintainable and customizable in a reasonable way. Much of what i will be listing is based on the book Clean Code which I already blogged about previously here.

1. Keep functions short and simple – and if you need to add in more functionality, refactor it into a new function that clearly defines in its name what it does. In the book Clean Code, the insistence is that all functions do exactly one thing.

2. Building on number 1, use small functions as a way to comment “funny” things you need to do. Sometimes, we have small things we needed to do – set some parameter to some funny value or the like and the person who makes this change may or may not comment this, since at the time it was so obvious and it may not even make the comment for the check in, since many files were checked in simultaneously, but if you get in the habit of creating a small function, such “nullOutParameterSoThatRequestHandlesCorrectly”, this comment is forever part of the code and will not get lost. It also means that when someone comes along later and sees it he can be sure it is there on purpose and not something wonder – it seems to me that removing this line is ok but i am not sure…

3. Comprehensive Unit Tests. This is a great way to document your code since we can see the kind of input you expect to receive for a given function and how you expect to handle it. As has been written about extensively on the ‘net, when we see these unit tests, we have greater confidence to make changes since we have a baseline to evaluate our changes. One other advantage of the unit tests, is that if you added a new class with dependencies, and you are using Dependency Injection, like Spring, even if the code never made it to production we have a working example to build off for later use. This actually happened to me, where code was written and not released, but lacking an example of how to set up this class in my Spring application context led to my lack of confidence in using this class.

4. Regression Tests. While Unit Tests are imperative, So are regression Tests. These are automated end to end tests so that we have a baseline of how the application works, in dailsychaining multiple functions together, what data was passed from beginning to end and what was received. I have used TestNG to make up regression tests due to the amazing features of paramterized testing, enabling to set up use cases that run multiple times with different inputs. (I hope to write more about this in an upcoming blog)

5. use the Source Control and DONT save pruned code. Remove it. Don’t worry about it since you can get it from the source control. It just creates confusion.

Remember, that YOU may be the one left to maintain the code so its in your best interest to get it right when developing it…