Java Quickies – What can go wrong during an application server update?

This little article will show you what can go wrong during an application server update. Most of the time, the update goes smoothly. I mean, have you ever had a trouble upgrading from tomcat 5 to tomcat 6, then 6 to 7? As far as I am concerned, it usually works fine. Thus, I have never really wondered what kind of problems could occur. We know they are adding new features, improving things, and a little something that may prevent your webapp from deploying!

Java EE certified servers

You know that application server implements the Java EE specifications. But what does it really mean? The java community is issuing JSR, which are some kind of RFC in the java world. These JSR defines, amongst others, the API we can find in servlet-api.jar and jsp-api.jar. You know, all the jars you need to code your webapp and that you set as provided in your pom.xml. If you are new to the Java EE world, that the word JSR does not mean anything to you, I would recommend you the Java EE introduction article.

Well let’s get back to the original object of this article, Java EE defines API and application servers implements the API, just like you would implement a java interface. For instance, the JSR 154 defines the servlet 2.4 specifications whereas the JSR 315 defines the servlet 3.0 specifications. Here is an interesting page, it shows what tomcat version support what java version and the API versions.

Now that you know, almost, everything about application server, it is time to see how these new features can mess up with your application. Most of the time, they will not… Except if you did some dark low level programming! Well let me show you what happened when I tried to upgrade from the resin 2.0 server to resin 4.0. Here is the stacktrace I was getting :

java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at xxxx.executeMethod(xxxx)
	at xxxx.forward(xxxx)
	at xxxx.start(xxxx)
	at xxxx.doPost(xxxx)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:159)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:97)
	at com.caucho.server.dispatch.ServletFilterChain.doFilter(ServletFilterChain.java:109)
	at com.caucho.server.webapp.WebAppFilterChain.doFilter(WebAppFilterChain.java:156)
	at com.caucho.server.webapp.AccessLogFilterChain.doFilter(AccessLogFilterChain.java:95)
	at com.caucho.server.dispatch.ServletInvocation.service(ServletInvocation.java:289)
	at com.caucho.server.http.HttpRequest.handleRequest(HttpRequest.java:838)
	at com.caucho.network.listen.TcpSocketLink.dispatchRequest(TcpSocketLink.java:1346)
	at com.caucho.network.listen.TcpSocketLink.handleRequest(TcpSocketLink.java:1302)
	at com.caucho.network.listen.TcpSocketLink.handleRequestsImpl(TcpSocketLink.java:1286)
	at com.caucho.network.listen.TcpSocketLink.handleRequests(TcpSocketLink.java:1194)
	at com.caucho.network.listen.TcpSocketLink.handleAcceptTaskImpl(TcpSocketLink.java:993)
	at com.caucho.network.listen.ConnectionTask.runThread(ConnectionTask.java:117)
	at com.caucho.network.listen.ConnectionTask.run(ConnectionTask.java:93)
	at com.caucho.network.listen.SocketLinkThreadLauncher.handleTasks(SocketLinkThreadLauncher.java:169)
	at com.caucho.network.listen.TcpSocketAcceptThread.run(TcpSocketAcceptThread.java:61)
	at com.caucho.env.thread2.ResinThread2.runTasks(ResinThread2.java:173)
	at com.caucho.env.thread2.ResinThread2.run(ResinThread2.java:118)
Caused by: java.lang.AbstractMethodError
	at com.caucho.server.webapp.DispatchFilterChain.doFilter(DispatchFilterChain.java:111)
	at com.caucho.server.dispatch.ServletInvocation.service(ServletInvocation.java:289)
	at com.caucho.server.webapp.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:298)
	at com.caucho.server.webapp.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:116)
	... 26 more

The AbstractMethodError error happens when you try to call an abstract method. Notice that it is an Error and not an Exception. Indeed, this Error should not happen, because the compiler will not let you compile if you did not implement every abstract methods. Then, why did I get this error?

I used a decompiler to this what was called in the DispatchFilterChain on line 111. Here is the result :

      ServletContext webApp;
      if ((request instanceof AbstractCauchoRequest))
      {
        AbstractCauchoRequest cReq = (AbstractCauchoRequest)request;
        ServletContext webApp;
        if (cReq.getRequest() != null) {
          webApp = cReq.getRequest().getServletContext();
        } else {
          webApp = cReq.getServletContext();
        }
      }
      else
      {
        webApp = request.getServletContext();
      }

The error is thrown when getServletContext is called on the request. This seems pretty weird to me, because Requests and Responses are directly provided by the application server in the doGet and doPost methods for example.

I tried again and made a direct call to getServletContext. Now I got the exact same error but with the name of the failing method:

Caused by: java.lang.AbstractMethodError: xxx.MultipartHttpServlet.getServletContext()Ljavax/servlet/ServletContext;

Well it would have been more help if I had gotten it earlier but… still was a pleasure to see it. Now, I went to the javadoc to see exactly what was the issue with this method. Here, we can see that the method was added by the servlet 3.0 specifications. What happened is, the webapp I was trying to deploy used an old framework that would use its own requests. Bad luck is that this framework used the servlet 2.5 specifications. Resin was expecting a servlet 3.0 Request and thus call a servlet 3.0 method, causing the error because there really is no implementation of this method…

To fix this, I had to decompile the framework Request class and make it implements the new interface defined by the servlet 3.0 specifications.

I thought that now that I fixed this error, everything was going to be alright. But I was wrong… Now my beans validation was not working anymore! Here is the stacktrade:

java.lang.TypeNotPresentException: Type org.joda.time.ReadableInstant not present
Caused by: java.lang.ClassNotFoundException: org.joda.time.ReadableInstant
	at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:270)
	at sun.reflect.generics.factory.CoreReflectionFactory.makeNamedType(CoreReflectionFactory.java:114)
	... 86 more

This Exception was happening during the validate call of Hibernate Validator. Exactly during the evaluation of the @Past annotation. Well, indeed, this project did not use joda time, even if it should have, but it should not be forced to use it.

Another Error that I was getting :

java.lang.NoSuchMethodError: javax.validation.ConstraintValidatorContext$ConstraintViolationBuilder.addPropertyNode(Ljava/lang/String;)Ljavax/validation/ConstraintValidatorContext$ConstraintViolationBuilder$NodeBuilderCustomizableContext;

This second error led me to the solution. Indeed, the addPropertyNode method has been added in the validation-api version 1.1.0. Now you know the drill, somewhere, someone injected an older version of the API. I checked all my pom.xml, double checked the libs embedded in my war. Everything was fine. So I checked our application server resin.
It was the real culprit, for some reasons in their version 4, resin has two jars embedded in its lib directory: validation-api-1.0.0.GA.jar and hibernate-validator-4.3.0.Final.jar. My project had validation-api-1.1.0.Final.jar and hibernate-validator-5.0.3.Final.jar.

Unfortunately the resin jars where loaded instead of mine. I do not want to turn this article into a “how does a classloader work” article, but I will explain quickly what you need to know to understand the problem. Java uses a classloader hierarchy. The parent of the hierarchy is called the “bootstrap classloader”, it is responsible to load the java.lang package. The child of the bootstrap classloader is the “extension classloader”, it will load the jars located in the java.ext.dirs directories. On a windows default installation, here is what the java.ext.dirs system property contains:

    String extDirs = System.getProperty("java.ext.dirs");
    System.out.println(extDirs);
    // C:Program FilesJavajdk1.7.0_51jrelibext;C:windowsSunJavalibext

Finally comes the “system classpath classloader”, this is the classpath you can define using the java -cp command. This classpath has the extension classloader as parent. The basic behavior of a classloader is to ask first its parent to load the class. Every classloaders will delegate to its parent, thus if no parent is able to load the class, it will look in its own classpath if the class is present. This behavior is perfect in a Java SE environment but it is usually not used as it is in the Java EE world. If you want to know more, you can read here how tomcat handles the mechanism.

Long story short, resin uses the default Java SE behavior, hence the wrong jars loaded when the class javax.validation.Validator was needed.

I fixed this by simply removing the two jars from resin. I do not know exactly why these jars were present in the lib directory, but the server seems to be working fine without them 😉

Leave a Comment