September, 2008

Sep 08

YourKit profiler

Profiling is probably one of the most joyous activity for me these days. Besides the fact that it feels good that you're actually got to the point that you have something to profile, you now get to see how the the software that you thought was so well designed is performing. You basically get to look in the guts of your software, forgetting all the abstractions that were provided to you and all the ones you've layered on top and actually see the how things collaborate to execute the instructions. The best part, you actually get to see what goes on in all the libraries that you thought were so great (or maybe not so great after your look under the hood). See this JIRA bug, which was a result of profiling.

Well, I've tried a few profilers for java and one that really stands out is YourKit. Although most profilers have pretty much the same capabilities, one thing that separates them is how intuitive and well polished the interface is. YourKit stands out in this category.

This weekend, I had to profile a few modules that weren't performing well. Although I have a free license of YourKit that we use on the Alex Build project, more on that later, I decided to download the EAP of version 8 trial to use it for a project not related to Alex Build. I'll most likely purchase a personal copy of YourKit once this trial runs out, as there are many projects I'd like to use YourKit for and for the very reasonable price of $499 you can't go wrong.

So I was profiling a TCP/IP application built on Apache MINA which had a thread model defined using JSE 5 executors. Outside of MINA, the application was scaling far beyond what it's throughput was using MINA. MINA being a very lightweight layer over Java's NIO should not bug down the performance and scalability this much. I saw a exponential performance decrease as throughput requirements increased. So I fired up YourKit and started pushing data into the application. At some point, I took a snapshot and then dove in.

Once you have a snapshot, you pretty much have all the details about the particular application runtime at the particular point in time you took it. I noticed that although the thread model was configured to have a fixed thread pool of 20 threads, only one worker thread was being used to process the payloads. This is very easy to see using the "Call stack (view by thread)" interface. This revealed a bigger problem with MINA 1.3, which basically serialized operations, although they claimed to support a thread model, what ever that means :-). MINA 2.0 change that support to support true concurrent IO Handlers and though I migrated to the new API. The next profile revealed that all worked as expected and the thread pool was utilized as expected.


The callstack also revealed that 99% of the time was spent in IO, which is not something that we can tune any further at this point, therefore we're probably getting the best throughput we're going to get.


Netbeans has a very decent profiler as well, but fortunately I used IntelliJ and don't plan on switching to netbeans if I don't have to, even if it's just for profiling. YourKit provides plugins for most IDEs, including IntelliJ and Eclipse.

One other thing I wanted to point out, unlike other commercial profilers, YourKit offers free Open Source licenses to qualified projects. They were very quick and courteous in the process. Thanks YourKit, you make the profiling process very enjoyable.

Sep 08

New Java Build System

So I was intrigued by Joe Ottinger’s call for requirements for a new java build system. I’ve had these thoughts for years now, especially after gruesome experiences with ant and maven.

I must admit, I like maven2. But I’ve experienced a rather steep learning curve using it. I now use it on all my projects. It definitely makes life easier, especially getting started. But I still find it rather difficult to at times find plugins or alternative ways of accomplishing something outside of maven’t conventions. Mostly I’ve adapter to maven’s conventions, but there are times when we just can’t escape the rest of the world. I was using GigaSpaces for a project and because of the OSGI-based build/deployment system, found it rather hard building and deploying the necessary artifacts from the standard maven build structure. After days of battling the obvious, I settled on a hybrid (maven/ant) approach. I build everything in maven, then package jars into GigaSpaces deployable units using ant. I also use ant for the rather custom deployment of PUs.

How much easier would that be had maven had straightforward scripting plugins. Just plugin a script tag into any lifecycle event as an AOP advice, for lifecycle before/after pointcuts.

Everything in maven requires a module. It’s pretty simple to incorporate and build a module, but you then have to maintain it outside of your build file and an average Joe (not Ottinger of course, he’s no average Joe:-), will most likely give up or find some rather dense and superfluous way of accomplishing a simple task.

I think an answer to this is a simple, extensible, DSL-based build system that would incorporate the best of both worlds. Convention and flexibility. We don’t have to sacrifice either one to accomplish what we want. Maven gives us convention, ant gives us great flexibility, Alex will give us both.

Anyone interested, the project is just starting up. You can get more details here…