So, now it’s time to dig a bit deeper into the entire ArrayList versus Vector thing. What if I build each of them up with integers from 1 to x before the test runs, and then as the test call contains() on each integer 1 to x? Is contains faster or slower for ArrayList?

So, it looks like the calls to contains() are slower for ArrayList. What about the calls to add() with a presized Collection?

Yet again, Vector is a little bit faster than ArrayList! I’ve looked through the Java source code and so help me I swear the y are doing the same effective work – but Vector is synchronized and SHOULD BE SLOWER!

So, now I need to branch out and do some testing on other JDK’s and other OS’s. 1.5 and Linux, here I come!

No autoboxing ( ints only )
[source:java]
public void runInternalTrial() throws Exception {
int i = getRandomWithoutAutobox();
}

public int getRandomWithoutAutobox() {
return rnd.nextInt(1000);
}
[/source]
Autoboxing
[source:java]
public void runInternalTrial() throws Exception {
int i = getRandom();
}

// Will Autobox an int to an Integer…
public Integer getRandom() {
return rnd.nextInt(1000);
}
[/source]
Note that the Autoboxing example has to Autobox the value returned by getRandom to an Integer and then the caller has to then unbox this back to an int primitive. The two examples for HalfBoxing are simply Autoboxing and then not unboxing back to an int. Here are the results:

ArrayList

[source:java]
public void runInternalTrial() throws Exception {
List list = new ArrayList();
int count = 0;
while ( count < getWorkload() ) {
for ( String s : data ) {
if ( ! list.contains( s )) {
list.add( s );
}
count++;
if ( count >= getWorkload() ) break;
}
}
}

[/source]

HashSet

[source:java]
public void runInternalTrial() throws Exception {
Set set = new HashSet();
int count = 0;
while ( count < getWorkload() ) {
for ( String s : data ) {
set.add( s );
count++;
if ( count >= getWorkload() ) break;
}
}
}

[/source]

Here are the results: ( on the time axis I’ve got the size of the data being processed )

Of interest from the results shown above is that the ArrayList implementation is actually faster for the extremely small data sizes of 5 & 10. I wasn’t exactly sure how the TreeSet would perform, but I thought it might out perform the ArrayList. I guess this is generally going to be dependent on the nature of the data being processed. HashSet, sure enough, is the hands down winner. Even with the small sized datasets where ArrayList wins, the amounts are down in the hundreds of nanoseconds realm. Once the size of the dataset hits ( still very small ) size 15, then HashSet is already 1,500 nanoseconds faster.

The new ConcurrentSkipListSet, while it is completely blown away performance-wise by even ArrayList and TreeSet, shouldn’t necessarily be counted out. If the building of the unique list isn’t something that happens too often, then even the cost of an extra 30,000 nanoseconds isn’t going to be noticable. If you are in your service layer and are using the ConcurrentSkipListSet to maintain state that may be accessed/mutated by multiple threads, the guaranteed thread safety may well be worth this extra speed cost.

Connection Pool Showdown

These results completely reinforce the fact that the expensive settings %F, %L and %C should not be used in your production environment. In the unfortunate case that you use all of them, then you’re looking at ( already expensive ) logging calls taking 9 times longer, verging on millisecond timings.

Log4J Conversion Patterns

JVM Version 1.6.0_05-ea
Windows XP[x86] – 2 processors
5.46MB/63.56MB Memory

But, then, how long does the individual usage of the perpetrators cost? I ran that one, too. Here are the results:

%l is the most expensive conversion pattern. On average it takes an extra 140,000 nanoseconds.
%C is the next most expensive, taking about 110,000 nanoseconds longer.
The rest ( %M, %F, %L ) each take about 100,000 nanoseconds longer.

Now you know.

Log4J Conversion Patterns

Freemarker vs Raw Java

JVM Version 1.6.0_05-ea
Windows XP[x86] – 2 processors
4.99MB/63.56MB Memory

So how bad is it…drum roll…

Class[10] indicates the pure Java method running a 10 Thread test. FTL[20] indicates the Freemarker test running a 20 Thread test. And so on… Results are listed from best to worst with the avg and max results in nanoseconds.

Class[10]: 1000100 trials, 18046 avg, 67973245 max 1.0x
Class[80]: 1000800 trials, 28960 avg, 104784259 max 1.6x
Class[20]: 1000200 trials, 30387 avg, 106770756 max 1.7x
Class[60]: 1000200 trials, 35917 avg, 96320112 max 2.0x
Class[40]: 1000400 trials, 37220 avg, 162898182 max 2.1x
FTL[10]: 1000100 trials, 293093 avg, 104114975 max 16.2x
FTL[20]: 1000200 trials, 587117 avg, 158432568 max 32.5x
FTL[40]: 1000400 trials, 1204641 avg, 190966244 max 66.8x
FTL[60]: 1000200 trials, 1843190 avg, 284686324 max 102.1x
FTL[80]: 1000800 trials, 2498693 avg, 358187778 max 138.5x

The last piece of info on each line is how much slower that particular test was ( on average ) than the fastest. Notice how the pure Java tests don’t jump right into the crapper when under load. But the FTL tests get progressively worse as the thread counts are increased. Under an 80 thread load, not only is the pure Java way 138 times faster on average, but the FTL way has max time of 358 milliseconds vs a 104 millisecond max for pure Java. This is a perceptible 250 millisecond difference when rendering a single button tag!

The case for easing up on the Freemarker templates is solidifying

When should a template engine be used? There seem to be two answers to this question which both carry equal weight with developers. They are:

1. You want to have an easily editable version of your dynamic view. This will allow easy customization.
2. You want the representation to be easily readable to future developers. This will keep it understandable and maintainable.

Now, this might seem like a minor point to pick at. But bear with me…

So, how much faster is pure java than freemarker templates? Well, this is going to depend largely on the complexity of the template. As an example, I’m going to pick a template of medium complexity that renders an HTML button tag in a consistent fashion. The output will be very simple and we’ll have a lot of HTML attributes that can be set within the template. Here’s the ftl file:

button.ftl

————————————–
[source:xml]
<@compress single_line=true>
<button <#if id?exists>
id=”${id}”
</#if>

<#if name?exists>
name=”${name}”
</#if>

<#if value?exists>
value=”${value}”
</#if>

<#if accesskey?exists>
accesskey=”${accesskey}”
</#if>

<#if classname?exists>
class=”${classname}”
</#if>

<#if style?exists>
style=”${style}”
</#if>

<#if title?exists>
title=”${title}”
</#if>

<#if href?exists || onclick?exists >
onclick=”
<#if href?exists>document.location.href=’${href}’;</#if>
<#if onclick?exists>${onclick}</#if>
”
</#if>

<#if ondblclick?exists>
ondblclick=”${ondblclick}”
</#if>

<#if onmouseout?exists>
onmouseout=”${onmouseout}”
</#if>

<#if type?exists>
type=”${type}”
<#else>
type=”button”
</#if>

<#if disabled?exists && disabled == ‘true’ >
disabled=”disabled”
</#if>
>
<div>
<div>
<#if icon?exists>${icon}</#if>

<span>
${content}
</span>
</div>
</div>

</button>
</@compress>
[/source]
It outputs something along the lines of :
[source:xml]
<button id=”buttonId” style=”buttonStyle” onclick=” alert(’hi’); ” type=”button” > <div> <div> <img src=’pretty.jpg’> <span> This is required </span> </div> </div> </button>
[/source]

Now, is this a good place to use a template like the above? The answer is Yes. But, as with most things, only in moderation. Or perhaps the answer is actually a definite maybe?

So, what can the downside be?

Speed.

Having just benchmarked this I have an unfair advantage, but the raw Java implementation that spits out the same exact templating logic presented above in button.ftl runs over 15 times faster.

Running 1000 trials of button.ftl (FTL) vs ButtonTemplate.java (Class) gives me the following results ( in microsecond timing ):

FTL 419033 micros
Class 31828 micros
Class is 13 times faster!

With only 100 trials:

FTL 15888 micros
Class 834 micros
Class is 19 times faster!

So, back to the moderation. Nobody is going to notice if a couple of buttons are rendered in 1 microSecond instead of 15 microSeconds, that’s for sure. But, if your entire page, overtime, has been rewritten to do absolutely everything using FTL, and the page ends up taking a second to load. Well, then, a 15x speed increase would get that page out in less than a tenth of a second.

I ran these tests with Freemarker 2.3.10 and then again with 2.3.12 and got the same results, so the optimization they added in 2.3.11 doesn’t seem to affect this type of simple usage ( using maps ).

Here is a screenshot showing the intense framework supporting the Freemarker template engine. Notice how the pure java class ( highlighted in blue ) just goes about its business directly.

So, back to the two reasons to use an FTL template.

1. You want to have an easily editable version of your dynamic view.
2. You want the representation to be easily readable to future developers.

Well, in regards to #1, I don’t think we really need to have an easily editable version of our <button> tag. I mean, really, a button’s a button’s a button. It might keep growing for a while, gathering more attributes and complexity over time, but I don’t think we’ll ever need to plug in a custom version for a different client.

As for #2, I don’t think the java code is really that much more difficult to read, and since we won’t be needing to change this (because it’s a button, for christ’s sake!) template that often, why take the speed hit here?

Here, btw, is the code for ButtonTemplate.java ( w/o Javabean boilerplate )
————————————————————————-
[source:Java]
public void render( Writer out ) throws IOException {
out.write( “<button” );
if ( id != null ) writeAttribute( out, “id”, id );
if ( name != null ) writeAttribute( out, “name”, name );
if ( value != null ) writeAttribute( out, “value”, value );
if ( accessKey != null ) writeAttribute( out, “accesskey”, accessKey );
if ( className != null ) writeAttribute( out, “class”, className );
if ( style != null ) writeAttribute( out, “style”, style );
if ( title != null ) writeAttribute( out, “title”, title );

if ( href != null || onClick != null ) {
out.write( ” onclick=\”" );
if ( href != null ) {
out.write( “document.location.href=’” );
out.write(href);
out.write(”‘;”);
}
if ( onClick != null ) {
out.write( onClick );
}
out.write( “\”" );
}

if ( onDblClick != null ) writeAttribute( out, “ondblclick”, onDblClick );
if ( onMouseOut != null ) writeAttribute( out, “onmouseout”, onMouseOut );
if ( type != null ) {
writeAttribute( out, “type”, type );
} else {
writeAttribute( out, “type”, “button” );
}
if ( disabled ) {
writeAttribute( out, “disabled”, “disabled” );
}

out.write( ” > <div> <div> ” );

if ( icon != null ) {
out.write( icon );
out.write( ” ” );
}
out.write( “<span> ” );
out.write( content );
out.write( ” </span> </div> </div> </button>” );
}

[/source]
Now, I mentioned the torrid love affair we’re in at work. Well, we are using a whole lot more FTL to simply render HTML elements : inputs, labels, selects, buttons, checkboxes, creditCards, datePickers, dropDowns, addresses ( which use the previous elements ) and even style and javascript includes. So, in conclusion, since Freemarker is the artsy smartsy way to do things and it will be used wherever and whenever possible once it takes root – I implore you to only use it where/when it is truly needed. That is, for large views that are truly complex and for views that will be frequently customized. For the simple cases, it’s just not worth the hit.

Time to go refactoring…perhaps…

Afterthough::: If only the freemarker templates would compile into java bytecode, then this would probably be a nonfactor. JSP’s can do it. Is anyone up to the task?

Well, here’s a fairly in-depth overview of both with pros and cons and similarities and differences included.

The first thing to note is that if you’re not in a multi-threaded environment, then DBCP is going to be faster than C3P0 and will also use significantly fewer connections than C3P0. For example, using the default settings for each and sizing the pool to contain 50 connections at most, will yield results like the following single-threaded test.

Single-threaded Benchmark – 50,000 calls to getConnection()

Suffice it to say that DBCP is obviously better suited to Single-threaded applications with high load. Needing 50 connections in the pool to do the work where only 1 is needed is certainly not desired. This is the case with C3P0’s default setting of “numHelperThreads=3″. Regardless of the environment you have ( single-threaded, multi-threaded ) I would always change this setting if you forsee high load on the program. I would always use at least “numHelperThreads=5″. As a bit of background/explanation, C3P0 doesn’t actually make a connection available in the pool when it is checked-in. Instead the HelperThreads will detect these and do the work to get them back in the pool. This is great in high-load, multi-threaded environments as it avoids blocking issues. But it inherently requires a much larger number of connections to provide the same functionality ( especially if you keep “numHelperThreads=3″ ). Also, notice that DBCP is 20% faster regardless of how we tweak numHelperThreads.

Score 2 points for DBCP. 2-0.

Next what if we run the same test with a smaller connection pool size – say of only 5 connections, and numHelperThreads=3 for C3P0?

The results are a bit counterintuitive ( to me at least ). DBCP does at least manage to stay the same – very fast and reasonable ( needing just 1 connection in its pool ). But C3P0, which I expected to take longer because it previously needed all 50 connections, but actually runs faster with a smaller pool : 6.25 seconds down from 6.7 seconds. The lesson here is that C3P0 is slowing itself down with the HelperThreads having to manage all the extra connections to get them back in the pool.

Score 1 point each. DBCP for making sense and C3P0 for speeding up.

3-1, DBCP in the lead.

Now for the good stuff – the multi-threaded tests. I’ll be varying the number of Threads which will be running.
Multi-threaded tests

Score yet another to DBCP. Faster across the board on all accounts. So, what is it about C3P0 that has everybody ( myself included ) using it and singing its praises? Multiple threads yield multiple points again for DBCP – 2 points awarded.

5-1, C3P0 will need a miracle comeback at this point.

I know, usually Connections aren’t just checked out and run with a small SQL statement and immediately returned like the Benchmarks I’ve been running. Maybe I need to put some delay in there to more closely mimic real world performance. I’m going to throw a new setting in here which is a delay in milliseconds that each call to getConnection() will endure before finishing with the connection. I’ll give it a shot with an extra 100ms of sleep time after each SQL statement is run.

Multi-threaded – 100ms of sleep time added

Application Servers: Tomcat, Jetty, Glassfish, JBoss

Frameworks: Spring MVC, Wicket, Seam, Struts