C3P0 vs DBCP – The Straight Dope

November 12th, 2007 deevis 5 comments

I’ve found it very difficult to find any accurate comparisons of C3P0 and DBCP. Why should I choose C3P0 over DBCP? When might DBCP be better?

Well, here’s a fairly in-depth overview of both with pros and cons and similarities and differences included.

The first thing to note is that if you’re not in a multi-threaded environment, then DBCP is going to be faster than C3P0 and will also use significantly fewer connections than C3P0. For example, using the default settings for each and sizing the pool to contain 50 connections at most, will yield results like the following single-threaded test.

Single-threaded Benchmark – 50,000 calls to getConnection()

Trials MaxPoolSize Connections Used Seconds Settings
DBCP 50,000 50 1 5.18
C3P0 50,000 50 50 6.72 *numHelperThreads=3
C3P0 50,000 50 39 6.6 numHelperThreads=4
C3P0 50,000 50 27 6.45 numHelperThreads=5
C3P0 50,000 50 30 6.63 numHelperThreads=6

Suffice it to say that DBCP is obviously better suited to Single-threaded applications with high load. Needing 50 connections in the pool to do the work where only 1 is needed is certainly not desired. This is the case with C3P0’s default setting of “numHelperThreads=3″. Regardless of the environment you have ( single-threaded, multi-threaded ) I would always change this setting if you forsee high load on the program. I would always use at least “numHelperThreads=5″. As a bit of background/explanation, C3P0 doesn’t actually make a connection available in the pool when it is checked-in. Instead the HelperThreads will detect these and do the work to get them back in the pool. This is great in high-load, multi-threaded environments as it avoids blocking issues. But it inherently requires a much larger number of connections to provide the same functionality ( especially if you keep “numHelperThreads=3″ ). Also, notice that DBCP is 20% faster regardless of how we tweak numHelperThreads.

Score 2 points for DBCP. 2-0.

Next what if we run the same test with a smaller connection pool size – say of only 5 connections, and numHelperThreads=3 for C3P0?

The results are a bit counterintuitive ( to me at least ). DBCP does at least manage to stay the same – very fast and reasonable ( needing just 1 connection in its pool ). But C3P0, which I expected to take longer because it previously needed all 50 connections, but actually runs faster with a smaller pool : 6.25 seconds down from 6.7 seconds. The lesson here is that C3P0 is slowing itself down with the HelperThreads having to manage all the extra connections to get them back in the pool.

Trials MaxPoolSize Connections Used Seconds Settings
DBCP 50,000 5 1 5.18
C3P0 50,000 5 5 6.18 *numHelperThreads=3

Score 1 point each. DBCP for making sense and C3P0 for speeding up.

3-1, DBCP in the lead.

Now for the good stuff – the multi-threaded tests. I’ll be varying the number of Threads which will be running.
Multi-threaded tests

Threads MaxPoolSize Connections Used Seconds Settings
DBCP 25 5 5 90
DBCP 25 10 10 107
DBCP 25 25 25 167
DBCP 50 50 50 198
DBCP 50 100 50 207
C3P0 25 5 5 156 *numHelperThreads=3
C3P0 25 10 10 142 *numHelperThreads=3
C3P0 25 25 25 137 *numHelperThreads=3
C3P0 50 50 50 252 *numHelperThreads=3
C3P0 50 100 100 252 *numHelperThreads=3
C3P0 50 100 100 269 numHelperThreads=6

Score yet another to DBCP. Faster across the board on all accounts. So, what is it about C3P0 that has everybody ( myself included ) using it and singing its praises? Multiple threads yield multiple points again for DBCP – 2 points awarded.

5-1, C3P0 will need a miracle comeback at this point.

I know, usually Connections aren’t just checked out and run with a small SQL statement and immediately returned like the Benchmarks I’ve been running. Maybe I need to put some delay in there to more closely mimic real world performance. I’m going to throw a new setting in here which is a delay in milliseconds that each call to getConnection() will endure before finishing with the connection. I’ll give it a shot with an extra 100ms of sleep time after each SQL statement is run.

Multi-threaded – 100ms of sleep time added

Threads MaxPoolSize Connections Used Seconds Settings
DBCP 50 25 25 9338
DBCP 50 10 10 20918
DBCP 100 50 50 9248
DBCP 0 50 50 0
DBCP 0 100 50 0
C3P0 50 25 25 9295 numHelperThreads=6
C3P0 50 10 10 20088 numHelperThreads=6
C3P0 100 50 50 9412 numHelperThreads=6
C3P0 0 50 50 0 *numHelperThreads=3
C3P0 0 100 100 0 *numHelperThreads=3
C3P0 0 100 100 0 numHelperThreads=6
Categories: Benchmarks, Open Source Tags:

Integrating Google Earth with a Java webapp

November 11th, 2007 deevis No comments

I did this at work the other day and thought I’d give the broad sweeping overview here should else want to do it. It’s not too bad after all’s said and done ( and sifted through… ). Here’s the quick and easy:

1) Add these mimetypes in web.xml:


application/vnd.google-earth.kml+xml
kml


application/vnd.google-earth.kmz
kmz


2) You’ll want to write back KML ( a flavor of XML ) with a content-type of “application/vnd.google-earth.kml+xml”:

response.setContentType(”application/vnd.google-earth.kml+xml”);

3) Make sure you give the stream a default filename that ends in .kml or else GoogleEarth won’t actually process the file.

response.setHeader(”Content-Disposition”, “attachment; filename=” + filename + “.kml”);

That’s really a 40,000 foot view of the process. Here’s Google’s KML Tutorial so you can learn more.

Categories: Uncategorized Tags:

C3P0 ConnectionPool Configuration Rules of Thumb

November 4th, 2007 deevis No comments

C3P0 is a fantastic open source ConnectionPool that is feature rich, stable, and positively production ready. I personally like it more than DBCP because there is too much blocking encountered when checking in/out connections with DBCP. C3P0 dodges that problem with an asynchronous return to the pool, but you might end up using a lot more connections than you need to if you’re not careful. Here’s a list of the most important C3P0 settings and good production settings to use – along with some words of wisdom as to why the change from default is a good thing.

acquireIncrement=5 default(3) Batching the retrieval of Connections is wise because the process of opening a Connection is expensive. Why do it 5 times instead of once? The downside to this is that you may only need the one connection. Read on to maxIdleTime settings to make sure idle connections make their way back into the pool.

maxIdleTime=3600 default(0) Setting the maxIdleTime to a non-zero value permits C3P0 to remove Connections from the pool and freeing up database resources. This will never happen when maxIdleTime is set to 0. Setting to a small value might have Connections being retrieved and returned on too frequent a basis to be practical, so I’ve picked 3600 seconds ( 1 hour ) as a nice happy medium. Once the peak traffic is over, then the connections can be returned.

maxIdleTimeExcessConnections=300 default(0) You need to set this in addition to maxIdleTime to have the Connections removed from the pool and returned to the DB.

maxPoolSize=100 default(15) You’ll have to fine tune this setting base on your own individual situation, but it’s important to keep this on the largish side for the simple reason that C3P0 doesn’t synchronously return connections to the pool. The implication of this is that the pool should actually be sized slightly larger than what you think the maximum number of Connections needed at peak time would be. For more on this, see numHelperThreads.

minPoolSize=20 default(3) This one too can be fine tuned based on your individual needs, but let’s just say that you don’t always want to be returning all your Connections and having to reacquire new ones. 20 is a good size to allow for handling of new bursts of sporadic traffic w/o having to reacquire them.

numHelperThreads=6 default(3) This is a very important property if you’re going to have large bursts of traffic on your site. C3P0 ( as was alluded to previously ) doesn’t return connections to the pool synchronously and as a result doesn’t have the same issues with blocking that DBCP has. However, as a result, Connections are not made available until some arbitrary time after they are returned to the pool. C3P0 has helperThreads which are responsible for doing the actual return work and making them available. Under large load I’ve experienced times where these spikes could require 400 Connections using the default setting of 3. Using 6 HelperThreads will keep this spike under 100 connections. If you can afford the connections or won’t have huge loads on the system, then the default of 3 will probably be fine for you.

unreturnedConnectionTimeout=3600 default(0) I’ve set this to 1 hour. If a Connection is used for longer than an hour, then C3P0 will assume it’s been orphaned and will reclaim the Connection to the pool – closing it in the process.

Categories: Open Source, Uncategorized Tags:

Checking Email with Java

November 3rd, 2007 deevis No comments

I’ve probably used Java to send email somewhere in the neighborhood of 30 times and have never really had much trouble with it. But how in blazes do I connect to a (Pop3) account and pull down email? Well, after much cursing at the javamail API the solution crystallized to something containing the following:

Properties props = new Properties();
Session session = Session.getDefaultInstance(props, null);
store = session.getStore("pop3");
store.connect( "mail.isp.com", "myUserName", "myPassword" );
inbox = store.getFolder("INBOX");
inbox.open(Folder.READ_ONLY);
Message[] msgs = inbox.getMessages();
// Do something with the messages here
inbox.close(false);
store.close();

It should be noted that the elusive activation.jar and mail.jar from J2EE 1.4 were used to get the above to compile and run. When I tried using the JEE5 libraries, I didn’t have a POP3 implementation and got “JavaMail – NoSuchProviderException: No provider for pop3″ exceptions.

Here’s a code snippet that will show you all of the providers you have available to you:


Properties props = new Properties();
Session session = Session.getDefaultInstance(props, null);
Provider[] p = session.getProviders();
for (int i=0;i System.out.println(p[i].getProtocol());
}

I get back the results: imap, pop3, smtp.

Categories: Open Source, Uncategorized Tags:

The Wicket Component Model Explained

October 30th, 2007 deevis No comments

Wicket is a Java/HTML approach to building the guts of a J2EE web application, and I must admit that when I first heard of it I was both eagerly excited (to try it) and more than mildly pessimistic (that an application written in Wicket would be maintainable beyond a certain trivial size ). The past few months I’ve heard only good things about Wicket, and haven’t heard anything in the negative. I’m eager to benchmark it and check it out for performance on the Application Servers, but figure first things first – let’s tear it apart, see what makes it tick, and find out what really matters in Wicket.

So, basically everything ( apart from the WebApplication itself ) in Wicket is a Component. Now the Wicket Component is no slouch; it’s a 3,000+ line abstract class with loads of functionality built right in. I’ll dig into this base class later, but for now, let’s focus on the two immediate subclasses of Component: MarkupContainer and WebComponent.

MarkupContainers include things like: WebPage, RedirectPage, Form, FormComponent, FormComponentLabel, Button, CheckBox, TextArea, and TextField.

WebComponents include things like: Image, Include, Label.

I’m surprised, initially, to find controls such as Button, CheckBox, TextArea and TextField as MarkupContainers and not as WebComponents.

Also of interest is that subclassing is used to achieve behavior that might seem best placed in attributes of some of the classes. For example, to create a hidden input field, I might expect to create a TextField instance and set its visible property false. Instead, TextField is subclassed as HiddenField. This is also true for adding password characteristics (PasswordTextField) or making the input required (RequiredTextField). This same pattern of subclassing is used most thoroughly for Links ( AjaxFallbackLink, BookmarkablePageLink, DownloadLink, PageLink, PopupCloseLink, and ResourceLink ).

So, there’s a very rich Component-Model existing within Wicket for creating all the elements you’ll need on a web page, but where does the web page come from? For this we look to another subclass of MarkupContainer. The class WebPage is the primary analog to a conventional webpage. So much so, that the Wicket convention is actually to sit an HTML file of the same (prefix) name as the (simpleName) of the Java class extending WebPage. That is, if you were to write a new class called FunPage like this:

package org.javatech.wicket.examples;
class FunPage extends WebPage {}

Then you would have an HTML file named FunPage.html located in org/javatech/wicket/examples which would contain the HTML used in rendering the webpage.

Cool.

But this is still a bit too abstract – should we really bother writing (yet another) Hello World application for Wicket? No – there are plenty of them out there. Instead, let’s discuss how the Components get added to the WebPage and then how the HTML is able to provide the layout used to create the desired web page. And since every website since the beginning of time has forms on it, let’s show how to add a form.

Adding a form using Wicket Components and a Wicket WebPage:

When I think of an html form, my mind drifts towards thoughts of Maps where the form element’s names are the keys and the values entered by the user are the corresponding values. Then there are just a few other lingering details like the form’s name, whether to use POST or GET, and to what URL the form data should be sent. Having isolated the Component Form as the candidate for the job, I’m immediately stunned by the Constructor which takes an id and/or an IModel. The id makes sense. I mean, really, every html element should have one. But what on god’s green earth is an IModel? I know one thing, it’s an interface – score one for the I-school of thought! Upon inspection, it turns out that an IModel is simply a wrapper for a model object used by a component. It provides methods to get/set the actual model object and to drill down into a nested model object. This being learned, I see that it isn’t required for such humble beginnings as just getting a form to work and shall be forgotten for at least two days now.

So, our first form will be called DemoForm and starts out on the Java side as:

package org.javatech.wicket.pages;

import wicket.markup.html.WebPage;
import wicket.markup.html.basic.Label;
import wicket.markup.html.form.Button;
import wicket.markup.html.form.Form;
import wicket.markup.html.form.TextField;

public class DemoForm extends WebPage {

public DemoForm() {
Form f = new Form( “myForm” );
f.add( new TextField( “myTextField” ));
Button button = new Button(”mySubmitButton”);
f.add( button );
f.setDefaultButton( button );
add( f );
}
}
Which should look something like:

When I start Wicket and hit the URL, I get an exception, because I haven’t created the appropriate HTML file.
WicketMessage: Markup of type ‘html’ for component ‘org.javatech.wicket.pages.LandingPage’ not found

So, it turns out that you need to have EVERY component added in the Java represented in the corresponding HTML. In this case, the HTML would need to contain:

<form wicket:id="myForm">
<input wicket:id="myTextField" />
<input type="submit" wicket:id="mySubmitButton"></button>
</form>

So I gotta admit that my Spidey-sense is tingling at this point. This has the potential to grow quite ugly – having to manage the same essential layout in both Java code and in the HTML. More on that later – so, now my form comes up and works.
I can type into the textbox and click on the button. When I do click on the button, however, I’m taken to a urHell:
http://localhost:8080/hello_wicket/?wicket:interface=:24:myForm::IFormSubmitListener – SAY WHAT?

So, enough suspense. Here are the remaining hoops that Wicket requires you to jump through:
1) You must have a backing model object for your form.
2) You must explicitly implement onSubmit() on the Form object being used.
3) The onSubmit() method will ( in Java code! ) specify the response page for the form.
4) You must (manually) pass the model object to the Confirmation Page.

We’ll have a new class called DemoFormModel defined as:


package org.javatech.wicket.pages;

import java.io.Serializable;

public class DemoFormModel implements Serializable {

private String myTextField;

public String getMyTextField() {
return myTextField;
}

public void setMyTextField(String myTextField) {
this.myTextField = myTextField;
}
}
An instance of DemoFormModel will be instantiated in the constructor of DemoForm and will then be passed ( in the new onSubmit() method ) to the DemoFormConfirmation WebPage. DemoForm now looks like:


package org.javatech.wicket.pages;

import wicket.markup.html.WebPage;
import wicket.markup.html.form.Button;
import wicket.markup.html.form.Form;
import wicket.markup.html.form.TextField;
import wicket.model.CompoundPropertyModel;

public class DemoForm extends WebPage {

public DemoForm() {
Form f = new Form( “myForm” ) {
@Override
protected void onSubmit() {
DemoFormModel o = (DemoFormModel)getModelObject();
DemoFormConfirmation landingPageSubmission = new DemoFormConfirmation( o );
setResponsePage( landingPageSubmission );
}

};
f.setModel( new CompoundPropertyModel( new DemoFormModel() ));
f.add( new TextField( “myTextField” ));
Button button = new Button(”mySubmitButton”);
f.add( button );
f.setDefaultButton( button );
add( f );
}
}
The Java and HTML for DemoFormConfirmation look like:

DemoFormConfirmation.java

package org.javatech.wicket.pages;

import wicket.markup.html.WebPage;
import wicket.markup.html.basic.Label;

public class DemoFormConfirmation extends WebPage {

public DemoFormConfirmation(DemoFormModel o) {
this.add( new Label( “myTextField”, o.getMyTextField() ));
}
}
DemoFormConfirmation.html

You entered:<span wicket:id="myTextField"></span>

So, in a mighty big nutshell, we needed to create 5 files to pull off a simple form submission and confirmation:

DemoForm.java, DemoForm.html, DemoFormModel.java, DemoFormConfirmation.java, and DemoFormConfirmation.html.

DemoForm specifies the form, the components on the form, the model object the components will bind to, and the form’s onSubmit() method. The onSubmit() method is responsible for retrieving the model object, instantiating the DemoFormConfirmation instance with that object, and finally redirecting to that instance.

Then, DemoForm.html needs to reference the components added in DemoForm.java by the same String id’s and DemoFormModel.java needs to have properties which match these names. DemoFormConfirmation.java and DemoFormConfirmation.html have the same sort of coupling but on a smaller scale ( 1 field only ).

The big lesson here is that naming conventions are super important and having a reliable, predictable mechanism for coming up with wicket id’s is going to be essential. Fortunately, there are numerous ways to do this with the simplest being a ModelObject first driven development cycle. Once your ModelObject is created, then it’s fieldNames should be used verbatim in both the Components added on the Java side and the wicket:id’s in the HTML.

I’m not particularly excited about hardcoding site navigation inside of onSubmit() methods in Java code. I’ve been completely brainwashed that this should be configurable behavior, but I’ve seen configurable behavior become a royal pain in the butt and am willing to entertain this backward seeming restriction. Besides, there are way to dynamically wire this stuff together if one is determined to do so ( custom code at this point in time ).

Server Side – Application Server Matrix

October 28th, 2007 deevis No comments

The ServerSide  has put together a great listing of all the J2EE Application Server solutions that are available ( both free and for cost ) in their Application Server Matrix.  It’s nice to have such a comprehensive list of offerings and I learned that if you want to use EJB3, then it looks like JBoss or Glassfish for you.

Categories: Application Servers Tags:

What’s the skinny on popular Open Source projects?

October 27th, 2007 deevis No comments

Spring, Hibernate, Tomcat, JUnit, Eclipse, Log4j, XFire, XStream, Seam, Wicket, Jetty, JFreeeChart, ActiveMQ, ServiceMix, Axis, Mule. What’s the optimal configuration? What should you avoid?

Categories: Open Source Tags:

Coming soon – lots of benchmarks

October 27th, 2007 deevis No comments

Webservices: Apache Axis vs XFire

Application Servers: Tomcat, Jetty, Glassfish, JBoss

Frameworks: Spring MVC, Wicket, Seam, Struts

Categories: Benchmarks Tags:

Hello world!

May 18th, 2007 deevis No comments

Welcome to WordPress. This is your first post. Edit or delete it, then start blogging!

Categories: Uncategorized Tags: