Cloudera Hadoop Training: Programming with Hadoop from Cloudera on Vimeo.

1) Locate the snapshotId you’d like to use ( snap-1781757e for Wikipedia Extraction (WEX))
2) Create a volume from your AWS console – use the snapshotId here. The volume size must be larger than the snapshot. Also, I’ve found that I need to create the volume in the same zone as the instance I’m going to attach it to.
3) Attach the volume to an instance. Select the volume and then click “attach volume”. Suppose for example you chose “/dev/sdf” as the mount point.
4) ssh to your instance and mount the volume: mount /dev/sdf /vol

There you have it. But we’re not quite done – we still need to import all of this data into MySQL or Postgresql. But that’s another post…

To get apache web server with php/mysql support installed I just ran:
yum install httpd php-mysql
This is every bit as easy as apt-get. Making some really good progess. So, the first time around I terminate the instance and figure I’ll pick up where I’ve left off.
The next time when I come back to the AWS console – my instance isn’t there for me to restart. Now, here’s the thing with the EC2 server instances :

WHEN YOU TERMINATE AN EC2 INSTANCE, IT’S LIKE YOU STUFFED THE CASE WITH TNT AND THREW IT INTO A CAMPFIRE.

That’s right, terminate doesn’t mean power-down. It means obliterate. Anything on the drives is G-O-N-E.  It is no more – it’s an ex-instance.

That was rather eye-opening.  So, back to the drawing board and this time, to be safe, I made sure to pick an AMI with EC2 utilities installed on it already.   I installed my software on the instance and this time even registered the services to startup automatically.
service mysql start
service httpd start
I restarted the instance and tested that it came back up with apache and mysql running – it did.  Good enough – it was time to save this instance as an AMI into one of my S3 buckets.  After a bit of finagling I found the correct couple of incantations.
ec2-bundle-vol –cert publicCert –privatekey privatePem –user awsAccountNumber
The above command will create an imageManifestXml which you’ll use in the following statement:
ec2-upload-bundle –bucket s3BucketName –accessKey awsAccessKey –secretKey awsSecretKey –manifest imageManifestXml

This took maybe ten minutes to run both of the above, but afterwards I had new starting point AMI to use – in theory at least…

I double checked everything by starting another instance ( before shutting down the good one ) with the new AMI and making sure that it passed snuff – which it did. I shut down both instances and retired for the night – all of this to save 10 cents an hour. But more importantly it is to understand the process. No, it’s the money
This virtualization stuff is really powerful – I’ll write some more on mounting Volumes and using existing Snapshots of open-sourced datasets soon.

First – I have to install Roo and get the Roo shell up and running.  I’m really excited at this point because Roo does look really cool – really RAD!

So I download Roo and explode it out to where I explode all of my 3rd party tools and then open cygwin (I’m on Windows and am trying hard to move away from Dos…) and navigate to the new directory.  I find a bin directory with two scripts in it: roo.bat and roo.sh.  Sweet!  They have a linux version ( cygwin == linux in many ways ).  So I run roo.sh and get:

./roo.sh

Exception in thread "main" java.lang.NoClassDefFoundError: org/springframework/roo/bootstrap/Bootstrap
Caused by: java.lang.ClassNotFoundException: org.springframework.roo.bootstrap.Bootstrap
        at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
Could not find the main class: org.springframework.roo.bootstrap.Bootstrap.  Program will exit.

OK – take a step back.  What have I done wrong here.

I back up a directory and also try ./bin/roo.sh.  I look in the readme.txt and create the symlink for install on linux that they recommend:

ln -s /path/to/roo/bin/roo.sh /usr/bin/roo

I type in ‘roo’ and get the same BootStrap Exception.

So then I invoke the wisdom of Google and learn that most everyone who encounters this error gets it because they are in Dos and are working in a different drive than where Roo is installed.  I’m on the same drive – but it’s not working in cygwin!  No hits on this somehow?!?  I find some other results which want me to additionally set an environment variable ROO_HOME.  I do this by running from the Roo install directory ( above bin, that is…):

export ROO_HOME=`pwd`

env | grep ROO

I still get the same problem (ClassNotFoundException: org.springframework.roo.bootstrap.Bootstrap), so I cave and go into a Dos prompt. Well, first I add %ROO_HOME%\bin to my path, and *then* go into a Dos prompt. I run ‘roo’ and get the same Bootstrap Exception again. But now, I am on the wrong drive, and cd to the same drive I installed Roo on – now things are working.

Time to even start using Roo in this example : 15 minutes

So, Roo is working now – I have a shell up and running. I’m stoked because it really is cool. It has ‘hint’ as a keyword and tab-based auto-completion which is contextual – it auto-completes the correct things at the correct times.
Creating a project with Roo is a very nice experience : 3 minutes

I already have maven2 installed so I’m able to exit out of Roo and run ‘mvn eclipse:eclipse’ and import the project into Eclipse : 2 minutes

Wow, now we’re flying.  Except, wait, there a bunch of “.aj” files in my project.  Cool, Roo is leveraging AspectJ.  Suck, I don’t have AJDT plugin configured for Eclipse and I’m missing a bunch of the stuff Roo has added to my project ( 9 .aj files in all ).  Thinking this might not matter, I run the JUnitTest that Roo created for my project and see only 1 test being run – the demo has a bunch of tests being run ( added via AspectJ ).  Also, when I auto-complete on my entity class I created, I don’t get any of the framework added static methods.  Translation: I need to install the AJDT plugin for Eclipse: 10 minutes ( http://download.eclipse.org/tools/ajdt/34/update ).

Now things are good to go – I now see the out-of-the box static methods added to my domain class ( Choice.java ).   And now when I run the JUnitTest I see that 8 new tests have been added and run.  But there’s something else stinky in the state of Denmark!

IllegalStateException: Entity manager has not been injected (is the Spring Aspects JAR configured as an AJC/AJDT aspects library?

No, the SpringAspects JAR is *NOT* configured as an AJC/AJDT aspects library!  I don’t know what the SpringAspects JAR is nor what it means to configure it as an AJC/AJDT aspects library.  Basically, this means you need to select the spring-aspects.jar to be in the Aspect Path of the AspectJ Build configuration of the project.  A picture is worth a 1,000 words here

Adding spring-aspects.jar to AspectJ Build path

10 minutes

So, we’re up to 40 minutes of time spent when the whole thing up until now should only be 5 minutes.

But wait, there’s more.  Now I find that I don’t have setters being woven into my Choice.java class by the Choice_Roo_Javabean.aj aspect.  I see the correct code in the aspect, but Eclipse’s autocomplete and the actual compilation fail when I attempt to access new Choice().setNamingChoice().    I can deal with this later because I’ve got to get on with the demo.

Next up – Roo will automatically update the companion aspect files it creates as new properties are added to the domain entity.  That is, if I simply add a new property called ‘description’ to my entity, and I have roo running in the background, then the Choice_Roo_Javabean.aj is automatically updated with the new getter/setter methods, and the Choice_Roo_ToString.aj is updated to include description.  SUPER COOL.

On a tangent, it is not mandatory that these woven aspects dictate the toString and javabean parts of the entity class.  The class has annotations in it to specify that the framework will handle these concerns: @RooJavaBean, @RooToString.  There is also a @RooEntity which adds all of the dao-ish methods to the class.  This is a pattern taken from Grails.

At this point we have an autogenerated JavaBean and a companion TestCase that works.

What about the webapp?  Inside of Roo shell, we can now run:

new controller automatic

With a few arguments to generate the framework jsps, the wiring, the formBackingObject functionality and even the crud forms for the domain entity.  The ordering of the fields on the crud forms matches the order in which they appear in the java file.  Changing the order of properties in the java file will change their order in the ToString.aj and crud JSPs as well. 2 minutes

The mighty behemoth, proprietary squelchor of innovation, and extremely expensive Oracle is about to acquire the open-source, free, and always innovative Sun ( and with that Java!).  Anyway I look at this it sucks – for Java, at least.  This will, in my humble opinion, accelerate Java’s path to Cobol – which it is destined to become as more agile languages come to maturity and continue to grow in popularity ( ruby, python, erlang ).

Rod Johnson, with Spring, is singing an optimistic tune ( Spring Keynote Address on Future of Java ) .  One that talks about Java having reached maturity and that it doesn’t really need to evolve anymore.  C doesn’t really evolve – it does what it does and does it very well.  As Rod says, “It doesn’t pretend to be C#.”  Point taken.  He goes on to say that in many ways the innovation in the Java community has already moved outside of the Java core.  Language improvements include Groovy and Scala.  Framework improvements come in the form of Grails ( and, he doesn’t mention this, Seam ).

Rod’s address is largely true, but it is also to drum up excitement for their new application generation project ROO.  This gives alot of the functionality one would get from Rails or Seam, but does things within a shell rather than all command line.  This is actually kind of nice because the shell can keep track of the context each subsequent command is running in and also allows for some pretty snazzy auto-completion features.

Should we Javaheads run around screaming the sky is falling?  Probably not.  But if we don’t get up to snuff with Grails and new projects like Seam and Roo which look to make things a whole lot RAD-der, then we’d better accept our role as stewards to the next generation of Cobol applications.

One more thing – Ruby, for example, is still missing a lot of enterprise tooling which we take for granted in the Java world.  Applications like JConsole and JProfiler don’t yet exists – so another way to look at this is on the larger programming stage, where what we’ve learned with Java should be created for Ruby, Python, and Erlang to ease the passing of the torch when the time comes.

MAVEN_OPTS="-Xmx1024m -Xms512m"

export MAVEN_OPTS

for file in `ls **/*.txt`;do perl -pi -w -e ’s/SEARCH_FOR_THIS/REPLACE_WITH_THIS/g;’ $file; done

The nested ‘perl’ command can work kind of like ’sed’.  Here are what the arguments being passed in mean:

-e means execute the following line of code.
-i means edit in-place
-w write warnings
-p loop

What I’ve got before me is a target/lib directory full of all of my maven2-managed dependencies.  I need to add each jar file explicitly into the classpath – barf!

Well, here’s a handy little shell script which will build that string for you:

for file in `ls lib`; do echo -n ‘lib/’ >> classpath.txt;echo -n $file >> classpath.txt;echo -n ‘:’ >> classpath.txt;done

Awesome – so just open the classpath.txt file and use that big ‘ol path however you want to.  Windows users would use a semi-colon instead of a colon above.  Also, remember that you need to specify the classpath option prior to the name of the class to run – otherwise the classpath is ignored ( as it is assumed to be an argument to the program and not a jvm option ).

You should login to mysql and select the correct database/schema prior to running that command.

mysql -uusername -p

use <schema>;

load data local infile 'data.csv' into table <tableName>
fields terminated by ','
enclosed by '"'
lines terminated by '\n'
(<columnName1>, <columnName2>, <columnName3>) ;

1. Using the latest and greatest versions of everything.  Eclipse 3.4.  Hibernate 3.3.1.  Spring 2.5.6.  Tomcat 6.  Maven 2.0.9.  Java 6 ( 7? ), DWR 3.0.
2. Using Eclipse plugins to do as much as possible without causing headaches
3. Using as little XML configuration as possible – conventions, conventions, conventions!!!
4. Using JPA annotations for ORM classes.
5. Using Annotations for IOC dependencies.
6. Using Annotations for Transactional demarcation.
7. Using Annotations for MVC mappings.
8. Using very basic JSTL-based JSP’s for views.
9. Using Sitemesh to present a uniform look and feel across all pages.
10. For icing on the cake I’d like to also include GWT components.

Using the latest and greatest versions of everything
If you install Eclipse 3.4 first, and then install the Maven Integration plugin, it will greatly simplify things.  Just add Maven Integration to your project and then use “Add Dependency” option to quickly scour the entire repository and find the exact groupIds, artifactIds, and versions you want.  It also has a great view of the dependency hierarchy!  Getting all of these bleeding-edge versions to play nice together has its fair share of challenges.

Using as little XML configuration as possible

Using Eclipse plugins to do as much as possible without causing headaches

I already mentioned Maven Integration above to help with getting the LAGV of all the technologies being used.  Also of great value and import are SysDeo and  Subclipse.  Sysdeo is great for deploying and debugging your webapp in Tomcat.  Subclipse is great for seamless integration with an SVN repostiory.  Yes – use SCM on personal projects, you never know when you’ll need to revert or bring the code down onto a new machine.

But where do we draw the line here?  When do we have so much annotation cruft that an externalized xml configuration will seem like the ideal solution ( from whence we currently come in search of clarity! ).

Using very basic JSTL-based JSP’s for views
OK – well, tonight I was getting ready to put together the first couple of jsp’s now that the @Controller mappings are working and the pages are being served correctly.  I’ve got the method mapped correctly with @RequestMapping and I’ve got the model being put into pagescope with @ModelAttribute.  I’m ready to loop through a List using JSTL forEach.  But wait, can this possibly be true?  Tomcat6 has no implementation of JSTL.  JSTL 1.2 is the LAGV and the JSTL 1.1 is abandoned.  Only *Glassfish* ships with the reference implementation of JSTL 1.2?!?  Oh brother.  Thanks to Peter Mularien for writing a nice post informing me of this.

As if needing to download and install glassfish in order to get the JSTL 1.2 implementation jars for Tomcat6 wasn’t bad enough, well I hit a couple more snags after that.  First of all, I want to know who decided on the namespaces for JSTL 1.1 vs 1.2.  Here they are:

1.0  == <%@taglib prefix=”c” uri=”http://java.sun.com/jsp/core” %>

1.1 & 1.2 ==<%@taglib prefix=”c” uri=”http://java.sun.com/jsp/jstl/core” %>

I mean all they did was add the “jstl” into the path?!?  I’m supposed to remember this?  OK – I’ll try harder to remember for next time.

Then, I learned that Maven2 doesn’t copy “system” provided jars into WEB-INF/lib by default and they need to be placed there manually as part of the project.  So, the appserv-jstl.jar I copied from glassfish wasn’t being included because I linked Maven to it as a system ( vs provided, or compile, or test, etc… ) dependency.