Go west young man

This is (likely) my last post to this blog. I've spent the past 2 and a 1/2 years using Maven primarily in my Java projects and my companies. Part of this I think is just growing up. When I first started using Maven I had just started work at a company who's development team consisted of some novice Java users. It of course did not help then that a lot of the software they had was written in Java. Queue some of the ugliest Ant build files I have ever seen. These people obviously had never been exposed to C/C++ and make in an academic environment... as they would have been heckled for such lack of simplicity. These Ant build scripts just wouldn't fly for me. So I found Maven. I used it to simplify the build environment, for my co-workers and myself. I encountered problems while using Maven which I solved. I watched the community slowly build up. I started a blog and shared my knowledge. It was fun.

I've progressed A LOT in the past two years. I am no longer a guy who is fresh out of college, smart but unprepared for nastiness which the enterprises of the world contain. I've honed my technique, worked on numerous projects (many more than the majority of people out of college for only a few years), and I have made honest appraisals of the things I have done. I'm at a point now where the expressivity of a tool such as Ant makes it much more viable than something like Maven. If Maven was Java, then Ant would be Python. I can do the same two things in both, but one is much easier to use (Python, duh).

Other things that have pulled me away from Maven, and more specifically Maven's ideology, are tools such as Gant (Groovy language + Ant) and scripts in Pylons (Python web framework). Being able to use the full power of a language right there in a build file, is a beautiful and powerful thing. I also would argue this is daunting for the novice. I have come full circle from Ant to Maven and back to Ant. I'm now ready to work on projects where the full responsibility is mine in making sure the build files are clean, coherent and useful. I recognize now that this is not a lot of responsibility, that the fact my previous co-workers could not handle it speaks volumes for the type of organization they were running.

Please drop a line if you have any questions about articles I've written here. You can reach me at mentalstasis. at. hotmail. com. I'm always interested in some convo with other computer professionals out there so don't hesitate. Thanks to all you readers out there, and best of luck.

-Rob O.

Utilizing Maven Plugins: Jetty plugin

The problem: You would like to use a non-standard Maven plugin, such as the Jetty plugin.

The solution: First things first you will probably need to register the plugins group, so that Maven will know where to look. To do this you should put an entry in your settings.xml file. For our example here we will use the Jetty plugin. It has a group id of 'org.mortbay.jetty'. In your settings.xml file put an entry such as:

<pluginGroups>
  <pluginGroup>org.mortbay.jetty</pluginGroup>
</pluginGroups>

What this little piece of configuration does is tell Maven to search for plugins registered under the given group id. You can have as many 'pluginGroup' elements in your xml file as you want/need. In all my work with Maven I have only actually registered two plugin groups, one of which is Jetty... but if you create your own you will need to register your plugin group.

Lets start by creating a web application using a Maven archetype:

mvn archetype:create -DgroupId=com.mycompany.app -DartifactId=my-webapp -DarchetypeArtifactId=maven-archetype-webapp

This will create a folder called 'my-webapp'. Change over to this directory. Now we are going to use the long way of specifying the plugin. This is ugly and you certainly will not want to use this form often (likely ever).

mvn org.mortbay.jetty:maven-jetty-plugin:run

You should now be able to hit your web app on http://localhost:8080. The above command could be broken down as 'mvn groupId:artifactId:goal'. More on this in a bit. Now lets run the same 'run' goal using a short command syntax. Here is a simpler way to do the same thing:

mvn jetty:run

Much easier. Now I'll explain a bit. Since we put the plugin group entry in our settings.xml file we do not need to specify it like we did in the long command above. In the long command, after the group id, we have 'maven-jetty-plugin'. Maven expects the artifact id to be in a form 'maven--plugin'. For the Jetty plugin the artifact id is 'maven-jetty-plugin'. In the part after the colon in the long command we specify this artifact id. Now in the simple version we just have 'jetty'. Maven will take the 'jetty' value and automatically know to look for 'maven-jetty-plugin'. This saves you from typing out the whole shebang.

If your are interested in creating your own plugin I suggest you check this link out. It is not nearly as hard as you may imagine.

Exercise: Continuum

Today's article is going to be a step by step on getting a pretty hefty Maven based project up and running on your machine. I will finish by having the project set up in Eclipse of better development. The project is Continuum. It is a continuous integration server created by the Maven community.

We will start by checking out Continuum. Use this command to get the project from Subversion:

svn checkout http://svn.apache.org/repos/asf/maven/continuum/trunk/ continuum

Once you have the project downloaded 'cd' on over into it. Do a ls or dir command. You will now find you are looking at a multi-module Maven project! Egads thats a different sight in comparison to a non-multi-module project. Well lets kick things off by running:

mvn install -Dmaven.test.skip

We don't want to test, for now. You will need to run install so that each module's artifact (jar) gets put into your local repo. This is important because a module can rely on artifacts created by other modules, and so the artifacts must be available via repo. This will take a bit, Continuum is fairly big. Once things are done you will want to prep Continuum for Eclipse. Run the command:

mvn eclipse:eclipse

This will go into each module and create the Eclipse meta files, so that you can import them into Eclipse no problems.

We now are at the last step. Open up Eclipse, right click in the 'package manager' view (window). Select 'import'. This pops up a window, select 'general > existing projects into workspace'. Point to the folder containing all the modules and click ok. You will notice Eclipse finds a project for each module. Click ok. After a bit of grunting Eclipse will present you will a view full of projects, one for each module.

At this point you may need one last thing. If you do not have the M2_REPO Eclipse you will need to set it. Check the 'problems' view, if you see a bunch of messages about this not being set... bingo. Go into preferences then "java > build path > classpath" and add the var, giving it a value of the location of your local repo. Mine is "/home/ottaway/.m2/repository".

Going Offline

The problem: You want Maven to stop making all those network calls when you issue it commands like clean, build, test, package etc. You might want to work on a project while you have no network connection.

The solution: You can use the off line mode for Maven to keep it from making any network calls. You can do so by issuing the -o flag to the mvn command. When using this mode Maven will need to have all the jars it needs in your local repository (~/.m2/repository by default). To make this easier you can use a trick provided by the dependency plugin. Imagine we just checked out a project. Now we want to work on this project off line. To get all the dependencies needed by the project you can issue the command "mvn dependency:go-offline". This will download the dependencies you need. Now note it will not make sure you have all the plugins you need. Say you want to run "mvn -o site:run", creating a site for your project and making it available via Jetty. You will need to have used this plugin before so that it is in your local repo. The same goes for compile, package, test and the like.

The million dollar question is why use off line mode? Well it could be that it makes quicker the process you are doing with Maven, since Maven will not try to call out to any network repos. I've timed the difference between using it and not using it for some fairly large projects and found it doesn't make a huge difference. For really big projects (I cannot really name one) it might shed enough seconds to make it worth using often, maybe even creating an alias. This 'huge' project mind you would probably have upwards of 200 jar (dependencies) or more being managed. Another reason one might use off line is as follows: you have a local network repository used by the developers at work. One day the person you administers this repository is going to move it somewhere new. This going to take a few hours. Using off line mode you could weather this time if prepared without noticing too much that the repository is down. This could also be helpful in situations where you are going to not be able to reach the repository, like working at home with no network connection, or the server hosting the repo crashed.

Using Maven for Bash Script Projects

I recently created a Maven project unlike any I have ever created. This project contained no Java sources! No it was not a Groovy project ;) It was a project that contained only bash scripts. Read up for the why and the how.

I created a set of bash scripts that allow invoking SOAP services, returning the XML response. The SOAP services provide information on email distribution lists, such as which addresses are on a list and which lists a user's email address is subscribed to. These scripts are likely to change and grow over time so I made sure to put them into Subversion. Not too shabby. But what about distributing the scripts? What about managing versions of the scripts? I have become accustomed to having the aforementioned handled in my projects via Maven. Maven makes such things simple to set up and automate. I determined I could move these bash scripts into a Maven project to cultivate these benefits.

Now the how. For packaging I use the Maven assembly plugin. It allows created automated creation of zip tar.gz and gzip archives. These archives work great work distributing a project. I started my work by creating a skeleton Maven project. I put the scripts and related files into the src/main/scripts folder of the project. Next I created a xml file used by the assembly plugin to direct how files will be structured in the created compressed archive or archives. Here my file:

<?xml version="1.0"?>
<assembly>
    <id>zimbrascripts</id>
    <formats>
        <format>zip</format>
    </formats>
    <includeBaseDirectory>false</includeBaseDirectory>
    <dependencySets>
        <dependencySet>
            <outputDirectory>hacksdb/lib</outputDirectory>
        </dependencySet>
    </dependencySets>
    <fileSets>
        <fileSet>
            <directory>src/main/scriptapp/bin</directory>
            <outputDirectory>zimbrascripts/bin</outputDirectory>
            <fileMode>0755</fileMode>
            <directoryMode>0755</directoryMode>
        </fileSet>
        <fileSet>
            <directory>src/main/scriptapp/etc</directory>
            <outputDirectory>zimbrascripts/etc</outputDirectory>
            <fileMode>0644</fileMode>
            <directoryMode>0755</directoryMode>
        </fileSet>
        <fileSet>
            <directory>src/main/scriptapp/tokens</directory>
            <outputDirectory>zimbrascripts/tokens</outputDirectory>
            <fileMode>0600</fileMode>
            <directoryMode>0777</directoryMode>
        </fileSet>
        <fileSet>
            <directory>src/main/scriptapp/xml</directory>
            <outputDirectory>zimbrascripts/xml</outputDirectory>
            <fileMode>0644</fileMode>
            <directoryMode>0755</directoryMode>
        </fileSet>
    </fileSets>
    <files>
        <file>
            <source>src/main/scriptapp/README.txt</source>
            <outputDirectory>zimbrascripts</outputDirectory>
        </file>
    </files>
</assembly>

I'll write up a more thorough article on using the assembly plugin later. For now suffice to say this file will instruct Maven to create a zip archive containing the files listed above. The last thing we want to do is configure the assembly plugin in the project's POM file. I used the following XML in my projects pom:

<build>
  <plugins>
    <plugin>
      <artifactId>maven-assembly-plugin</artifactId>
      <configuration>
        <descriptors>
          <descriptor>src/main/assembly/assembly.xml</descriptor>
        </descriptors>
      </configuration>
      <executions>
        <execution>
          <id>make</id>
          <phase>package</phase>
          <goals>
            <goal>assembly</goal>
          </goals>
        </execution>
      </executions>
    </plugin>
  </plugins>
</build>

In the above config I also bind the assembly goal to the package phase. This means when I execute "mvn package" the assembly plugin will fire off and create the zip archive. Nice! Otherwise you can also use "mvn assembly:assembly" should do the job. Now so that I could use the scm and release plugins to check in changes to SVN and create tagged releases of the project I added the following to the POM:

<scm>
  <connection>scm:svn:https://svn.com/DistributionLists/DistListScriptsMaven/trunk/</connection>
  <developerConnection>scm:svn:https://svn.com/DistributionLists/DistListScriptsMaven/trunk/</developerConnection>
  <url>https://svn.com/DistributionLists/DistListScriptsMaven/</url>
</scm>

Now to release the current version of the project I can type "mvn release:prepare". This will tag the current version in SVN, then update the POM to the new version number and check those changes into the SVN trunk. This saves me the hassle and reduces the chances of user error in this boring, repetitive process. In the vein of the SCM configuration you could now use other plugins such as the JIRA reporting plugin, site plugin etc. to bolster this simple bash script project.

Big Project Troubleshooting Pt. 1: Transitive Dependencies

Maven is complex, but so is Ant or Make once a project passes a certain size. With Maven many things are done in a standard way, so that when a project becomes large you still should be able to work with it easily. With Ant or Make a large project will likely have many arbitrarily named targets which you will have to study and learn. With Maven all those named targets are pretty much standardized by the plugins which provide all functionality. Large projects in Maven commonly have the same sorts of issues. These can be pretty nasty to for the uninitiated. So to help you out I'm going to write a series of articles tackling the problems faced in big Maven projects. In this article I'm going to give you some information on issues related to dependencies.

The dependency mechanism is the main reason many people become interested in Maven. It sure makes life easy... but everything cannot always be automated. Sometimes you will need to give Maven some help. It will try to correctly resolve the dependencies needed by your project by creating a graph. More on this in a second.

Maven has what is called a 'transitive dependency'. Let me explain transitive dependencies for those who don't know. A dependency of a project is defined via a project's pom.xml. Every dependency has a pom.xml (in the repository with the dependency) which is used by Maven to find the dependencies relied upon by a dependency. These 'relied upon' dependencies are also known as transitive dependencies. They are included into your projects class path automatically. Now you can have any number of levels of transitive dependencies because Maven also will get the each transitive dependency's dependencies. In Maven this forms what is called the dependency graph. Yes this is the same graph I spoke of earlier. It looks like a tree, with your project as the root node, tree level one as the dependencies of your project and subsequent levels being transitive dependencies.

Now lets talk about the sticky points. If there are two or more versions of the same dependency in the graph of the dependencies for your project then Maven will need to make a choice as to which one to use. By default Maven will use the version that is the closest to the top node of the tree. If this version is older than the other version(s) you may have a problem. Likely a NoClassDef where a class from the new dependency cannot be found, because it does not exist in the older version. When you get this error it should be obvious which dependency the error is being caused by. The stack trace should give the full package path to the class. We can use this to figure out how to fix the problem.

At this time you probably could open a terminal session and cd to your local repository location (~/.m2/repository by default) and then cd into the dependency's folder in the repository. This can be helpful since it will allow you to see which versions of the dependency are being used and which classes are in each dependency. You could execute a "jar -tf {jarname} | grep {ClassName}" command on each jar to search it for the given class.

Let us discuss troubleshooting. First you will want to run the mvn dependency:resolve command. This will print out a listing of the resolved dependencies for your project. Resolved dependencies are those calculated by Maven via the dependency graph to be used in the project. You should be able to see which version of the troubled dependency is being used. I had you cd to the dependencies repo location. Check now which versions you have in your repository. You should be able to use the "jar -tf " command to check if the class causing the NoClassDef exists in one of the newer version jars (if you have newer version jars). Once you have found a jar that contains the class make note of the version.

Now for some debugging. This is where you get waist deep in it. We are going to use the -X option to Maven, which enables debugging. We also want to run just the compile phase which will give us all the info we need on the dependency graph. You might want to do something like "mvn -X compiler:compile > junk.txt" just so that you can use a text editor like Vi to search about the file. Now when you view this output you will be able to find a listing of the absolute classpath, including with jars are being used. You also will get a printout of the dependency graph including info on what ended up making it in. It is pretty easy to find the version of a dependency being used: find the one with the least amount of indent. If one or more versions share the least amount of indent then the newest version will be selected.

To get around version problems you can do a number of things. The easiest thing to do is define the newer version of the dependency directly in the pom.xml of the project. This will make sure that the newer version is used because it is 1) at the top of the dependency tree 2) the version is highest. You can also use excludes for a dependency definition in the pom.xml file for the old version of the dependency. Since the old version is excluded the newer (yet further down the graph) version will be used. When using excludes you will need to figure out which dependency in the pom.xml file has the transitive dependency to be excluded. Neither of these fixes are tough to perform, and will become easy to do in time.

Maven Theory: Archetypes

Maven is a complex system to swallow in its entirety. Most persons first encounter it because they want to wield the dependency magic it offers. These persons are not savvy to the notion of repositories and with them the administrative work required to build and manage them. They are not likely to understand that Maven can build complex websites, containing useful reports on project health, with little configuration and user effort. These are things that Maven accomplishes in a very elegant manor (in my opinion). You see Maven 2 was built almost from scratch, using the knowledge gained in the creation of the original Maven. Hence a sound architecture was built in Maven 2 that has proven adaptable to many tasks.

In this post I want to talk more about one of the extension points of Maven 2. Archetypes. These are a lot like templates. When invoked an archetype will either create a new project, or add to an existing one. Archetypes can be created by an end user and deployed into a repository. This allows others to access the archetype and reuse it. A little while back I was doing a fair bit of SOAP service programming. I was using XFire in the creation of these projects, along with other Java technologies. After the first couple projects I decided creating an archetype for XFire projects would be handy. So I created an archetype project that could generate a bare bones Maven 2 project. This project would contain a pom with all the needed dependencies, and a simple "Hello World" SOAP service and an integration test. Now whenever I wanted to create a new XFire project I would just invoke this archetype; giving me a good head start! Now I would like to point out that, if you are like me, you would normally just find the 'similar' project then copy and paste. Then you trim down the copy or whatever. That is a viable alternative, although I feel one is better off with a nearly clean canvas when striking out on a project.

So now that I have talked your ear off, let me give an example of using a real archetype to generate a full project. The base Maven has an archetype for creating Maven projects. Seems reasonable eh? So to create your next computational masterpiece start with the command:

mvn archetype:create -DgroupId=com.someguy -DartifactId=masterpiece

And tadah you now have a folder called 'masterpiece' which contains a very basic Maven 2 project. There are other archetypes you can run with out any special setup, including an archetype to create archtypes! Now I'm not advocating creating an archetype as a precursor to any new project. Create one whenever you start to notice a pattern between projects. A good candidate for me was Spring, Hibernate and other dependencies I used very often in web projects. I created a simple archetype to create a bare bones web MVC application using these dependencies. With this I could start creating the controllers and .jsp pages in my application right away, rather than mucking around copying some old ant project :)

I will write up some better examples on archetypes down the road. I would like to give an example of creating a simple archetype. Keep tuned in for more, and do not hesitate to leave a comment or contact with ideas.