Backup home directory to USB harddisk

As I am keen to install Ubuntu 9.10 on my work laptop, its time to do an extra backup of my home directory. My mp3 player has about 30 Gb free so I'll use that.

First attempt:

cp -R /home/erik /media/H300/Backup/

You wont believe how slow this is! All those little pesky files, we'll need to aggregate them.

Second attempt:

tar -cjf /media/H300/Backup/erik.tar.bz2 /home/erik

Waiting ... waiting ... Oops, that fails! The FAT file system on the mp3 player only supports files up to 2Gb.

After a long but fruitless search in the documentation of tar (and zip) for multi-volume archives options, I suddenly thought of the simplest thing that could work:

tar -cj /home/erik | \ split -d -b 1G - /media/H300/Backup/erik.tar.bz2.part

This creates a bunch of 1Gbyte files ending in part00, part01, etc.

You've got to love unix!

Wicket do's and dont's

Just published an article on my employer's blog: Wicket do's and dont's.

Extending my e-mail stack with Roundcube

I don't trust anyone with my most precious data: e-mail. That is why I run my own e-mail server. The server runs Ubuntu, Postfix, Dovecot and several tools for spam interception. I access my e-mail from several machines through the IMAP protocol (with TLS). Though any good IMAP client would do it is always Thunderbird (yes, even on my Mac).

It is however not always feasible to have Thunderbird around. Time to add an IMAP web client! Candidates are IMP, Squirrelmail and Roundcube. All of these projects continuously release security updates. So unless you go for the next (beta) Ubuntu release, the packaged version is almost always a few versions behind.

In the past I have already used IMP and Squirrelmail. IMP has a nice UI but was difficult to install. Squirrelmail looks really old, but has a huge amount of nice plugins.

So it became Roundcube this time. Roundcube is not out of beta for that long. But it does look really slick, almost as if it is a desktop app. I did go with the Ubuntu package after all. Lacking more documentation I had to figure out for myself that I needed to use /etc/roundcube/apache.conf as the basis for a virtual host file in the /etc/apache2/sites-available directory. After some smaller configuration tweaks in /etc/roundcube/main.inc.php I had the application as I wanted it.

One more thing: Roundcube (like IMP and Squirrelmail) continuously open and close new IMAP connections. This makes the site amazingly slow. The fix: package imapproxy. This package starts an IMAP server that caches IMAP connections to another IMAP server. Some small config changes, and Roundcube was a lot quicker.

More Wicket filter options

Wicket has this very clever idea to serve requests from a servlet Filter instead of a Servlet. The brilliance of it is that you can serve pages on the root of your context, but still allow the servlet container to process requests that Wicket has nothing to do with.

By default this works correct automatically. Incoming requests that are not recognized by Wicket are just passed through.

However, there is an optimization that is not very well documented. When you already know that certain paths are never going to be handled by Wicket (for example because you configured a servlet for them), you can tell the Wicket filter to ignore those paths in the same file as where you define the servlets: in the web.xml.

Example web.xml configuration:

<filter> <filter-name>wicket.filter</filter-name> <filter-class>org.apache.wicket.protocol.http.WicketFilter</filter-class> <init-param> <param-name>ignorePaths</param-name> <param-value>images/,rest/</param-value> </init-param> </filter>

This will let the Wicket filter immediately pass through to the servlet container for all URLs that start with http://host/app-context/images/ and http://host/app-context/rest/.

Note: make sure the paths are comma separated without additional whitespace.

Simon 2 in beta

Java Simon, simple java monitoring, version 2 is in beta. I am quite proud of this because firstly the major change in the version was a result of my performance investigations, and secondly it contains my Spring integration code. See the java simon pages for news, downloads, etc.

Wicket course preparations

Soon I will start teaching Wicket courses in The Netherlands. To prepare I spend some days in London with jWeekend teacher Cemal Bayramoglu (also know for organizing the London Wicket meetups). Most of our time we were at jWeekend's nice classroom, right in the middle of London, with close access to a very nice Thai restaurant, several coffee corners, etc.

The first day we went through the entire course that Al Maw and Cemal created. We had great fun discussing all the ins-and-outs. The second day we focused more on how the course should be taught.

I am really glad my company JTeam also thought it was a good idea to team up with jWeekend. The course is excellent and as the number of eyes on the material has only increased, will be even more excellent in the future.

Please contact as at info@jteam.nl if you are interested in getting a Wicket course.

Amsterdam Wicket meetup March 24 2009

The Wicket meetup has finally been given a date and time! March 24 2009, 19:00 - 22:00 in the Mövenpick hotel Amsterdam.

Presentations

Registration (do not register on the page above)

Reliably sending email with Spring

Update 2009-09-12: I no longer recommend this library. Please see the comments.

My colleague Allard just pointed me to an old but very useful library: HA-JavaMail.

The email sender that the JVM provides has some serious shortcomings. It does not automatically open a new connection when the connection was closed and you can forward your e-mail to 1 SMTP server only. Furthermore, it is not so fast. HA-JavaMail circumvents these problems by wrapping the JVM implementation.

In this tiny article I explain how you configure HA-JavaMail from Spring. First make sure you have the HA-JavaMail jar, the Spring jar, the JavaMail mail.jar and the JAF activation.jar library on your classpath. The latter two are available by default in full JEE containers.

Convert your mail sender bean declaration from this:

<bean id="mailSender" class="org.springframework.mail.javamail.JavaMailSenderImpl"> <property name="mail.host" value="localhost"/> </bean>

to something like this:

<bean id="mailSender" class="org.springframework.mail.javamail.JavaMailSenderImpl"> <property name="javaMailProperties"> <value> mail.transport.host=localhost,mail.example.com mail.transport.pool-size=1 mail.transport.connect-timeout=0 mail.transport.connect-retry-period=60 mail.transport.sender-strategy=net.sf.hajavamail.SimpleSenderStrategy </value> </property> </bean>

Note that the example adds a second SMTP server that will be used when the first is unreachable. Only a value for mail.transport.host is required. The other values shown here are also the defaults.

That’s it. You now have a much more reliable and faster email service.

Runtime monitoring libraries for Java

Update 2013-02-06: Nowadays, I am a full user of Coda Hale's Metrics library. I maintain a fork of the metrics-scala component for different scala versions.

Followers of this blog may have noticed an emphasis on monitoring lately. This is because my employer has decided to give me some time to investigate monitoring and create new components to ease the use of monitoring (in particular for Wicket applications). Blogging about the results is one of the requisites of this exercise.

This article lists all open source Java runtime monitoring tools I could find. By runtime monitoring I mean something that can left active in a production environment and measures things that occur inside the JVM.

Jamon
Jamon started in 2003 and is probably the oldest monitoring framework listed here. Although Jamon’s last release is from 2007, it is quite stable and used a lot; Sourceforge statistics show almost a 1000 downloads per month. Jamon is the work of a single person.

Features:

• Programmatic access to start and stop timers, timers use milliseconds.
• In memory database for aggregate statistics (no details are kept).
Supports monitors that keep track of data in any unit (e.g. dollar value of a transaction).
• Can aggregate statistics on data ranges to provide a more detailed view when this is necessary.
• Extensions to easily configure monitoring in software such as JDBC drivers, several servlet containers/application servers.
• A JSP page to view the realtime data.

Jamon does not support a persistence mechanism and there is therefore no way to analyze historic data. To fill this gap you can use Jarep which provides several Jamon exports. Another way is to export the data through JMX.

Java Simon
Simon started last year as a direct competitor of Jamon. It is much faster, measures in nanoseconds and has a hierarchy of monitors so that you can enable/disable a tree of monitors. Simon is still in active development. Be prepared for API changes in Simon 2.0 which is to be released somewhen before summer 2009.

Features:

• Programmatic access to start and stop timers, timers use nanoseconds.
• In memory database for aggregate statistics (no details are kept).
Supports stopwatches and counters.
• Like Jamon, has a JDBC monitoring proxy driver

The Simon team is working on a JMX export and has yet to be determined plans for a persistence mechanism.

Currently Simon does not provide anything to view the statistics. Again you can use Jarep for this.

Usemon
Usemon seems to target monitoring of large clusters with support for tracing messages through the cluster. Measurements are send to a collector which stores it in a database.

Features:

• Byte code manipulation to add timers to configured EJBs, Servlets, MDBs, custom defined POJOs, and classes implementing the SqlStatement, MessageSender and TopicSender interfaces.
• Measures method invocation durations, exception counts and JVM information such as heap size.
• Measurements are aggregated for 1 minute and then actively send to a central collector for storage.
• A nice graphic interface for viewing the data (the project page is very vague about the details).
• A clever database to handle huge amounts of data while still allowing efficient retrieval.

The Usemon site does not offer a lot of information. You'll probably need to just try it out to get more information.

Moskito
I found it a bit difficult to quickly grap Moskito. The website only shows a few use cases and leaves the rest undocumented.

Features:

• Easy monitoring of servlets.
• Measurements are aggregated per interval (e.g. 5 miniutes). Afterwards you can do something with the data (e.g. log it or send to a central location).
• A nice very complete looking web interface.

A nice idea (that has not been implemented) is the support for virtual monitors, a monitor that shows the aggregated statistics for several other monitors.

Unfortunately I could not figure out how to instrument code except for servlets (extending a base class). The project website suggest that the project has been abandoned, so Moskito is not likely a good choice.

Commons monitoring
Commons monitoring is a Apache sandbox project. The website has good documentation but the newest information is from March 2008. SVN shows quite some activity so I am curious what the first release will bring.

Features:

• Programmatic access to start and stop timers, timers use nanoseconds.
• Support for AOP instrumentation.
• In memory database for aggregate statistics (no details are kept).
• Supports timers, counter (only increases) and gauges (increases and decreases).
• Extensions to export data through a servlet in several formats, including HTML.

JETM
JETM (Java™ Execution Time Measurement Library) is again a library for doing raw measurements.

Features:

• Programmatic access to start and stop timers, timers use nanoseconds.
• Very nice: a Spring namespace to configure monitors on your beans.
• In memory database for aggregate statistics (no details are kept).
• In memory database for aggregate statistics (details are kept until it is time to calculate the aggregates).
• Several renderers to export data.
• Web interface to see current data.
• Integrates with several projects as Spring, RRD4J and JMX.

Although JETM looks nice, it has been silent for more then a year, and simple questions on the mailing list are not answered.

Project Broadway
Broadway provides an entirely different kind of monitoring. The idea of Broadway is to define events of interest (for example in Groovy) on which can be reacted with some action.

Improving Jamon’s performance

I was browsing through Jamon’s code to see why it is so much slower under high contention then Simon (see my previous article). In this article I will present these differences and show you a way to speed up Jamon right now.

Differences
Jamon has more features then Simon like listeners and data ranges. This obviously need some computing, if only to see if they are used. Another interesting difference is that Jamon uses double’s for aggregate statistics, whereas Simon uses long’s. I don’t really see the advantage of double’s (please surprise me) and it is probably a bit slower as well.

Synchronization
Jamon uses 5 synchronization points during a timing operation. There are 3 to get and create the monitor: on the MonitorFactory method, on the map of all monitor datas and on the map of all data range definitions. The last 2 synchronization points are for starting and stopping the timer (on the monitor data).

Simon (does not support data ranges) uses only 3 synchronization points: one for getting the monitor (on the SimonManager method) and 2 for starting and stopping the timer (on the timer itself).
Simon 2.0 no longer needs synchronization on starting a timer and therefore only needs 2 synchronization points.

Jamon defines its maps for monitors and data ranges as Collections.synchronizedMap(new HashMap(50)). However, it is not necessary to synchronize on these maps when the code that uses these maps is already synchronized on the MonitorFactory (in method getMonitor). Unfortunately when the synchronization wrapper is removed, all code that uses these maps will need to be analyzed to see if they properly synchronize.

Another solution would be to use a map that needs less synchronization: Java 5’s ConcurrentHashMap! Luckily Jamon provides a method to change the map implementation for exactly this reason. I included the following line in my test application (from my previous post):

MonitorFactory.setMap(new ConcurrentHashMap(200));

And these are the results. The measurements are in ms, the scale is logarithmic.

Quite astonishing: Jamon is no longer 5 up to 200 times slower, but just a bit slower for low contention, up to 15 times slower under heavy contention, and 2 times slower under extreme contention. Note that only the map for monitors was changed, the map for range definitions (always empty in my test) was not changed.

Conclusions
Even though Jamon is quite a bit slower under heavy contention, there is room for improvement. By calling a single method, you can make Jamon 20 times faster right now.

‘Effective Wicket’ presentation now on Vimeo

My Effective Wicket talk on NL-JUG’s J-Fall conference of 2008-11-12 is now online! Watch the presentation (Dutch spoken) on Vimeo: part 1 and part 2 (provided by bachelor-ict.nl).

Feel free to download the slides of Effective Wicket (mirror) (PDF 4.1 Mb). If you want to see the notes skip to page 55.