Amsterdam Wicket meetup March 24 2009

The Wicket meetup has finally been given a date and time! March 24 2009, 19:00 - 22:00 in the Mövenpick hotel Amsterdam.

Presentations

Registration (do not register on the page above)

Reliably sending email with Spring

Update 2009-09-12: I no longer recommend this library. Please see the comments.

My colleague Allard just pointed me to an old but very useful library: HA-JavaMail.

The email sender that the JVM provides has some serious shortcomings. It does not automatically open a new connection when the connection was closed and you can forward your e-mail to 1 SMTP server only. Furthermore, it is not so fast. HA-JavaMail circumvents these problems by wrapping the JVM implementation.

In this tiny article I explain how you configure HA-JavaMail from Spring. First make sure you have the HA-JavaMail jar, the Spring jar, the JavaMail mail.jar and the JAF activation.jar library on your classpath. The latter two are available by default in full JEE containers.

Convert your mail sender bean declaration from this:

<bean id="mailSender" class="org.springframework.mail.javamail.JavaMailSenderImpl"> <property name="mail.host" value="localhost"/> </bean>

to something like this:

<bean id="mailSender" class="org.springframework.mail.javamail.JavaMailSenderImpl"> <property name="javaMailProperties"> <value> mail.transport.host=localhost,mail.example.com mail.transport.pool-size=1 mail.transport.connect-timeout=0 mail.transport.connect-retry-period=60 mail.transport.sender-strategy=net.sf.hajavamail.SimpleSenderStrategy </value> </property> </bean>

Note that the example adds a second SMTP server that will be used when the first is unreachable. Only a value for mail.transport.host is required. The other values shown here are also the defaults.

That’s it. You now have a much more reliable and faster email service.

Runtime monitoring libraries for Java

Update 2013-02-06: Nowadays, I am a full user of Coda Hale's Metrics library. I maintain a fork of the metrics-scala component for different scala versions.

Followers of this blog may have noticed an emphasis on monitoring lately. This is because my employer has decided to give me some time to investigate monitoring and create new components to ease the use of monitoring (in particular for Wicket applications). Blogging about the results is one of the requisites of this exercise.

This article lists all open source Java runtime monitoring tools I could find. By runtime monitoring I mean something that can left active in a production environment and measures things that occur inside the JVM.

Jamon
Jamon started in 2003 and is probably the oldest monitoring framework listed here. Although Jamon’s last release is from 2007, it is quite stable and used a lot; Sourceforge statistics show almost a 1000 downloads per month. Jamon is the work of a single person.

Features:

• Programmatic access to start and stop timers, timers use milliseconds.
• In memory database for aggregate statistics (no details are kept).
Supports monitors that keep track of data in any unit (e.g. dollar value of a transaction).
• Can aggregate statistics on data ranges to provide a more detailed view when this is necessary.
• Extensions to easily configure monitoring in software such as JDBC drivers, several servlet containers/application servers.
• A JSP page to view the realtime data.

Jamon does not support a persistence mechanism and there is therefore no way to analyze historic data. To fill this gap you can use Jarep which provides several Jamon exports. Another way is to export the data through JMX.

Java Simon
Simon started last year as a direct competitor of Jamon. It is much faster, measures in nanoseconds and has a hierarchy of monitors so that you can enable/disable a tree of monitors. Simon is still in active development. Be prepared for API changes in Simon 2.0 which is to be released somewhen before summer 2009.

Features:

• Programmatic access to start and stop timers, timers use nanoseconds.
• In memory database for aggregate statistics (no details are kept).
Supports stopwatches and counters.
• Like Jamon, has a JDBC monitoring proxy driver

The Simon team is working on a JMX export and has yet to be determined plans for a persistence mechanism.

Currently Simon does not provide anything to view the statistics. Again you can use Jarep for this.

Usemon
Usemon seems to target monitoring of large clusters with support for tracing messages through the cluster. Measurements are send to a collector which stores it in a database.

Features:

• Byte code manipulation to add timers to configured EJBs, Servlets, MDBs, custom defined POJOs, and classes implementing the SqlStatement, MessageSender and TopicSender interfaces.
• Measures method invocation durations, exception counts and JVM information such as heap size.
• Measurements are aggregated for 1 minute and then actively send to a central collector for storage.
• A nice graphic interface for viewing the data (the project page is very vague about the details).
• A clever database to handle huge amounts of data while still allowing efficient retrieval.

The Usemon site does not offer a lot of information. You'll probably need to just try it out to get more information.

Moskito
I found it a bit difficult to quickly grap Moskito. The website only shows a few use cases and leaves the rest undocumented.

Features:

• Easy monitoring of servlets.
• Measurements are aggregated per interval (e.g. 5 miniutes). Afterwards you can do something with the data (e.g. log it or send to a central location).
• A nice very complete looking web interface.

A nice idea (that has not been implemented) is the support for virtual monitors, a monitor that shows the aggregated statistics for several other monitors.

Unfortunately I could not figure out how to instrument code except for servlets (extending a base class). The project website suggest that the project has been abandoned, so Moskito is not likely a good choice.

Commons monitoring
Commons monitoring is a Apache sandbox project. The website has good documentation but the newest information is from March 2008. SVN shows quite some activity so I am curious what the first release will bring.

Features:

• Programmatic access to start and stop timers, timers use nanoseconds.
• Support for AOP instrumentation.
• In memory database for aggregate statistics (no details are kept).
• Supports timers, counter (only increases) and gauges (increases and decreases).
• Extensions to export data through a servlet in several formats, including HTML.

JETM
JETM (Java™ Execution Time Measurement Library) is again a library for doing raw measurements.

Features:

• Programmatic access to start and stop timers, timers use nanoseconds.
• Very nice: a Spring namespace to configure monitors on your beans.
• In memory database for aggregate statistics (no details are kept).
• In memory database for aggregate statistics (details are kept until it is time to calculate the aggregates).
• Several renderers to export data.
• Web interface to see current data.
• Integrates with several projects as Spring, RRD4J and JMX.

Although JETM looks nice, it has been silent for more then a year, and simple questions on the mailing list are not answered.

Project Broadway
Broadway provides an entirely different kind of monitoring. The idea of Broadway is to define events of interest (for example in Groovy) on which can be reacted with some action.

Improving Jamon’s performance

I was browsing through Jamon’s code to see why it is so much slower under high contention then Simon (see my previous article). In this article I will present these differences and show you a way to speed up Jamon right now.

Differences
Jamon has more features then Simon like listeners and data ranges. This obviously need some computing, if only to see if they are used. Another interesting difference is that Jamon uses double’s for aggregate statistics, whereas Simon uses long’s. I don’t really see the advantage of double’s (please surprise me) and it is probably a bit slower as well.

Synchronization
Jamon uses 5 synchronization points during a timing operation. There are 3 to get and create the monitor: on the MonitorFactory method, on the map of all monitor datas and on the map of all data range definitions. The last 2 synchronization points are for starting and stopping the timer (on the monitor data).

Simon (does not support data ranges) uses only 3 synchronization points: one for getting the monitor (on the SimonManager method) and 2 for starting and stopping the timer (on the timer itself).
Simon 2.0 no longer needs synchronization on starting a timer and therefore only needs 2 synchronization points.

Jamon defines its maps for monitors and data ranges as Collections.synchronizedMap(new HashMap(50)). However, it is not necessary to synchronize on these maps when the code that uses these maps is already synchronized on the MonitorFactory (in method getMonitor). Unfortunately when the synchronization wrapper is removed, all code that uses these maps will need to be analyzed to see if they properly synchronize.

Another solution would be to use a map that needs less synchronization: Java 5’s ConcurrentHashMap! Luckily Jamon provides a method to change the map implementation for exactly this reason. I included the following line in my test application (from my previous post):

MonitorFactory.setMap(new ConcurrentHashMap(200));

And these are the results. The measurements are in ms, the scale is logarithmic.

Quite astonishing: Jamon is no longer 5 up to 200 times slower, but just a bit slower for low contention, up to 15 times slower under heavy contention, and 2 times slower under extreme contention. Note that only the map for monitors was changed, the map for range definitions (always empty in my test) was not changed.

Conclusions
Even though Jamon is quite a bit slower under heavy contention, there is room for improvement. By calling a single method, you can make Jamon 20 times faster right now.

‘Effective Wicket’ presentation now on Vimeo

My Effective Wicket talk on NL-JUG’s J-Fall conference of 2008-11-12 is now online! Watch the presentation (Dutch spoken) on Vimeo: part 1 and part 2 (provided by bachelor-ict.nl).

Feel free to download the slides of Effective Wicket (mirror) (PDF 4.1 Mb). If you want to see the notes skip to page 55.

Evaluating Simon - Java monitoring

Recently a new monitoring kid appeared on the block: Java Simon (not to be confused with Dejal's Simon or some other simons).

Simon claims to be the successor of JAMon. As I just started a project to improve the documentation of Jamon, and integrate this better with Wicket projects, I thought this would be a good time to evaluate Simon and also to compare it to Jamon. Here are the results.

Features
Simon (only just started) can measure task durations (min, max, count, maximum concurrency and some timestamps) and has task counters. Other statistics can be added (supported now: standard deviation). The JDBC proxy driver can be used to monitor database usage. The monitors are grouped in a tree. You can disable each monitor individually, or per complete subtree.

Jamon adds the ability to measure in any unit (for example euro value of a transaction) and has quite a few monitors for Tomcat, Jetty, etc. Jamon does not organize monitors in a tree, so monitors can be disabled individually, or all at the same time.

So Jamon clearly wins on the feature front, but if you're just into the basic stuff Simon is fine.

Documentation
One of the weak points of Jamon is its documentation (hence my little project). Simon on the other hand starts with a good set of wiki pages. If these are maintained properly, Simon is the clear winner here.

Simon interface
Simon's interface is fairly simple. One gets a monitor through the singleton SimonManager, and call start and stop on it:

Stopwatch simon = SimonManager.getStopwatch("simon.test").start(); // Do something simon.stop();

(I will not discuss the Jamon interface further, but making a measurement is virtually the same.)

Getting data out of a Simon monitor (a simon) is a bit more difficult. You can just get the measurements like this:

Stopwatch simon = SimonManager.getStopwatch("simon.test"); long hitCount = simon.getCounter(); long max = simon.getMax(); long total = simon.getTotal(); simon.reset();

But that would be very thread unsafe. As each method is synchronized separately, the three measurements may represent different moments in time.

The idiomatic solution would be to use method sample:

Stopwatch simon = SimonManager.getStopwatch("simon.test"); Map<String, String> sample = simon.sample(true); long hitCount = Long.parseLong(sample.get("counter")); long max = Long.parseLong(sample.get("max")); long total = Long.parseLong(sample.get("total"));

This is more how it should be except for the boolean flag [F3, N7], and the return type of sample: a map! First of all, maps are very memory hungry [G21], secondly I have to assume that the values are longs [CE6] and parse them back from a String [GE37], and thirdly the keys are magic values [CE6, G25].

Here is another approach:

Stopwatch simon = SimonManager.getStopwatch("simon.test"); long hitCount; long max; long total; synchronized (simon) { hitCount = simon.getCounter(); max = simon.getMax(); total = simon.getTotal(); simon.reset(); }

Ouch. This works but uses client-side locking. That this is possible at all is a mistake in the Java language (as admitted by James Gosling, sorry, I lost the reference). Even so, Simon should not make it possible for clients to get the same lock that is used to protect the monitor data. This should be the only way to get actual data.

Recommendation: move all data retrieval methods (getMax(), getTotal(), etc.) from the monitor to an immutable data class, e.g. StopwatchMeasurements. Method sample should return instances of this class, or subclasses when a different analyzer was configured.

Update 2008-12-25: added following section on visiting all simons.
Update 2009-01-0: the iterator now actually works.

As I am using Jarep to browse through historic measurements, I want to expose the Simon data just like Jarep's servlet exposer for Jamon does. In other words: we need to visit each simon and extract the data as described above.

There are 2 approaches to visit all nodes. First is to get all simon names from SimonManager and then request them one by one. The other is to walk the tree of simons. I choose the latter. Unfortunately there is no tree iterator so I created one with commons-collections:

Simon rootSimon = SimonManager.getRootSimon(); ObjectGraphIterator simonIterator = new ObjectGraphIterator(rootSimon, new Transformer() { private Object previous; public Object transform(Object input) { if (input == previous) { previous = null; return input; } else { previous = input; Simon currentSimon = (Simon) input; return new IteratorChain( new SingletonIterator(currentSimon), currentSimon.getChildren().iterator()); } }});

Admittedly ugly, but it works. Still there is one glaring bug here. The children are stored in a regular TreeSet. When we request the children through getChildren we get a collection that wraps the original set. Unfortunately that means that when an simon is added while we iterate the complete simon tree, we get a ConcurrentModificationException! Not quite acceptable.

Recommendation: store children of a simon in a CopyOnWriteArrayList instead of a TreeSet. Another option is change the complete set of simons in the manager into a ConcurrentMap and provide an iterator over that.

Performance To measure the performance of Simon and Jamon I wrote a test application that mimics a random web applications. It has a variable number of threads to mimic the number of concurrent requests that are being handled. There is one monitor to measure the duration of the entire request, one monitor to measure a specific request, one monitor to measure the service layer, and 0 to 3 monitors to measure the database layer.

The test application was run from IntelliJ with Soy Latte 1.0.3 (a 1.6 JDK, part of the OpenJDK for Debian) on a 2.4GHz Intel Core 2 Duo.

The measurements below display the time spend in obtaining, starting and stopping a single monitor. Measurements were done in nanoseconds, but are displayed here in milliseconds (logarithmic scale).

Simon's performance is pretty good! The overhead in my test application doesn't come anywhere near the millisecond range even in highly concurrent situations. Jamon is not so bad either, even though its 5 to 200 times slower, depending on thread contention. With 5 monitors in a request, the overhead per request slowly grows to 0.7 ms for Simon, and quickly grows to 12ms for Jamon.

Note: the test application does nothing else but getting monitors and doing measurements. So the right side of the chart is unrealistic for most environments.

Improving the performance
Even though the performance is pretty good. I think it can be improved further. I investigated two potential performance improvements:

• Remove the ThreadLocal variable in StopWatchImpl.
• Replace synchronized HashMap with a ConcurrentHashMap in EnabledManager.

Remove the ThreadLocal variable in StopWatchImpl Stopwatch method start stores the start time in a thread local. When you call stop the start time is retrieved and used to calculate the duration.

ThreadLocals have seen an enormous performance improvement in recent JVMs, but you are still better off without.

The idea is as follows: method start will no longer return the monitor itself but an object that contains only the start time and a stop method. Lets call it a StopwatchMeasurement. Only in the stop method will it be necessary to synchronize and update the counters.
The advantages are:

• We no longer need to synchronize the start method.
• We no longer need the thread local variable to store the start time.
• We no longer need the start(key) and stop(key) variants.
• The stopwatch is no longer susceptible for memory leak when a malicious client keeps starting monitors in new threads, but never stops them (someone forgot a finally).
• The stopwatch is no longer susceptible for memory leak when a malicious client keeps starting monitors with start(key) with a new key, but never stops them (again, someone forgot a finally).

Disadvantage is that we need to use something else for the active counter. An AtomicLong would be very suitable as it uses hardware assisted atomic operations (and therefore has no synchronization overhead).
If it is desirable to keep the firstUsage and lastUsege timestamps (I don't use them), these can lift along with the start time and be stored during the stop method. Note: this may be unacceptable if you need information on long running operations.

To test it out I hacked StopwatchImpl to return a StopwatchMeasurement from the start method. If you look at the implementation you see that StopwatchMeasurement implements Stopwatch so that the idiomatic usage still works:

Stopwatch simon = SimonManager.getStopwatch("simon.test").start(); // Do something simon.stop();

Of course this is a kludge that should be improved upon for a real implementation. In addition, I made no attempt to keep the active counter and the first/lastUsage timestamps.

Here are the results (again in ms per measurement):

Not bad! Although the results are quite dramatic for the higher thread counts, it still shaves off 20% in situations with less contention.

Recommendation: remove the thread local from StopwatchImpl and change the API as described above.

Replace synchronized HashMap with a ConcurrentHashMap in EnabledManager
The second thing I noticed is that getting a monitor from the SimonManager (the SimonManager.getStopwatch("simon.test") part) happens in a synchronized block and uses a plain HashMap as store. Since Java 5 there is a ConcurrentHashMap implementation which supposedly performs better in almost any circumstances.

Lets see. I remove some synchronized keywords, changed the map type, and made the code to add new monitors thread safe. This resulted in a modified EnabledManager.

Here are the results. The graph displays a few attempts that differ in the concurrency level parameter to the ConcurrentHashMap constructor. For more accuracy this graph displays the improvement in nanoseconds.

To my big surprise there is no measurable difference with modest contention. During high contention there is a reasonable positive effect that is suddenly converted to a negative effect for very high contention. Only when the currency level is set very high, does the effect stay positive.

Recommendation: don't change the manager.

Conclusions Although Simon is only young, it is off with a good start. Its clearly faster then Jamon and also on the documentation front Simon wins. Jamon has much more features so it is still a valid choice. When the recommendations in this article are followed, Simon could be better still.

Discussion
Follow the discussion of this evaluation on the Java Simon mailing list.

References
Code smells from Clean Code by Robin C. Martin:
[F3] Flag arguments
[N7] Names should describe side effects
[G14] Feature Envy
[G21] Understand the algorithm
[G25] Replace magic numbers with named constants

Code smells by me:
[CE6] Incomplete contract
[GE37] Unnecessary conversions

Don't use Intellij

IntelliJ is okay, but it has one bug (for more then 4 years) that absolutely drives me nuts: its steals focus, big time. And not just once either. It can steal the focus for at least 4 times within 30 seconds!

If are thinking about using IntelliJ: don't do it!

If you are already hooked up: please vote for this bug!