Day to day stuff

X forwarding as root

2012-01-16T09:38:00.003+01:00

Often is useful to run jconsole on a remote (production) machine. One basically has 2 options to do so. First, you can pierce your firewall and let your application listen to the appropriate JMX and RMI ports. This however always tricky (in particular the RMI). In addition, creating the connection string that jconsole accepts is not nice at all.

The other option, is to do X forwarding. I'll describe here how to do this on MacOS X, with jconsole on any Unix server with, ssh-server, no root password and sudo installed.

Start X11 (you'll need to install it from the installation CDs).
Start iterm2.
Login to the target system with ssh -X remote_user@system
Switch to root with sudo -s.
Get X authorities with xauth merge ~remote_user/.Xauthority
Start jconsole

Breaking a Java HashSet

2011-12-13T12:03:00.004+01:00

Can it be?

Set set1 = new HashSet(5);
Set set2 = new HashSet(5);
// add of bunch of strings to both sets
assert set1.equals(set2) == false;

Yes it can!

We actually had this problem in an integration test. The cause was that the strings to one set were added concurrently. Interestingly, the sets seemed to be the same, when printed they contained the same strings, just in a different order (which is also interesting). However, the internals were apparently so damaged that an equals invocation return false.

We replaced the HashSet with a ConcurrentSkipSet, and all was fine again.

Conclusion: Okay, so this is a boring conclusion, but just be careful with using normal collections in concurrent situations.

DNS for Version 99 is off-line

2011-08-15T20:14:00.007+02:00

To alleviate some pain with using the commons-logging framework, I created version 99. It was hosted at the hostname no-commons-logging.zapto.org courtesy of no-ip.com. Unfortunately, due to lack of traffic I had to affirm usage of the the hostname once per month. Though this was slightly annoying, a complete pain is that I missed the last deadline, and that recreation is out as dashes in hostnames are no longer allowed.

As of 2011-08-15 16:11:00 GMT no-ip did credit to its name; no ip for no-commons-logging.zapto.org.

As 1) the current situation is disruptive for any version-99 user anyway, 2) there is a better workaround using the provided scope, I decided to not put back version 99 online under another hostname.

If you really need to use version 99 for some more time, just add an entry to your /etc/hosts file:

83.163.41.27 no-commons-logging.zapto.org

As promised, I'll keep version 99 for 5 years, that is until October 2012. Please check back here for IP address changes.

Lock-less singleton pattern

2011-06-16T12:26:00.004+02:00

Although the singleton pattern is now know as an anti-pattern, I think it still is a valid choice when you only need one instance in a particular context. Anyway, in Java there are several ways to implement a singleton.

For example this one uses a synchronized block:

private Singleton singleton = null;

protected Singleton createSingleton() {
  synchronized (this) {  // locking on 'this' for simplicity
    if (singleton == null) {
      singleton = new Singleton();
    }
    return singleton;
  }
}

To prevent locking all the time you can use the double-check idiom on a modern JVM. But I just thought about another implementation that switches locking for preventing code re-ordering, making it a lock-less implementation:

private AtomicReference singletonRef = new AtomicReference(null);

protected Singleton createSingleton() {
  Singleton singleton = singletonRef.get();
  if (singleton == null) {
    singleton = new Singleton();
    if (!singletonRef.weakCompareAndSet(null, singleton)) {
      singleton = singletonRef.get();
    }
  }
  return singleton;
}

There is only one catch to this implementation: although createSingleton() will always return the same instance, it must be allowed to call the constructor of Singleton twice without side-effects.

Update 2011-06-18: changed compareAndSet to weakCompareAndSet as suggested by Adriano.

Apache Wicket Cookbook — book review

2011-05-19T21:35:00.004+02:00

Some time ago I reviewed the drafts of the new book from Wicket rockstar programmer Igor Vaynberg: Apache Wicket Cookbook. If you are serious about using Wicket, this book is for you. It is fast, to the point, has very clear code samples and teaches you all the relevant (both clean and dirty) stuff you need and which Wicket in Action could not cover.

Conclusion: this is the book to read after 'Wicket in Action'.

Virtualizing a PC

2011-03-31T13:56:00.003+02:00

All hardware is eventually decommissioned. It may be broken, stolen, or just too old and slow. However, the software on it might still be needed. Not all software is easily transferable to a new PC with the latest OS on it. This quick guide helps you convert the old machine to a virtual machine, so that you can run it in VirtualBox. The process is called physical to virtual (P2V).

This article assumes you are at home in a Linux shell.

Boot the old hardware from a Ubuntu live CD or USB stick (or any other live cd, for example the gparted cd).
Prepare copying each file system with the following steps:
1. Mount the filesystem
2. Remove the contents of /tmp and trashcans in the home directories.
3. Clear free space with the following commands:
  sudo -s dd if=/dev/zero of=/usr/bigfile; rm -f /usr/bigfile
Now copy each physical disk to the target host computer (you could also do this per partition). With nc (netcat) and dd this is easy. Netcat essentially opens a pipe between two computers. We then use dd to stream the entire disk to and from the pipe. One of the two netcat's are in listening mode, the other connects to that one. It doesn't matter which one does the listening, except I found that netcat in MacOSX does not properly support listening. To speed up the transfer, gzip is used. Because we cleared empty sections of the disk, gzip is a quite efficient addition.
Here are the two examples of copying the disk /dev/sda. The first puts netcat in listen mode on the target, the second on the source system (execute in the given order):
on target host: nc -l 19001 | gzip -d -c | dd of=hd-copy.raw on source machine: dd if=/dev/sda | gzip -c --fast | nc 19001

on source machine: dd if=/dev/sda | gzip -c --fast | nc -l 19001 on target host: nc 19001 | gzip -d -c | dd of=hd-copy.raw

Make sure you have a decent network connection, 1 Gbit/s is fine.
The next step is to convert the raw image to a VirtualBox image. Do this with the following command (tested with VirtualBox 4.0.4):
VBoxManage convertfromraw hd-copy.raw hd-copy.vdi --variant Standard
If there really was a lot of empty space on the disk, you can compact it with the following command:
VBoxManage modifyhd hd-copy.vdi --compact

References

Chapter 8 of the VirtualBox manual
Another P2V technique which also works in VMWare.

On making a custom Ubuntu Bootable USB stick

2011-03-24T08:59:00.007+01:00

In my upcoming Wicket course the students will use a rental laptop. How do we guarantee that they will be up and running in no time? Colleague Jason had the solution: burn a Ubuntu Live CD to a USB stick!

Though simple this may sound, in the end it took us almost 3 days to put it together. This article gives an overview of the steps I took to create the USB sticks and how I tested them with VirtualBox.

Preparations

For starters you need a fast PC with about 15 GB free diskspace. I used Ubuntu 10.10, but other version probably work similarly. You'll need sudo rights as well. If your PC is somewhat older building the image might take quite some time.

Secondly, install the Ubuntu Customization Kit (package name uck). If you are on Ubuntu 10.10, make sure package gfxboot-dev is installed as well.

Finally you will need a clean Ubuntu iso image. Any Ubuntu, Kubuntu or Xubuntu image will do. This will be your starting point so choose carefully. I took ubuntu-10.10-desktop-i386.iso as I wanted to be sure it ran on older (32 bit) PCs as well.

Be prepared to repeat the whole customization process: log carefully what you do and keep a copy of all content you change or add.

Customizing the CD, first build

Uck puts everything in $HOME/tmp so I created that directory first. Then I simply executed uck-gui. The questions that are asked are quite straight-forward to answer. Don't select too much; to make the resulting image fit on a 1Gb USB stick you will need all the space you have. The last question makes you choose between running a packet manager, running a command shell, or continuing the build. Choose the command shell as it makes it a lot easier to keep track of your changes. Tip: use the history command to see what you did.

Removing packages

In the command shell you can remove packages and make other customizations. The Ubuntu wiki gives much information. The shell is chrooted to the new filesystem, so everything you do in that shell is restricted to the image you are creating. To be extra clear: if you remove a package in the UCK shell, you remove it from the live cd image and not from the rest of your system.

I purged lots of packages I didn't expect my students would need. Comments with more large packages that are probably not needed in a course are appreciated.

apt-get purge ubuntu-docs openoffice.org-core openoffice.org-java-common openoffice.org-common openoffice.org-writer openoffice.org-help-en-us openoffice.org-calc evolution evolution-common libevolution evolution-data-server evolution-webcal gbrainy gnome-games-common gwibber-service telepathy-gabble telepathy-gabble libtelepathy-glib0 python-telepathy empathy-common hplip-data hpijs hplip-cups hplip

Installing packages

Then I installed Java and cleaned up:

sudo add-apt-repository "deb http://archive.canonical.com/ lucid partner"
sudo apt-get update
sudo apt-get install sun-java6-jdk
apt-get clean

I also removed some example content:

rm -rf /usr/share/backgrounds/*
rm -rf /usr/share/example-content/Ubuntu_Free_Culture_Showcase

and placed my own background. See the ubuntu wiki link above for more details.

In /etc/skel I removed the link to the example contents and I added the following to .profile:

# Java
export JAVA_HOME=/usr/lib/jvm/java-6-sun

# Maven
export M2_HOME=/opt/maven3
export PATH=${PATH}:${M2_HOME}/bin
export MAVEN_OPTS="-Xmx512m -XX:MaxPermSize=512m"

The next step was to unzip recent Maven, Eclipse and IntelliJ distributions in /opt. The easiest way I found to do this is open a new shell, switch to root with sudo -s and start a nautilus. This gives you convenient access to $HOME/tmp/remaster-root. Make sure all placed files have root:root as owner.

Close the shell with exit (if another shell opens, close this one as well), and the Uck menu will appear again. Select Continue build from the Uck menu. And within a few minutes you have your first $HOME/tmp/remaster-new-files/livecd.iso!

Using the live cd

To test the live cd I installed VirtualBox (I installed 4.0.4 from the Oracle repository). I created a new machine with 8Gb of virtual harddisk space and mounted the new live cd iso file as virtual cd. That's it, just start the machine and play with it.

Once the virtual machine was fully started I selected 'Try Ubuntu'. I used Maven to make sure all required dependencies were sucked in to $HOME/.m2. Hint: I used the following commands in the largest example project:

mvn install
mvn eclipse:eclipse -DdownloadSources=true
mvn jettty:run

To ease starting Eclipse and IntelliJ I created a launcher on the desktop. Just right-click the desktop and select 'create launcher'. This will create two .desktop files on the desktop.

Tip: check the settings of the background image.

Now open a new Nautilus window from the 'Places' menu and open a sftp connection to the guest system (do Ctrl-L and enter something like sftp://host/home/user). This allows you to copy all changed and new content to the guest system for inclusion in the next build. I used this to copy the seeded maven repository and the two new launchers.

Satisfied I had a record of all my changes, I shutdown the virtual host, and applied the changes during a second run of uck-gui.

Making a bootable USB stick

After a few builds, I was confident the live cd was ready. The next step is to create a bootable USB stick. This is made easy with the 'Startup Disk Creator' you will find in the 'System/Administration' menu. If you want the USB stick to be useful without installing Ubuntu to the guest computer, make sure there is plenty of space for writing changes. You will probably need a 2GB stick, unless you remove many more packages a 1Gb stick does not leave enough space for any serious development.

Test the USB stick by booting your computer from it.

Testing the installation from the USB stick turned out to be much hard then expected. I didn't have a spare laptop, and VirtualBox's BIOS does not support booting from a USB device. The easiest way I found to circumvent this was to create a vdi image from the stick in the following way:

sudo dd if=/dev/sdb of=raw-usb-image.bin
sudo chown user:user raw-usb-image.bin
VBoxManage convertfromraw raw-usb-image.bin usb-image.vdi

You can now mount the vdi file to your virtual machine (make sure its the first drive) and boot from it.

Cloning the USB

Once you have created the USB stick, you can clone it easily with the dd command.

Be careful! Selecting the wrong device may kill your harddisk! You have been warned.

# Copy the USB to an image on disk
sudo dd if=/dev/sdb of=raw-usb-image.bin
# Write it out the another USB image
sudo dd if=raw-usb-image.bin of=/dev/sdc

A USB 2.0 hub with many ports helps because the throughput of the USB channel is much higher then the write speed of the sticks. For me, writing 8 USB sticks took the same time as a single stick.

USB cloning in action

Update 2011-03-27 One problem with cloning disks like this is that they all get the same UUID. In some situations this may give problems. I have yet to find a solution for this. I did not find a tools for just changing the UUID of a FAT file system.

Conclusions

With the help of the Ubuntu Customization Kit, creating a customized Ubuntu startup USB stick becomes a surprisingly straight forward -though lengthy- process. Make sure you can repeat your steps as you probably need to. I found VirtualBox to be an excellent tool to test both the Live CD and the bootable USB sticks.

Why Functional Programming Matters - An exploration of functional Scala - Part 2

2010-10-05T20:56:00.005+02:00

package nl.grons.whyfp
import scala.math.abs
/**

A translation of Miranda examples in chapter 4 of the paper Why Functional Programming Matters by John Hughes, 1984.

This article is the continuation of part 1 - Chapter 3 examples. I will show you lazy evaluation and how this can be useful.

@author Erik van Oosten
*/

object Chapter4_LazyList {

Chapter 4’s examples make heavy use of Miranda’s lazy evaluation. In particular, it uses the list as defined in chapter 3 in a lazy way. This allows the creation of lists of unbounded length. Scala supports lazy lists as well. You can find an implementation in the Stream class. However, as I want to stay close to the original, I redefine the list from chapter 3 as follows:

/** The list from chapter 3, redefined to make it a lazy list. */
sealed abstract class List[+A] {
  def head: A
  def tail: List[A]
}

/** Lazyly evaluate the argument of the constructor AND the value of tail. */
final class Cons[A](hd: A, tl: => List[A]) extends List[A] {
  override val head: A = hd
  override lazy val tail: List[A] = tl
}

If you compare this to the List implementation in the previous article, you note that parameter ‘tl’ of ‘Cons’ is passed as an by-name parameter (because of the =>). This means that the given expression is only evaluated when the parameter is used. Too prevent the tail expression to be evaluated again and again, the ‘tail’ value is declared as a val. Now ‘tl’ is evaluated exactly once. It is also declared as lazy so that the evaluation of the expression is delayed until the first time it is needed instead of during construction time.

These 3 things together allow the list to behave lazily.

You might also notice that ‘Cons’ is no longer a case class (not possible with by-name parameters). Therefore I define the following:

object Cons {
  def apply[A](hd: A, tl: => List[A]) = new Cons(hd, tl)
  // Warning: using unapply leads to evaluation of the tail.
  def unapply[A](c: Cons[A]) = Some(c.head, c.tail)
}

‘Nil’ did not change:

/** The empty list. */
case object Nil extends List[Nothing] {
  override def head: Nothing = {
    throw new NoSuchElementException("head of empty list")
  }
  override def tail: List[Nothing] = {
    throw new NoSuchElementException("tail of empty list")
  }
}

Just for fun: here is an example of how the list can be used to construct an infinite list.

/** A list of all integer numbers starting at x and higher. */
// Miranda: nat x = cons x nat(x+1)
def nat(x: Int): List[Int] = Cons(x, nat(x + 1))

The Miranda version of ‘nat’ is limited by the available memory, the Scala version is limited to scala.Int.MaxValue.

Function ‘map’ from the previous chapter won’t work. It will try to completely iterate the list which will obviously fail for unbounded lists. Lets implement it again with some simple pattern matching:

/** Create a new lazy list where each element a is replaced by f(a). */
def map_wrong[A, B](f: A => B)(l: => List[A]): List[B] = l match {
  case Nil => Nil
  case Cons(head, tail) => Cons(f(head), map(f)(tail))
}

This implementation seemed to work for quite some time, even though everything was slow. If you look carefully, you will see that it evaluates ‘tail’ in ‘Cons.unapply’ which is invoked to match the pattern Cons(head, tail). The result is that exactly one more item in the list is evaluated then necessary. Though this is merely wasteful for most algorithms, it is catastrophic (as in stack overflow) when that evaluation contains another recursive call to ‘map’. This actually happens in the last example of this article.

Here is an implementation that does not have this problem:

/** Create a new lazy list where each element a is replaced by f(a). */
def map[A, B](f: A => B)(l: => List[A]): List[B] = l match {
  case Nil => Nil
  case c: Cons[A] => Cons(f(c.head), map(f)(c.tail))
}

Notice that the recursive expression, which includes c.tail, is now used in a place that is lazily evaluated.

On the console:

scala> import nl.grons.whyfp.Chapter4_LazyList._

scala> def printFirst[A](count: Int, l: List[A]) {
  if (count > 0 && l != Nil) { println(l.head); printFirst(count - 1, l.tail) } }
printFirst: [A](count: Int,l: nl.grons.whyfp.Chapter4_LazyList.List[A])Unit

scala> printFirst(3, map[Int,Int](_ * 2)(nat(3)))
6
8
10

*/
}

object Chapter4_1_NewtonRaphsonSquareRoots {

import Chapter4_LazyList._
/*

Here’s an imperative Scala implementation that does a numerical calculation of the square root of a number.

/** Approximate the square root of n to within eps
 * starting with approximation a0. */
def imperative_squareroot(a0: Double, eps: Double, n: Double): Double = {
  // Warning: imperative scala!
  var x = a0
  // The initial value of y does not matter so long as abs(x-y) > eps
  var y = a0 + 2 * eps
  while (abs(x - y) > eps) {
    y = x
    x = (x + n/x) / 2
  }
  x
}

Here are my translations of the rest:

/** Next approximation of the square root of n,
 * based on the previous approximation x. */
// Miranda: next N x = (x + N/x) / 2
def next(n: Double)(x: Double): Double = (x + n/x) / 2

/** Produce an infinite list that starts with a and where all other values
 * are the result of applying f to the previous value. */
// Miranda: repeat f a = cons a (repeat f (f a))
def repeat[A](f: A => A, a: A): List[A] = Cons(a, repeat(f, f(a)))

/** Give the first value of the list that is within eps of its preceding value. */
// Miranda:
// within eps (cons a (cons b rest)) =
//         = b,                        if abs(a-b) <=  eps
//         = within eps (cons b rest), otherwise
def within(eps: Double, l: List[Double]): Double = {
  val a = l.head
  if (a.isNaN) throw new RuntimeException("nan")
  val rest = l.tail
  val b = rest.head
  if(abs(a - b) <= eps) b else within(eps, rest)
}

When a calculation overflow or underflow occurs (not unlikely when working with approximations), ‘within’ goes amok and will loop forever. This is prevented by validating the head of the list against NaN, Not A Number.

/** Approximate the square root of n to within eps
 * starting with approximation a0. */
// Miranda: sqrt a0 eps N = within eps (repeat (next N) a0)
def sqrt(a0: Double, eps: Double, n: Double) = within(eps, repeat(next(n), a0))

/** Gives the first value of the list that of which the ratio of change
 * with its preceding value is lower then eps. */
// Miranda:
// relative eps (cons a (cons b rest)) =
//         = b,                          if abs(a-b) <= eps*abs b
//         = relative eps (cons b rest), otherwise
def relative(eps: Double, l: List[Double]): Double = {
  val a = l.head
  if (a.isNaN) throw new RuntimeException("nan")
  val rest = l.tail
  val b = rest.head
  if(abs(a - b) <= eps * abs(b)) b else relative(eps, rest)
}

/** Approximate the square root of n to within ratio eps starting with
 * approximation a0. */
// Miranda: relativesqrt a0 eps N = relative eps (repeat (next N) a0)
def relativesqrt(a0: Double, eps: Double, n: Double) =
  relative(eps, repeat(next(n), a0))

On the Console:

scala> import nl.grons.whyfp.Chapter4_LazyList._
scala> import nl.grons.whyfp.Chapter4_1_NewtonRaphsonSquareRoots._

scala> relativesqrt(0.1, 0.01, 4)
res0: Double = 2.000010925778043

*/
}

object Chapter4_2_NumericalDifferentiation {

import Chapter4_LazyList._
import Chapter4_1_NewtonRaphsonSquareRoots._
/*

This chapter is about numerical differentiation.

/** An approximation of the differentiation of f for x over h. */
// Miranda: easydiff f x h = (f(x+h)-f x) / h
def easydiff(f: Double => Double, x: Double)(h: Double): Double =
  (f(x + h) - f(x)) / h

// Miranda:
// differentiate h0 f x = map (easydiff f x)(repeat halve h0)
// halve x = x/2
def differentiate(h0: Double, f: Double => Double, x: Double) =
  map(easydiff(f, x))(repeat(halve, h0))
def halve(x: Double) = x / 2

On the console:

scala> import nl.grons.whyfp.Chapter4_LazyList._
scala> import nl.grons.whyfp.Chapter4_1_NewtonRaphsonSquareRoots._
scala> import nl.grons.whyfp.Chapter4_2_NumericalDifferentiation._

scala> val f: Double => Double = math.pow(_, 3.0)
f: (Double) => Double = <function1>

scala> within(0.001, differentiate(1, f, 2))
res0: Double = 12.000732436776161

Some more examples:

// Miranda:
// elimerror n (cons a (cons b rest)) =
//        = cons ((b*(2**n)-a)/(2**n-1)) (elimerror n (cons b rest))
def elimerror(n: Double, l: List[Double]): List[Double] = {
  val a = l.head
  val bandrest = l.tail
  val b = bandrest.head
  Cons(
    (b * math.pow(2.0, n) - a) / (math.pow(2.0, n) - 1.0),
    elimerror(n, bandrest))
}

/** Calculate the order of n to be used in elimerror. */
// Miranda:
// order (cons a (cons b (cons c rest))) =
//         = round(log2( (a-c)/(b-c) - 1 ))
// round x = x rounded to the nearest integer
// log2 x = the logarithm of x to the base 2
def order(l: List[Double]): Double = {
  val LOG2: Double = math.log(2.0)
  def log2(x: Double) = math.log(x) / LOG2

  val a = l.head
  val b = l.tail.head
  val c = l.tail.tail.head
  val o = math.round(log2((a-c)/(b-c) - 1))
  if (o.isNaN || o == 0) 1 else o
}

// Miranda: improve s = elimerror (order s) s
def improve(s: List[Double]): List[Double] = elimerror(order(s), s)

Note that my implementation of ‘order’ is not only real (no vague descriptions of ‘round’ and ‘log2’), but it is more robust then the original. First, it protects against NaN for the case that b-c reaches 0. Secondly, it prevents a result of 0 as that leads to a division by 0 in ‘elimerror’.

On the console:

scala> within(0.001, improve(differentiate(1, f, 2)))
res0: Double = 11.9998779296875

scala> // Improve can be improved recursively

scala> within(0.001, improve(improve(improve(differentiate(1, f, 2)))))
res1: Double = 12.0

// Miranda:
// super s = map second (repeat improve s)
// second (cons a (cons b rest)) = b
def superimprove(s: List[Double]): List[Double] = map(second)(repeat(improve, s))
def second(l: List[Double]): Double = l.tail.head

def superdifferentiate(f: Double => Double, x: Double): Double = {
  val eps: Double = 0.00000001
  within(eps, superimprove(differentiate(x + 2 * eps, f, x)))
}

}

object Chapter4_3_NumericalIntegration {

import Chapter4_LazyList._
/*

This chapter is about numerical integration.

// Miranda: easyintegrate f a b = (f a + f b)*(b-a)/2
def easyintegrate(f: Double => Double, a: Double, b: Double) =
  (f(a) + f(b)) * (b-a) / 2

// Miranda: integrate f a b = cons (easyintegrate f a b)
//                                 (map addpair (zip (integrate f a mid)
//                                                   (integrate f mid b)))
//                  where mid = (a+b)/2
def integrate_simple(f: Double => Double, a: Double, b: Double): List[Double] = {
  def addpair(pair: (Double, Double)) = pair._1 + pair._2
  val mid = (a + b) / 2
  Cons(
    easyintegrate(f, a, b),
    map(addpair)(zip(integrate(f, a, mid), integrate(f, mid, b)))
  )
}

/** Convert two lists to a list of pairs. */
// Miranda: zip (cons a s) (cons b t) = cons (pair a b) (zip s t)
def zip[A,B](as: => List[A], bs: => List[B]): List[(A,B)] = as match {
  case Nil => Nil
  case a: Cons[A] => bs match {
    case Nil => Nil
    case b: Cons[B] => Cons((a.head, b.head), zip(a.tail, b.tail))
  }
}

Again note that ‘zip’ was defined such that a.tail and b.tail are defined as a lazily evaluated argument.

// Miranda:
// integrate f a b = integ f a b (f a) (f b)
// integ f a b fa fb = cons ((fa+fb)*(b-a)/2)
//                          (map addpair (zip (integ f a m fa fm)
//                                            (integ f m b fm fb)))
//         where m = (a+b)/2
//               fm = f m
def integrate(f: Double => Double, a: Double, b: Double): List[Double] = {
  def integ(a: Double, b: Double, fa: Double, fb: Double): List[Double] = {
    def addpair(pair: (Double, Double)) = pair._1 + pair._2

    val mid = (a + b) / 2
    val fm = f(mid)
    Cons(
      (fa + fb) * (b-a) / 2,
      map(addpair)(zip(integ(a, mid, fa, fm), integ(mid, b, fm, fb)))
    )
  }

  integ(a, b, f(a), f(b))
}

And of course, seeing is believing:

scala> import nl.grons.whyfp.Chapter4_LazyList._
scala> import nl.grons.whyfp.Chapter4_1_NewtonRaphsonSquareRoots._
scala> import nl.grons.whyfp.Chapter4_2_NumericalDifferentiation._
scala> import nl.grons.whyfp.Chapter4_3_NumericalIntegration._

scala> def f(x: Double): Double = 1/(1+x*x)
f: (x: Double)Double

scala> within(0.001, integrate(f, 1, 2))
res0: Double = 0.3218612363325556

scala> relative(0.001, integrate(f, 1, 2))
res1: Double = 0.3217782239721789

scala> relative(0.001, superimprove(integrate(f, 1, 2)))
res2: Double = 0.3217525935931843

*/
}
/*

Conclusion

Scala is an excellent language for functional style programs and I really like that static typing prevents tons of errors. My only problem with Scala — in terms of the ‘Why Functional Programming matters’ paper — is that lazy evaluation is not the natural way of working. The way expressions are evaluation is more closely to the Java world. To make use of lazy evaluation you have to be very much aware of what your doing and carefully make use of the lazy keyword and by-name parameters. With Miranda this comes naturally though probably at the cost of some performance for the case laziness is not needed. Perhaps that the standard collections library (scala.collection.immutable.Stream) hides this sufficiently and makes life easy.

A more fundamental way to fix this problem would be to express the laziness nature of an expression in its type. I am out of the academic world for a looong time, so I have no idea if this is being researched at all. It will take some hard thoughts to make this practical and simple to use though.

The next challenge would be to repeat this experiment with the standard collections. But I’ll leave that to someone with more time...

Why Functional Programming Matters - An exploration of functional Scala.

2010-09-21T22:04:00.012+02:00

package nl.grons.whyfp
/**

A translation of Miranda examples in the paper Why Functional Programming Matters by John Hughes, 1984.

@author Erik van Oosten
*/

object WhyFp_Abstract {
/*

Scala is in the focus of attention because it allows a smooth migration from one of the most popular programming languages ever to a language that supports (among other things) a functional style of programming.

In this article I explore the functional aspects of Scala based on a very influential paper on functional programming: ‘Why Functional Programming matters’. To do so I will translate the Miranda examples of chapter 3 and 4 (next article) to Scala. To keep things pure, the standard Scala libraries are avoided.

I will conclude that Scala has sufficient features to live up to Hughes’ standards with the added bonus of being statically type safe.

Just like the paper I just assume you know a bit about functions, partial functions and partially applied functions. In addition I assume you know basic Scala syntax. However, as I am still a Scala newby myself, any comments that will improve this article are welcomed.

This article focusses on the Miranda -> Scala translation and will not repeat the message of the paper. So go ahead, and read the paper first.

All text in this article is properly commented, so the whole article can be copy pasted into your favourite text-editor.

*/
object Chapter3_GlueingFunctionsTogether {
import RichFunction2._
/*

The original article completely defines a list right there in just one line:

listof X ::= nil | cons X (listof X) (Miranda)

Scala is not so powerful and this mainly because we need to translate Miranda’s ‘|’ operator to a class hierarchy.

Lets define List more or less as the Scala library does it (as anonymous points out, it can be done much shorter).

/** The base type for our list. */
sealed abstract class List[+A] {
  def head: A
  def tail: List[A]
}

/** Cons represents an element and the rest of the list. */
case class Cons[A](hd: A, tl: List[A]) extends List[A] {
  override def head: A = hd
  override def tail: List[A] = tl
  override def toString = "Cons(" + hd + "," + tl + ")"
}

/** Nil represents the empty list. */
case object Nil extends List[Nothing] {
  override def head: Nothing = {
    throw new NoSuchElementException("head of empty list")
  }
  override def tail: List[Nothing] = {
    throw new NoSuchElementException("tail of empty list")
  }
  override def toString = "Nil"
}

Note that Cons is a case class, therefore we can use Cons(x,y) as synonym for the constructor and also as extractor during pattern matching.

We can now use the list as follows:

[]         => Nil
[1]        => Cons(1, Nil)
[1,2,3]    => Cons(1, Cons(2, Cons(3, Nil)))

As in the original article, what follows is a number of definitions of ‘sum’. I name each definition differently so that this whole article can be compiled as is.

My first approach mimics the Miranda approach with 2 partial functions. Scala does not automatically combine partial functions se we’ll need to do ourselves.

// Miranda:
// sum nil = 0
// sum (cons num list) = num + sum list
//
val sum_nil: PartialFunction[List[Int], Int] = { case Nil => 0 }
val sum_cons: PartialFunction[List[Int], Int] =
       { case Cons(num, list) => num + sum_combinedpartials(list) }
def sum_combinedpartials = sum_nil orElse sum_cons

The second, more idiomatic Scala, version wraps this into a single method:

def sum_idiomatic(li: List[Int]): Int = li match {
  case Nil => 0
  case Cons(num, list) => num + sum_idiomatic(list)
}

Note that both approaches use pattern matching, just like the Miranda original.

The following definitions of ‘sum’ are more interesting. They are very close to the original example. Note that this actually defines an immutable reference (a val) to a function. In case you are wondering, List[Int] => Int is the type of the function: it accepts one argument, List[Int], and has Int as return type.

// Miranda: sum = reduce add 0
val sum: List[Int] => Int = reduce(add, 0)

Note that nowadays most languages instead of ‘reduce’, use the term ‘foldr’ (fold-right).

Scala has type inference, so if we let the compiler derive the return type, its even closer. Note that we need to tuck an underscore to the end to let the compiler know we really want a partially applied function.

val sum_noTypes = reduce(add, 0) _

And of course we need a definition for ‘add’:

// Miranda: add x y = x + y
def add(x: Int, y: Int): Int = x + y

For function ‘reduce’, I again use pattern matching just like the Miranda original. Notice that ‘reduce’ defines 2 parameter lists. One with parameters f and x, and the second with parameter l. This allows for easy creation of partially applied functions. And this is exactly what happened in the definitions of ‘sum’ and ‘sum_noTypes’ above.

// Miranda
// (reduce f x) nil = x
// (reduce f x) (cons a l) = f a ((reduce f x) l)
//
/**
 * Right fold a list with a function f and a constant x.
 *
 * @param f a function to apply
 * @param x the result for an empty list
 * @param l the list to fold
 * @param [A] the type of list elements
 * @param [B] the result type
 * @return the result of the expression where in l,
 *   each Cons is replaced by f, and Nil by x
 */
def reduce[A, B](f: (A, B) => B, x: B)(l: List[A]): B = l match {
  case Nil => x
  case Cons(head, tail) => f(head, reduce(f, x)(tail))
}

More examples:

// Miranda: product = reduce multiply 1
def product: List[Int] => Int = reduce(multiply, 1)

Just for fun I define ‘multiply’ a little bit different then ‘add’. Where ‘add’ is a method (which can be used as a function), ‘multiply’ is directly defined as a function. Each underscore declares an anonymous parameter. To avoid the ugly (_: Int) syntax, the type of these anonymous parameters are derived form the return type of ‘multiply’.

val multiply: (Int, Int) => Int = _ * _

We can also inline these functions. If you do so the compiler still needs some hints to infer the types of the anonymous parameters. In the following definitions of ‘anytrue’ and ‘alltrue’ I solve that by adding the generic types of ‘reduce’ (the [Boolean,Boolean] part).

With ‘alltrue’ the return type is inferred by the compiler (again, this requires an underscore to confirm we want partial application of ‘reduce’). In the definition of ‘alltrue’, the return type is defined and therefore the underscore is not needed.

Personally I prefer the explicit return type as the function is too complex to just see what it should be. My rule is omit the return type only when it is blindingly obvious (as in ‘add’ above).

// Miranda
// anytrue = reduce or false
// alltrue = reduce and true
//
def anytrue: List[Boolean] => Boolean = reduce[Boolean, Boolean](_ || _, false)
def alltrue                           = reduce[Boolean, Boolean](_ && _, true) _

Lets also explore how ‘reduce’ works and calculate the sum of [1,2,3]. Note how we replace Cons with the given function ‘add’ and Nil with the given constant ‘0’.

Starting point:       Cons(1,     Cons(2,     Cons(3, Nil)))
Reduce applied:        add(1,      add(2,      add(3,  0 ))) == 6
And for multiply: multiply(1, multiply(2, multiply(3,  1 ))) == 6

Here is my translation of ‘append’.

// Miranda: append a b = reduce cons b a
def append[A](a: List[A], b: List[A]): List[A] = reduce[A, List[A]](Cons[A], b)(a)

Note that the first parameter we pass to ‘reduce’ is actually the Cons object (with generic type parameter [A]). This is possible because anything with an ‘apply’ method is recognized as a Scala function.

The generic type ‘[A]’ on Cons is essential; without it the compiler generates nothing but misleading error messages. It took me many hours to figure this one out.

If you are new to this complex kind of type declarations (like I am), you’ll find that careful reading is in order. I am afraid this is just something you need to practice. Also, it helps greatly when you wrote this kind of stuff yourself.

Lets skip ahead to ‘doubleall’.

// Miranda:
//        doubleall = reduce doubleandcons nil
// where  doubleandcons num list = cons (2*num) list
def doubleall_1: List[Int] => List[Int] = {
  def doubleandcons(num: Int, list: List[Int]) = Cons(2 * num, list)
  reduce(doubleandcons, Nil)
}

As you can see Miranda’s where keyword nicely translates to nested methods.

Next version:

// Miranda:
//        doubleandcons = fandcons double
// where  double n = 2*n
//        fandcons f el list = cons (f el) list
def doubleandcons: (Int, List[Int]) => List[Int] = {
  def double(n: Int) = 2 * n
  def fandcons[A, B](f: A => B)(el: A, list: List[B]) = Cons(f(el), list)
  fandcons(double)
}

So far, I had little problem with translating this to Scala. However, the next line needed quite a bit more thought:

fandcons f = cons . f

Scala (Miranda as well) only defines the ‘compose’ method on functions of arity 1. For Miranda this is no problem as all functions written down with multiple arguments are actually syntactic sugar for functions with a single argument that produce further functions with 1 argument. So the type of ‘add’ in

add x y = x + y

is not (Int, Int) => Int as it would be in Scala, but Int => Int => Int.

In our Scala code Cons.apply has 2 arguments and thus arity 2. To be precise, the type of Cons[B] _ is (B, List[B]) => List[B]. (Note in Scala we write Cons[B] _ to indicate we need it as a partially applied function).

In order to apply ‘compose’ we first need to convert Cons[B].apply to something of type B => List[B] => List[B] (a unary function that accepts a B, and gives another unary function that accepts a List[B], and gives a List[B]). This is called currying and is performed with the ‘curried’ function.

The result of currying is a unary function that we can compose with ‘f’. The next step is to uncurry back to a function of arity 2. This is done with ‘uncurried’. Not lets put this in code:

// Miranda: fandcons f = cons . f
def fandcons[A, B](f: A => B): (A, List[B]) => List[B] =
       Function.uncurried( (Cons[B] _).curried.compose(f) )

However, this is very a unwieldy syntax and I am going to need it a lot in the upcoming examples. Therefore, I’ll pimp the Scala library to include ‘compose’ also on functions of arity 2 (see RichFunction2 below).
We can now condense the code to:

// Miranda: fandcons f = cons . f
def fandcons_short[A, B](f: A => B): (A, List[B]) => List[B] =
       (Cons[B] _).compose(f)

And then we try it out with some arguments to see it is correct:

fandcons(f, el)(list) = Cons.compose(f)(el)(list)
                      = Cons(f(el), list)

And finally (no longer defining ‘double’ as a nested function):

// Miranda: doubleall = reduce (cons . double) nil
def double(n: Int) = 2 * n
def doubleall: List[Int] => List[Int] =
       reduce((Cons[Int] _).compose(double), Nil) _

The next step is to extract ‘map’:

// Miranda:
// doubleall = map double
// map f = reduce (cons . f) nil
def doubleall_map: List[Int] => List[Int] = map(double)
def map[A, B](f: A => B): List[A] => List[B] = 
      reduce((Cons[B] _).compose(f), Nil) _

Map could for example be used to calculate the sum in a matrix, a list of list of int.

// Miranda: summatrix = sum . map sum
val summatrix: List[List[Int]] => Int = sum.compose(map(sum))

Here is a fragment of the scala console in which I try out summatrix:

scala> import nl.grons.whyfp.Chapter3_GlueingFunctionsTogether._
scala> val a: List[List[Int]] = Cons(Nil, Cons(Cons(1, Cons(2, Nil)), Cons(Cons(4, Nil), Nil)))
a: nl.grons.whyfp.Chapter3_GlueingFunctionsTogether.List[nl.grons.whyfp.Chapter3_GlueingFunctionsTogether.List[Int]] = Cons(Nil,Cons(Cons(1,Cons(2,Nil)),Cons(Cons(4,Nil),Nil)))

scala> summatrix(a)
res0: Int = 7

Lets move on and create a tree.

//Miranda: treeof X ::= node X (listof (treeof X))
sealed abstract class Tree[+A] {
  def label: A
  def children: List[Tree[A]]
}

case class Node[A](l: A, c: List[Tree[A]]) extends Tree[A] {
  override def label: A = l
  override def children: List[Tree[A]] = c
}

The tree 1 o / \ / \ 2 o o 3 | | o 4 in Scala would look like:

Node(1,
     Cons( Node(2, Nil),
           Cons(Node(3,
                Cons(Node(4,
                     Nil),
                Nil)),
           Nil)))

Here is my version of ‘redtree’:

// Miranda:
// redtree f g a (node label subtrees) =
//         f label (redtree' f g a subtrees)
// redtree' f g a (cons subtree rest) =
//         g (redtree f g a subtree) (redtree' f g a rest)
// redtree' f g a nil = a
/**
 * Reduce a tree with functions f and g and constant a.
 * @param f a function to apply to a node label and
 *   the result of g on the node's subtree
 * @param g a function to apply to a subtree and
 *   the result of g on the following sibling subtrees
 * @param a the result for an empty tree
 * @param tree the tree to reduce
 * @param [F] the return type of f and redtree itself
 * @param [G] the return type of g
 * @param [A] the type of the tree labels
 * @return the result of the expression where in tree,
 *   each Node is replaced by f, each Cons by g, and Nil by x
 */
def redtree[F, G, A](f: (A, G) => F, g: (F, G) => G, a: G)(tree: Tree[A]): F = {
  def redchildren(children: List[Tree[A]]): G = children match {
    case Cons(subtree, rest) => g(redtree(f, g, a)(subtree), redchildren(rest))
    case Nil => a
  }

  f(tree.label, redchildren(tree.children))
}

Notice how Scala’s nested functions are more elegant then Miranda’s top level function ‘redtree' ’ (renamed to ‘redchildren’ here); in Scala the arguments of ‘redtree’ are not passed to ‘redchildren’ as the nested function can access them directly.

Lets use redtree to sum all integer labels in a tree of type Tree[Int]:

// Miranda: sumtree = add add 0
val sumtree: Tree[Int] => Int = redtree(add, add, 0)

Next: collecting all the labels in a list:

// Miranda: labels = redtree cons append nil
def labels[A]: Tree[A] => List[A] =
     redtree(Cons[A], append[A], Nil)

And the last example: create a new tree by applying a function to each label:

// Miranda: maptree f = redtree (node . f) cons nil
def maptree[A, B](f: A => B): Tree[A] => Tree[B] =
     redtree((Node[B] _).compose(f), Cons[Tree[B]], Nil)

} */

Parting thoughts

Although it took quite some time to write the code and then translate this to my 100th article, I had great fun doing so. In addition, I learned quite a bit on writing more complex Scala code.

The code for chapter 4 has already been translated, so expect another article soon.

P.S. RichFunction2

As was written above, Scala does not define ‘compose’ on Function2 (the class representing functions with 2 parameters), only on Function1 ‘compose’ is available. To remedy this, we’ll define it ourselves with the pimp-my-library pattern.

class RichFunction2[T1, T2, R](f: Function2[T1, T2, R]) {
 /**
  * (f compose g)(x, y) == f(g(x), y)
  */
 def compose[A](g: (A) => T1): (A, T2) => R = {
   (x: A, y: T2) => f(g(x), y)
 }
}
object RichFunction2 {
  // Note: complete return type is required here because the method
  // is used before it is declared.
  implicit def richFunction2[T1, T2, R](f: Function2[T1, T2, R]):
       RichFunction2[T1, T2, R] = new RichFunction2(f)
}

Open source - why bother with anything else?

2010-08-12T14:02:00.001+02:00

Since I work in a fine small company where we are breathing open source for at least a decade, it is sometimes weird to be confronted again by open source adversaries or agonists. For example, a colleague wrote a fine technical design based on Mule. All of a sudden we're asked to compare this to BEA AquaLogic and see whether we could implement the project with that. Now this is probably possible, and AquaLogic is probably a fine product family, but why bother?

Since there was already a Mule prototype, I found it a cumbersome idea. To get up to speed with AquaLogic you first have to find out what products in the AquaLogic family you need, and of course you only need a tiny bit of most of them. Secondly you have to go into a trajectory to get the software, including development licenses. Somehow money is not always a problem, but the time to get the products and start working always is, and the deadline won't move. Did I already mention I really hate bureaucracy?

Starting with open source often just takes 5 lines in a pom.xml and a few minutes of download time.

Okay, well suppose this was all taken care of and you are happily underway with development. You then run into a problem. Yes, you will, this is no different from open source. Now lets see how I typically deal with problems with using open source and see how this applies to closed source software.

Finding a solution to a problem with an open source product
1) (Re-)read the documentation
The first step is always to read the documentation. With the source jars attached in your favorite IDE, the javadoc is one key-press away.

2) Debugging
In step 2 we'll do some debugging. Again, with the source jars attached, tracing through your own code is as easy as tracing the open source code. I may not understand all the code, but I know what I need and I can read the JavaDoc of code encountered underway.
More often then not tracing leads to a deeper understanding of the used products, and I frequently find things that are useful for other parts of the project. The deeper understanding helps to form a solution. This can be changing your code, or patching the open source product. Otherwise it helps you formulate a more precise problem statement for the next steps.

3) Read more documentation
As I have now seen the code, I can search the available documentation more efficiently. So I do this first before going into the next step. Documentation in this phase can be anything, from manuals to blogs and forums.

4) Post questions
The last step is to post a question on a forum or mailing list. If you get this far, you are either lazy, you just missed something, or you found out that the library has a bug. The results of this step are not always satisfactory; it really depends on the community around the product. At least you can always change the open source product yourself.

Finding a solution to a problem with a closed source product

1) (Re-)read the documentation
Again, the first step is to read the documentation. However, javadoc is not always available. Rarely is it available in the form of a source jar.

2) Debugging
Oops, we can only trace our own code. Perhaps you can use de-compilation. Note that this might actually be illegal in your country (not in The Netherlands luckily). Secondly, the source might be obfuscated. And of course, I am not even talking about products that run as a complete separate program.

3) Read more documentation
As it is unlikely that debugging gave us more insight, we skip this step.

4) Post questions
In rare cases there are user communities around closed source. These are very valuable! However, usually you just have to ask the manufacturer. By lack of insight, the question won't be as detailed as with open source. And then we rely on the manufacturer. Some react quick and accurate, some don't even give you the the ability to ask questions. More often then not it costs lots of money and time. And of course meanwhile you will have to code workarounds yourself.

Conclusions

Despite the title I am not at all against closed and commercial software. Its just that as a programmer I find it hardly ever worth the bother.

Another Wicket course coming up

2010-04-01T10:29:00.004+02:00

I'll be doing the public Wicket introduction course from jweekend again on May 27 and 28. For more information see the link above or drop me a note.

Wicket root mounts

2010-04-01T10:14:00.003+02:00

Just in case you missed it: I recently wrote an article that shows how to mount pages on the root in Wicket. As a lot of time went into creation the example application I posted the article on my employers blog.

Backup home directory to USB harddisk

2009-10-30T19:21:00.006+01:00

As I am keen to install Ubuntu 9.10 on my work laptop, its time to do an extra backup of my home directory. My mp3 player has about 30 Gb free so I'll use that.

First attempt:

cp -R /home/erik /media/H300/Backup/

You wont believe how slow this is! All those little pesky files, we'll need to aggregate them.

Second attempt:

tar -cjf /media/H300/Backup/erik.tar.bz2 /home/erik

Waiting ... waiting ... Oops, that fails! The FAT file system on the mp3 player only supports files up to 2Gb.

After a long but fruitless search in the documentation of tar (and zip) for multi-volume archives options, I suddenly thought of the simplest thing that could work:

tar -cj /home/erik | \
split -d -b 1G - /media/H300/Backup/erik.tar.bz2.part

This creates a bunch of 1Gbyte files ending in part00, part01, etc.

You've got to love unix!

Wicket do's and dont's

2009-09-16T09:57:00.002+02:00

Just published an article on my employer's blog: Wicket do's and dont's.

Extending my e-mail stack with Roundcube

2009-06-12T21:18:00.007+02:00

I don't trust anyone with my most precious data: e-mail. That is why I run my own e-mail server. The server runs Ubuntu, Postfix, Dovecot and several tools for spam interception. I access my e-mail from several machines through the IMAP protocol (with TLS). Though any good IMAP client would do it is always Thunderbird (yes, even on my Mac).

It is however not always feasible to have Thunderbird around. Time to add an IMAP web client! Candidates are IMP, Squirrelmail and Roundcube. All of these projects continuously release security updates. So unless you go for the next (beta) Ubuntu release, the packaged version is almost always a few versions behind.

In the past I have already used IMP and Squirrelmail. IMP has a nice UI but was difficult to install. Squirrelmail looks really old, but has a huge amount of nice plugins.

So it became Roundcube this time. Roundcube is not out of beta for that long. But it does look really slick, almost as if it is a desktop app. I did go with the Ubuntu package after all. Lacking more documentation I had to figure out for myself that I needed to use /etc/roundcube/apache.conf as the basis for a virtual host file in the /etc/apache2/sites-available directory. After some smaller configuration tweaks in /etc/roundcube/main.inc.php I had the application as I wanted it.

One more thing: Roundcube (like IMP and Squirrelmail) continuously open and close new IMAP connections. This makes the site amazingly slow. The fix: package imapproxy. This package starts an IMAP server that caches IMAP connections to another IMAP server. Some small config changes, and Roundcube was a lot quicker.

More Wicket filter options

2009-05-25T10:54:00.007+02:00

Wicket has this very clever idea to serve requests from a servlet Filter instead of a Servlet. The brilliance of it is that you can serve pages on the root of your context, but still allow the servlet container to process requests that Wicket has nothing to do with.

By default this works correct automatically. Incoming requests that are not recognized by Wicket are just passed through.

However, there is an optimization that is not very well documented. When you already know that certain paths are never going to be handled by Wicket (for example because you configured a servlet for them), you can tell the Wicket filter to ignore those paths in the same file as where you define the servlets: in the web.xml.

Example web.xml configuration:

<filter>
 <filter-name>wicket.filter</filter-name>
 <filter-class>org.apache.wicket.protocol.http.WicketFilter</filter-class>
 <init-param>
  <param-name>ignorePaths</param-name>
  <param-value>images/,rest/</param-value>
 </init-param>
</filter>

This will let the Wicket filter immediately pass through to the servlet container for all URLs that start with http://host/app-context/images/ and http://host/app-context/rest/.

Note: make sure the paths are comma separated without additional whitespace.

Simon 2 in beta

2009-04-13T12:44:00.003+02:00

Java Simon, simple java monitoring, version 2 is in beta. I am quite proud of this because firstly the major change in the version was a result of my performance investigations, and secondly it contains my Spring integration code. See the java simon pages for news, downloads, etc.

Wicket course preparations

2009-04-06T15:16:00.006+02:00

Soon I will start teaching Wicket courses in The Netherlands. To prepare I spend some days in London with jWeekend teacher Cemal Bayramoglu (also know for organizing the London Wicket meetups). Most of our time we were at jWeekend's nice classroom, right in the middle of London, with close access to a very nice Thai restaurant, several coffee corners, etc.

The first day we went through the entire course that Al Maw and Cemal created. We had great fun discussing all the ins-and-outs. The second day we focused more on how the course should be taught.

I am really glad my company JTeam also thought it was a good idea to team up with jWeekend. The course is excellent and as the number of eyes on the material has only increased, will be even more excellent in the future.

Please contact as at info@jteam.nl if you are interested in getting a Wicket course.

Amsterdam Wicket meetup March 24 2009

2009-03-17T13:09:00.004+01:00

The Wicket meetup has finally been given a date and time! March 24 2009, 19:00 - 22:00 in the Mövenpick hotel Amsterdam.

Presentations

Registration (do not register on the page above)

Reliably sending email with Spring

2009-01-20T12:02:00.009+01:00

Update 2009-09-12: I no longer recommend this library. Please see the comments.

My colleague Allard just pointed me to an old but very useful library: HA-JavaMail.

The email sender that the JVM provides has some serious shortcomings. It does not automatically open a new connection when the connection was closed and you can forward your e-mail to 1 SMTP server only. Furthermore, it is not so fast. HA-JavaMail circumvents these problems by wrapping the JVM implementation.

In this tiny article I explain how you configure HA-JavaMail from Spring. First make sure you have the HA-JavaMail jar, the Spring jar, the JavaMail mail.jar and the JAF activation.jar library on your classpath. The latter two are available by default in full JEE containers.

Convert your mail sender bean declaration from this:

<bean id="mailSender"
    class="org.springframework.mail.javamail.JavaMailSenderImpl">
  <property name="mail.host" value="localhost"/>
</bean>

to something like this:

<bean id="mailSender"
    class="org.springframework.mail.javamail.JavaMailSenderImpl">
  <property name="javaMailProperties">
    <value>
      mail.transport.host=localhost,mail.example.com
      mail.transport.pool-size=1
      mail.transport.connect-timeout=0
      mail.transport.connect-retry-period=60
      mail.transport.sender-strategy=net.sf.hajavamail.SimpleSenderStrategy
    </value>
  </property>
</bean>

Note that the example adds a second SMTP server that will be used when the first is unreachable. Only a value for mail.transport.host is required. The other values shown here are also the defaults.

That’s it. You now have a much more reliable and faster email service.

Runtime monitoring libraries for Java

2009-01-12T11:20:00.007+01:00

Followers of this blog may have noticed an emphasis on monitoring lately. This is because my employer has decided to give me some time to investigate monitoring and create new components to ease the use of monitoring (in particular for Wicket applications). Blogging about the results is one of the requisites of this exercise.

This article lists all open source Java runtime monitoring tools I could find. By runtime monitoring I mean something that can left active in a production environment and measures things that occur inside the JVM.

Jamon
Jamon started in 2003 and is probably the oldest monitoring framework listed here. Although Jamon’s last release is from 2007, it is quite stable and used a lot; Sourceforge statistics show almost a 1000 downloads per month. Jamon is the work of a single person.

Features:

Programmatic access to start and stop timers, timers use milliseconds.
In memory database for aggregate statistics (no details are kept).
Can aggregate statistics on data ranges to provide a more detailed view when this is necessary.
Extensions to easily configure monitoring in software such as JDBC drivers, several servlet containers/application servers.
A JSP page to view the realtime data.

Jamon does not support a persistence mechanism and there is therefore no way to analyze historic data. To fill this gap you can use Jarep which provides several Jamon exports. Another way is to export the data through JMX.

Java Simon
Simon started last year as a direct competitor of Jamon. It is much faster, measures in nanoseconds and has a hierarchy of monitors so that you can enable/disable a tree of monitors. Simon is still in active development. Be prepared for API changes in Simon 2.0 which is to be released somewhen before summer 2009.

Features:

Programmatic access to start and stop timers, timers use nanoseconds.
In memory database for aggregate statistics (no details are kept).
Like Jamon, has a JDBC monitoring proxy driver

The Simon team is working on a JMX export and has yet to be determined plans for a persistence mechanism.

Currently Simon does not provide anything to view the statistics. Again you can use Jarep for this.

Usemon
Usemon seems to target monitoring of large clusters with support for tracing messages through the cluster. Measurements are send to a collector which stores it in a database.

Features:

Byte code manipulation to add timers to configured EJBs, Servlets, MDBs, custom defined POJOs, and classes implementing the SqlStatement, MessageSender and TopicSender interfaces.
Measures method invocation durations, exception counts and JVM information such as heap size.
Measurements are aggregated for 1 minute and then actively send to a central collector for storage.
A nice graphic interface for viewing the data (the project page is very vague about the details).
A clever database to handle huge amounts of data while still allowing efficient retrieval.

The Usemon site does not offer a lot of information. You'll probably need to just try it out to get more information.

Moskito
I found it a bit difficult to quickly grap Moskito. The website only shows a few use cases and leaves the rest undocumented.

Features:

Easy monitoring of servlets.
Measurements are aggregated per interval (e.g. 5 miniutes). Afterwards you can do something with the data (e.g. log it or send to a central location).
A nice very complete looking web interface.

A nice idea (that has not been implemented) is the support for virtual monitors, a monitor that shows the aggregated statistics for several other monitors.

Unfortunately I could not figure out how to instrument code except for servlets (extending a base class). The project website suggest that the project has been abandoned, so Moskito is not likely a good choice.

Commons monitoring
Commons monitoring is a Apache sandbox project. The website has good documentation but the newest information is from March 2008. SVN shows quite some activity so I am curious what the first release will bring.

Features:

Programmatic access to start and stop timers, timers use nanoseconds.
Support for AOP instrumentation.
In memory database for aggregate statistics (no details are kept).
Supports timers, counter (only increases) and gauges (increases and decreases).
Extensions to export data through a servlet in several formats, including HTML.

JETM
JETM (Java™ Execution Time Measurement Library) is again a library for doing raw measurements.

Features:

Programmatic access to start and stop timers, timers use nanoseconds.
Very nice: a Spring namespace to configure monitors on your beans.
In memory database for aggregate statistics (no details are kept).
In memory database for aggregate statistics (details are kept until it is time to calculate the aggregates).
Several renderers to export data.
Web interface to see current data.
Integrates with several projects as Spring, RRD4J and JMX.

Although JETM looks nice, it has been silent for more then a year, and simple questions on the mailing list are not answered.

Project Broadway
Broadway provides an entirely different kind of monitoring. The idea of Broadway is to define events of interest (for example in Groovy) on which can be reacted with some action.

Improving Jamon’s performance

2009-01-10T10:49:00.006+01:00

I was browsing through Jamon’s code to see why it is so much slower under high contention then Simon (see my previous article). In this article I will present these differences and show you a way to speed up Jamon right now.

Differences
Jamon has more features then Simon like listeners and data ranges. This obviously need some computing, if only to see if they are used. Another interesting difference is that Jamon uses double’s for aggregate statistics, whereas Simon uses long’s. I don’t really see the advantage of double’s (please surprise me) and it is probably a bit slower as well.

Synchronization
Jamon uses 5 synchronization points during a timing operation. There are 3 to get and create the monitor: on the MonitorFactory method, on the map of all monitor datas and on the map of all data range definitions. The last 2 synchronization points are for starting and stopping the timer (on the monitor data).

Simon (does not support data ranges) uses only 3 synchronization points: one for getting the monitor (on the SimonManager method) and 2 for starting and stopping the timer (on the timer itself).
Simon 2.0 no longer needs synchronization on starting a timer and therefore only needs 2 synchronization points.

Jamon defines its maps for monitors and data ranges as Collections.synchronizedMap(new HashMap(50)). However, it is not necessary to synchronize on these maps when the code that uses these maps is already synchronized on the MonitorFactory (in method getMonitor). Unfortunately when the synchronization wrapper is removed, all code that uses these maps will need to be analyzed to see if they properly synchronize.

Another solution would be to use a map that needs less synchronization: Java 5’s ConcurrentHashMap! Luckily Jamon provides a method to change the map implementation for exactly this reason. I included the following line in my test application (from my previous post):

MonitorFactory.setMap(new ConcurrentHashMap(200));

And these are the results. The measurements are in ms, the scale is logarithmic.

Quite astonishing: Jamon is no longer 5 up to 200 times slower, but just a bit slower for low contention, up to 15 times slower under heavy contention, and 2 times slower under extreme contention. Note that only the map for monitors was changed, the map for range definitions (always empty in my test) was not changed.

Conclusions
Even though Jamon is quite a bit slower under heavy contention, there is room for improvement. By calling a single method, you can make Jamon 20 times faster right now.

‘Effective Wicket’ presentation now on Vimeo

2009-01-07T13:45:00.006+01:00

My Effective Wicket talk on NL-JUG’s J-Fall conference of 2008-11-12 is now online! Watch the presentation (Dutch spoken) on Vimeo: part 1 and part 2 (provided by bachelor-ict.nl).

Feel free to download the slides of Effective Wicket (mirror) (PDF 4.1 Mb). If you want to see the notes skip to page 55.

Evaluating Simon - Java monitoring

2008-12-24T13:54:00.032+01:00

Recently a new monitoring kid appeared on the block: Java Simon (not to be confused with Dejal's Simon or some other simons).

Simon claims to be the successor of JAMon. As I just started a project to improve the documentation of Jamon, and integrate this better with Wicket projects, I thought this would be a good time to evaluate Simon and also to compare it to Jamon. Here are the results.

Features
Simon (only just started) can measure task durations (min, max, count, maximum concurrency and some timestamps) and has task counters. Other statistics can be added (supported now: standard deviation). The JDBC proxy driver can be used to monitor database usage. The monitors are grouped in a tree. You can disable each monitor individually, or per complete subtree.

Jamon adds the ability to measure in any unit (for example euro value of a transaction) and has quite a few monitors for Tomcat, Jetty, etc. Jamon does not organize monitors in a tree, so monitors can be disabled individually, or all at the same time.

So Jamon clearly wins on the feature front, but if you're just into the basic stuff Simon is fine.

Documentation
One of the weak points of Jamon is its documentation (hence my little project). Simon on the other hand starts with a good set of wiki pages. If these are maintained properly, Simon is the clear winner here.

Simon interface
Simon's interface is fairly simple. One gets a monitor through the singleton SimonManager, and call start and stop on it:

Stopwatch simon =
    SimonManager.getStopwatch("simon.test").start();
// Do something
simon.stop();

(I will not discuss the Jamon interface further, but making a measurement is virtually the same.)

Getting data out of a Simon monitor (a simon) is a bit more difficult. You can just get the measurements like this:

Stopwatch simon = SimonManager.getStopwatch("simon.test");
long hitCount = simon.getCounter();
long max = simon.getMax();
long total = simon.getTotal();
simon.reset();

But that would be very thread unsafe. As each method is synchronized separately, the three measurements may represent different moments in time.

The idiomatic solution would be to use method sample:

Stopwatch simon = SimonManager.getStopwatch("simon.test");
Map<String, String> sample = simon.sample(true);
long hitCount = Long.parseLong(sample.get("counter"));
long max = Long.parseLong(sample.get("max"));
long total = Long.parseLong(sample.get("total"));

This is more how it should be except for the boolean flag [F3, N7], and the return type of sample: a map! First of all, maps are very memory hungry [G21], secondly I have to assume that the values are longs [CE6] and parse them back from a String [GE37], and thirdly the keys are magic values [CE6, G25].

Here is another approach:

Stopwatch simon = SimonManager.getStopwatch("simon.test");
long hitCount;
long max;
long total;

synchronized (simon) {
  hitCount = simon.getCounter();
  max = simon.getMax();
  total = simon.getTotal();
  simon.reset();
}

Ouch. This works but uses client-side locking. That this is possible at all is a mistake in the Java language (as admitted by James Gosling, sorry, I lost the reference). Even so, Simon should not make it possible for clients to get the same lock that is used to protect the monitor data. This should be the only way to get actual data.

Recommendation: move all data retrieval methods (getMax(), getTotal(), etc.) from the monitor to an immutable data class, e.g. StopwatchMeasurements. Method sample should return instances of this class, or subclasses when a different analyzer was configured.

Update 2008-12-25: added following section on visiting all simons.
Update 2009-01-0: the iterator now actually works.

As I am using Jarep to browse through historic measurements, I want to expose the Simon data just like Jarep's servlet exposer for Jamon does. In other words: we need to visit each simon and extract the data as described above.

There are 2 approaches to visit all nodes. First is to get all simon names from SimonManager and then request them one by one. The other is to walk the tree of simons. I choose the latter. Unfortunately there is no tree iterator so I created one with commons-collections:

Simon rootSimon = SimonManager.getRootSimon();
ObjectGraphIterator simonIterator =
  new ObjectGraphIterator(rootSimon, new Transformer() {
    private Object previous;
    public Object transform(Object input) {
      if (input == previous) {
        previous = null;
        return input;
      } else {
        previous = input;
        Simon currentSimon = (Simon) input;
        return new IteratorChain(
          new SingletonIterator(currentSimon),
          currentSimon.getChildren().iterator());
      }
    }});

Admittedly ugly, but it works. Still there is one glaring bug here. The children are stored in a regular TreeSet. When we request the children through getChildren we get a collection that wraps the original set. Unfortunately that means that when an simon is added while we iterate the complete simon tree, we get a ConcurrentModificationException! Not quite acceptable.

Recommendation: store children of a simon in a CopyOnWriteArrayList instead of a TreeSet. Another option is change the complete set of simons in the manager into a ConcurrentMap and provide an iterator over that.

Performance To measure the performance of Simon and Jamon I wrote a test application that mimics a random web applications. It has a variable number of threads to mimic the number of concurrent requests that are being handled. There is one monitor to measure the duration of the entire request, one monitor to measure a specific request, one monitor to measure the service layer, and 0 to 3 monitors to measure the database layer.

The test application was run from IntelliJ with Soy Latte 1.0.3 (a 1.6 JDK, part of the OpenJDK for Debian) on a 2.4GHz Intel Core 2 Duo.

The measurements below display the time spend in obtaining, starting and stopping a single monitor. Measurements were done in nanoseconds, but are displayed here in milliseconds (logarithmic scale).

Simon's performance is pretty good! The overhead in my test application doesn't come anywhere near the millisecond range even in highly concurrent situations. Jamon is not so bad either, even though its 5 to 200 times slower, depending on thread contention. With 5 monitors in a request, the overhead per request slowly grows to 0.7 ms for Simon, and quickly grows to 12ms for Jamon.

Note: the test application does nothing else but getting monitors and doing measurements. So the right side of the chart is unrealistic for most environments.

Improving the performance
Even though the performance is pretty good. I think it can be improved further. I investigated two potential performance improvements:

Remove the ThreadLocal variable in StopWatchImpl.
Replace synchronized HashMap with a ConcurrentHashMap in EnabledManager.

Remove the ThreadLocal variable in StopWatchImpl Stopwatch method start stores the start time in a thread local. When you call stop the start time is retrieved and used to calculate the duration.

ThreadLocals have seen an enormous performance improvement in recent JVMs, but you are still better off without.

The idea is as follows: method start will no longer return the monitor itself but an object that contains only the start time and a stop method. Lets call it a StopwatchMeasurement. Only in the stop method will it be necessary to synchronize and update the counters.
The advantages are:

We no longer need to synchronize the start method.
We no longer need the thread local variable to store the start time.
We no longer need the start(key) and stop(key) variants.
The stopwatch is no longer susceptible for memory leak when a malicious client keeps starting monitors in new threads, but never stops them (someone forgot a finally).
The stopwatch is no longer susceptible for memory leak when a malicious client keeps starting monitors with start(key) with a new key, but never stops them (again, someone forgot a finally).

Disadvantage is that we need to use something else for the active counter. An AtomicLong would be very suitable as it uses hardware assisted atomic operations (and therefore has no synchronization overhead).
If it is desirable to keep the firstUsage and lastUsege timestamps (I don't use them), these can lift along with the start time and be stored during the stop method. Note: this may be unacceptable if you need information on long running operations.

To test it out I hacked StopwatchImpl to return a StopwatchMeasurement from the start method. If you look at the implementation you see that StopwatchMeasurement implements Stopwatch so that the idiomatic usage still works:

Stopwatch simon =
    SimonManager.getStopwatch("simon.test").start();
// Do something
simon.stop();

Of course this is a kludge that should be improved upon for a real implementation. In addition, I made no attempt to keep the active counter and the first/lastUsage timestamps.

Here are the results (again in ms per measurement):

Not bad! Although the results are quite dramatic for the higher thread counts, it still shaves off 20% in situations with less contention.

Recommendation: remove the thread local from StopwatchImpl and change the API as described above.

Replace synchronized HashMap with a ConcurrentHashMap in EnabledManager
The second thing I noticed is that getting a monitor from the SimonManager (the SimonManager.getStopwatch("simon.test") part) happens in a synchronized block and uses a plain HashMap as store. Since Java 5 there is a ConcurrentHashMap implementation which supposedly performs better in almost any circumstances.

Lets see. I remove some synchronized keywords, changed the map type, and made the code to add new monitors thread safe. This resulted in a modified EnabledManager.

Here are the results. The graph displays a few attempts that differ in the concurrency level parameter to the ConcurrentHashMap constructor. For more accuracy this graph displays the improvement in nanoseconds.

To my big surprise there is no measurable difference with modest contention. During high contention there is a reasonable positive effect that is suddenly converted to a negative effect for very high contention. Only when the currency level is set very high, does the effect stay positive.

Recommendation: don't change the manager.

Conclusions Although Simon is only young, it is off with a good start. Its clearly faster then Jamon and also on the documentation front Simon wins. Jamon has much more features so it is still a valid choice. When the recommendations in this article are followed, Simon could be better still.

Discussion
Follow the discussion of this evaluation on the Java Simon mailing list.

References
Code smells from Clean Code by Robin C. Martin:
[F3] Flag arguments
[N7] Names should describe side effects
[G14] Feature Envy
[G21] Understand the algorithm
[G25] Replace magic numbers with named constants

Code smells by me:
[CE6] Incomplete contract
[GE37] Unnecessary conversions

Don't use Intellij

2008-12-15T15:06:00.004+01:00

IntelliJ is okay, but it has one bug (for more then 4 years) that absolutely drives me nuts: its steals focus, big time. And not just once either. It can steal the focus for at least 4 times within 30 seconds!

If are thinking about using IntelliJ: don't do it!

If you are already hooked up: please vote for this bug!

Giving up on compiling a linux module

2008-12-06T22:19:00.004+01:00

I recently build my new home server PC. The motherboard is a D94GCLF, a small mini-ITX (17x17cm) from intel with an Atom processor. Unfortunately, the on board ethernet card, a RTL8102EL is not supported by Ubuntu 8.10. The r8169 kernel module works but it drops most received packets so its not as fast as it should be.

Realtek does provide the source to build the module yourself. Unfortunately, the instructions are a bit sparse. Too say the least. After a 7 hour struggle, installing kernel and header sources, lots of googling, making symbolic links here and there I finally compiled a .ko file. Insmod refused to load it.

So I gave up.

Now my new server is up to full speed again. If you can't solve your problem in software, add hardware! I put in an old 100Mbit/s ethernet card, and all is well again.

Making Mac's open/save dialog fast again

2008-11-17T11:21:00.006+01:00

Out of nowhere I had to wait 20 to 40 seconds on my year old iMac before I could open or save a file. Some people reported that iDisk could be the problem, but I don't even have a mac-account! However, those reports gave me a hint.

Indeed, a ls /Volumes/ showed this:

MacintoshHD Wuala ._Ipod_van_Corine

There it is! My girlfriend's IPod was not in sight, and I dumped Wuala because it stops working after a hibernate. A simple

sudo rm /Volumes/Wuala
sudo rm /Volumes/._Ipod_van_Corine

and all was well again.

Effective Wicket presentation

2008-11-13T10:14:00.009+01:00

Yesterday was J-Fall, the fall conference of the NL-JUG. One of the presentations was my Effective Wicket talk. Feel free to download the slides of Effective Wicket (mirror) (PDF 4.1 Mb). If you want to see the notes skip to page 55.

If you want the original material (code/images etc.) for your own presentation, please drop me a note. Its all creative commons, copyright by JTeam.

Update 2009-01-07: Watch the presentation (Dutch spoken) on Vimeo: part 1 and part 2 (provided by bachelor-ict.nl).

Wicket extreme consistent URLs

2008-10-15T10:49:00.023+02:00

Setting up a page to be behind a particular URL (aka mounting) in Wicket is fairly easy. Daan recently wrote REST like URLs for Wicket in which this is nicely explained. However, consistently keeping nice URLs, for example after a form submit, is a whole lot harder. So hard in fact, that even Wicket champion Igor believes it is currently not possible (it is an old post). His reasoning is of course completely correct, but he forgot one thing. Let me explain.

The problem
Many web applications have a login page. On it there is a form where you fill in your username and password. Suppose the page is mounted to:

http://example.com/login

with code like this in the application's init method:

mountBookmarkablePage("login", LoginPage.class);

When the user enters a wrong password, form validation (I assume you have a password validator) will fail and Wicket will redirect the user to a URL that points to the second version of the login page (the one with the error message). These URLs typically look like:

http://example.com/?wicket:interface=:0:::

For many applications this is just fine, in some its just too ugly.

A non solution
One of the proposed solutions is that you redirect to another page in the onError method of the form. E.g.

add(new Form(...) {
  @Override
  protected void onError() {
    setResponsePage(LoginPage.class);
  }
}

The redirection will happen, and the URL is indeed that of the login page, but you will have no error messages. Instead of going to the second version of the login page, you have created a new instance of the login page!

Another attempt
We could pass the error code in the URL:

add(new Form(...) {
  @Override
  protected void onError() {
    PageParameters pp = new PageParameters();
    pp.setString("error", "wronglogin");
    setResponsePage(new LoginPage(pp));
  }
}

Unfortunately, with the default URL encoding stratey the URL will now be:

http://example.com/login/error/wronglogin

Not at all attractive.

Getting close
A fairly recent addition to Wicket is the HybridUrlCodingStrategy (and subclasses). Let's mount the login page with one of these:

mount(new HybridUrlCodingStrategy("login", LoginPage.class));

If a user now enters wrong credentials, Wicket will redirect you to

http://example.com/login.2

The .2 means the second version of the login page for the current session. If the user would delete the .2 from the URL, the HybridUrlCodingStrategy will find the latest available version of the page and serve that. A whole lot nicer but still not perfect.

The solution
If only if we could force Wicket to not display the version number. Well... we can! We'll have to do some coding first:

// First attempt, DO NOT USE
public class NonVersionedHybridUrlCodingStrategy
        extends HybridUrlCodingStrategy {
  // ... trivial ctor

  @Override
  protected String addPageInfo(
        String url, PageInfo pageInfo) {
    // Do not add the version number as
    // super.addPageInfo would do.
    return url;
  }
}

And now mount with:

mount(new NonVersionedHybridUrlCodingStrategy(
         "login", LoginPage.class));

Indeed, the version number is gone! However, there is still a tweak to be done. In some circumstance Wicket will redirect you to the latest version of the page, but now that redirect will be to the same URL. This is no problem for Firefox and IE, but Safari, Opera and many tools do not allow this. Here is the final version that does not have the problem:

/**
 * UrlCodingStrategy that will give the same
 * URL for every version of a page.
 * @author Erik van Oosten
 */
public class NonVersionedHybridUrlCodingStrategy
        extends HybridUrlCodingStrategy {
  public NonVersionedHybridUrlCodingStrategy(
        String mountPath, Class pageClass) {
    super(mountPath, pageClass, false);
  }

  @Override
  protected String addPageInfo(
        String url, PageInfo pageInfo) {
    // Do not add the version number as
    // super.addPageInfo would do.
    return url;
  }
}

I named the URL strategy 'non versioned' as it no longer makes sense to have multiple versions of a page, just the last one will do. You should therefore also add the following fragment to each page constructor:

setVersioned(false);

Proof
Try it out! Go to tipSpot.com and try to login.

JAOO Observations

2008-09-29T14:41:00.003+02:00

Here are my day 1 observations from the JAOO conference.

Languages evolve slowly. Despite an 100000 increase in storage capacity, 10000 increase in memory and 1000 increase in processing capacity in the last 10 years, programming languages look and feel the same as 10 years ago. The only real difference is that the language has become only a small part of what you need to learn. The major part is now in the libraries, which did grow by 1000 increase.
For new languages, it becomes harder to have your own infrastructure. That is why many are written on the JVM or the CLR.

If you are writing a social website, you must look at open social. The open social specification has been handed over to a foundation. It seems to have become really easy to incorporate it into your social website.

A long discussion on the beautiful (yep, that´s really the best word) presentation ´Designed as designer´ gave the following somewhat disappointing results:

´Designed as designer´, or by substitution ´Work of art as designer´ is the wrong title. It is true that great works of art (including software projects) write themselves after the first draft, but the real topic of the presentation was that all works of art are produced by groups of people, not single individuals.
However, since we already operate in the group modus, our lives are not changed at all.

Java transaction boundary tricks

2008-08-13T10:11:00.011+02:00

Controlling transactions is one of the fundamental things you must control well if your program is to be used by more then 1 user and has a database (as in about every web application). This article shows a little trick to get more out of transactions then most programs do. Since I am using Spring to demonstrate the principle, I will introduce Spring's transactions first. Feel free to skip that section if you know all about Spring transactions.

Using transaction from Spring
With Spring, controlling where a transaction is started can be as simple as adding the @Transactional annotation (and one extra line in the Spring configuration). The @Transactional is typically placed on the class that implements the interface of your service layer. For example you may have a MemberService:

public interface MemberService {
  List findMembers(MemberCriteria criteria);
  Member getMember(long id);
  Member updateMember(Member member);
}

with the following implementation:

@Transactional
public class DefaultMemberService implements MemberService {
  public List findMembers(MemberCriteria criteria) { ... }
  public Member getMember(long id) { ... }
  public Member updateMember(Member member) { ... }
}

When you let Spring instantiate DefaultMemberService as a bean, all of its public methods will be automatically proxied in such a way that the method is executed within a transaction.

Java transactions are bound to a thread (think ThreadLocal), the proxy can therefore join an existing transaction in the current call stack, and by default that is what happens. As a result you can safely call other transactional service methods from within your own transaction methods.

Introducing the CommandService
Here is a kind of weird service that I came up with a couple of years ago, named after the Command pattern, the CommandService:

@Transactional
public class DefaultCommandService implements CommandService {
  public void inTransaction(Runnable command) {
    command.run();
  }

  public <T> T inTransaction(Callable<T> command) {
    try {
      return command.call();
    } catch (Exception e) {
      throw new RuntimeException(
          "CommandService#inTransaction("
          + command.toString() + ")", e);
    }
  }
}

(Download link at bottom of article.)

As you can see it doesn't to anything except for executing the runnable or callable it is given. How can this be useful? Well, notice the @Transactional. Your block of code is now executed within a transaction!

Again, how can this be useful? Here are 3 use cases:

Use case 1: prevent fine grained service methods
Suppose you have a webpage that will update only one property of an entity, e.g. the email address of a member, and another page to update the password. Now to prevent concurrent update problems you want to get and update the member within the same transaction. You could make a new method on the service interface for each property you want to change. However, it is a good practice to keep interfaces concise. So instead we do something like this:

final long memberId = ...;
final String newEmail = ...;
Member freshMe = 
    commandService.inTransaction(new Callable<Member>() {
  public Member call() throws Exception {
    Member freshMe = memberService.getMember(memberId);
    freshMe.setEmail(email.getNewEmail());
    return memberService.updateMember(freshMe);
  }
});

The member entity is retrieved and updated in the same transaction. This way we only need 2 methods in the member service interface to safely update any property of a member.

Use case 2: bundle operations for performance reasons
I once wrote some code that would keep thousands of LDAP records in sync with a relational database. In my test environment, a dodgy laptop, it would take 50 minutes to do the initial import of 15,000 records. Each record was persisted to the database in its own transaction. After a small code change, I used the CommandService to persist the records in groups of 20. What do you think happened? The total time to import these 15000 records dropped from 50 to 2 minutes! Not bad, 25 times faster through 4 lines of extra code.

Careful positioning of transaction boundaries can have a dramatic effect on performance. CommandService can help you do that.

Use case 3: transactions in unusual places
Let us look at a little bit more detailed example of the PeriodicRetriever from my previous article (changes in italic).

public class CachingPostalCodeService {
  private final Object postalCacheLock = new Object();
  private List<PostalCode> postalCache;
  private PeriodicExecutor postalCacheReloader;
  private HibernatePostalCodeDao hibernatePostalCodeDao;
  
  public CachingPostalCodeService() {
    postalCacheReloader = new PeriodicExecutor(
          TimeUnit.MINUTES.toMillis(10), new Runnable() {
      public void run() {
        refreshPostalCache();
      }
      
      public String toString() {
        return "Postal code cache reloader";
      }
    });
  }
  
  public List<PostalCode> getPostalCodes() {
    postalCacheReloader.requestStart();
    synchronized (postalCacheLock) {
      return postalCache;
    }
  }
  
  private void refreshPostalCache() {
    List<PostalCode> newPostalCodes =
        Collections.unmodifiableList(
          hibernatePostalCodeDao.getAll()
        );
    synchronized (postalCacheLock) {
      postalCache = newPostalCodes;
    }
  }

  // ... setter for hibernatePostalCodeDao ...
}

If you try this, you may notice that the data is retrieved the first time, but not the second time! The exception message is clear too, something like "Hibernate requires a transaction". To understand this lets follow the two execution paths.

The first time getPostalCodes() is called, it is called through Spring's transaction proxy, so the method is executed within a transaction. So when the PeriodicRetriever is called, the runnable (which calls refreshPostalCache()) is immediately called within the same thread and thus also within the same transaction. No problem!

The second time getPostalCodes() is called (11 minutes later), PeriodicRetriever is called again but returns immediately. Meanwhile it starts a new thread which executes the runnable defined in CachingPostalCodeService's constructor. The important thing here is that the new thread does not join the transaction of its parent thread as transactions are bound to threads. The runnable then calls refreshPostalCache(). You may think you'll get a new transaction at that moment. However, a method call within the same class will never go through the transactional proxy. This is because the instance on which we call refreshPostalCache() is simply this, and not the proxy that was obtained from Spring. In other words: refreshPostalCache() is not executed within a transaction.

CommandService to the rescue (again, changes in italic):

public class CachingPostalCodeService {
  private final Object postalCacheLock = new Object();
  private List<PostalCode> postalCache;
  private PeriodicExecutor postalCacheReloader;
  private HibernatePostalCodeDao hibernatePostalCodeDao;
  private CommandService commandService;
  
  public CachingPostalCodeService() {
    postalCacheReloader = new PeriodicExecutor(
          TimeUnit.MINUTES.toMillis(10), new Runnable() {
      public void run() {
        commandService.inTransaction(new Runnable() {
          public void run() {
            refreshPostalCache();
          }
        });
      }

      public String toString() {
        return "Postal code cache reloader";
      }
    });
  }

  // ... same as above ...
  
  // ... setters for hibernatePostalCodeDao and commandService
}

Download
Feel free to use the complete version of CommandService in any way you see fit.

Conclusions
Watching your transaction boundaries can be very rewarding, both in code size as in performance. A command service, such as the one presented in this article can help you do that.

Asynchronous cache updates

2008-08-06T10:50:00.008+02:00

This is the second article on Java concurrency. The first is Java thread control.

Many applications use more or less static reference data such as postal codes, exchange rates and externally stored text. Often this type of data can be safely cached in memory for performance reasons. To make sure that the cached data does not become stale for too long, it can be reloaded every 10 minutes, once a day or on any other schedule you might like.

Rather then creating custom code for every type of reference data, one can extract the hard concurrency code to a utility class. This article presents the PeriodicExecutor, a utility class I wrote years ago, and proved to be so useful and reliable that is has been traveling with me from project to project.

Here are the requirements I had in mind while writing PeriodicExecutor:

Support any data retrieval and any cache mechanism.
Refresh the cache in the background, do not slow down request handling.
No idle threads, only start a thread when one is needed.
Be resilient against errors, retry when they occur.
Support flexible reloading schedules.
Support initial loading of the memory cache.

Lets discuss these requirements and see how they are implemented.

1. Support any data retrieval and any cache mechanism.
Each type of data source needs its own retrieval code. E.g. JDBC code for database stuff and things like HttpClient for remote stuff. The cache implementation can also be different from case to case. For example in one case we just need to store a String, in another case we first transform the data to a Map. Therefore the utility class will only solve the the multi-threaded scheduling problem.

PeriodicExecutor solves this by letting the client code provide its reload task as a Runnable. The task is also responsible to make the results available when reloading is finished.

2. Refresh the cache in the background, do not slow down request handling.
When its time for a data reload, it does not make sense to delay normal request processing until the reload is done. Remember, we are caching for performance reasons! As a consequence PeriodicExecutor executes the client task in a separate thread.

PeriodicExecutor guarantees that the task is never run in parallel, but the client task must make sure that there are no concurrency problems the moment the cache is updated.

3. No idle threads, only start a thread when one is needed.
There are 2 approaches to thread handling. We can either start a thread upon initialization and keep that thread running all the time, or we can start a new thread each (short) time a reload is required. The first approach has the disadvantage that more resources are used then is necessary (though since Java 5 we could share an executor service with all reloaders). Inconvenient is that we need explicit code to shut down the thread (or executor service). Advantage of this approach is that a thread can schedule reloads precisely.

I chose the second approach: each time the cache is accessed by some code, that code must call method PeriodicExecutor#requestStart() to see if a reload is dictated by the schedule. If that is the case, the reload is started in a newly started background thread. A small disadvantage to this approach is that after a long period of inactivity, the first caller will trigger a reload, but until the reload is finished all cache users will see stale data.

4. Be resilient against errors, retry when they occur.
The reference data may be retrieved from an unreliable data source. When the reload fails, it should be tried again later. Normal processing can continue as the new background thread isolates it from exceptions. In addition, the reload task should only update the memory cache until after the reload succeeded.

PeriodicExecutor detects reload errors by catching exceptions thrown by the reload task. When an error occurred the next call to requestStart() will trigger a second reload attempt. After 2 failed attempts the next reload task is delayed for a couple of minutes to prevent trashing.

5. Support flexible reloading schedules.
PeriodicExecutor normally reloads with a fixed delay. However, it is easy to override this by providing your own next reload time. A future improvement would be to extract this code according to the strategy pattern.

6. Support initial loading of the memory cache.
Initially the memory cache is empty. As we are talking about reference data, it does not make sense to continue for most programs. PeriodicExecutor therefore also support synchronous loading and exception rethrowing. During a synchronous load the caller is blocked until the reload task is done. Before one task completed, the default is to run synchronously and rethrow exceptions from the reload task. Once one task completed, the default is to run asynchronously and to swallow exceptions (they are logged of course).

Example
Enough theory, how do you use it? Here is a simple example:

public class CachingPostalCodeService {
  private final Object postalCacheLock = new Object();
  private List<PostalCode> postalCache;
  private PeriodicExecutor postalCacheReloader;
  
  public CachingPostalCodeService() {
    postalCacheReloader = new PeriodicExecutor(
          TimeUnit.MINUTES.toMillis(10), new Runnable() {
      public void run() {
        refreshPostalCache();
      }
      
      public String toString() {
        return "Postal code cache reloader";
      }
    });
  }
  
  public List<PostalCode> getPostalCodes() {
    postalCacheReloader.requestStart();
    synchronized (postalCacheLock) {
      return postalCache;
    }
  }
  
  private void refreshPostalCache() {
    List<PostalCode> newPostalCodes = ....
    // Don't forget to make newPostalCodes unmodifiable.
    synchronized (postalCacheLock) {
      postalCache = newPostalCodes;
    }
  }
}

Notice that only a limited amount of code is needed to get all the discussed features. In this example a list of postal codes is cached in memory. The only user of the cache, method getPostalCodes, calls requestStart every time. Initially, when no data is in the cache, requestStart will block until the data has been loaded. After 10 minutes of use, the cache will be reloaded in the background. Note how the example synchronizes on postalCacheLock to prevent concurrency problems during the cache update.

Variations
Sometimes you only need to reload once a day, preferably early in the morning. The following example shows how one can override scheduled reload to 5 o'clock in the morning.

  public CachingPostalCodeService() {
    postalCacheReloader = new PeriodicExecutor(
          0, new Runnable() {
      // ...as above...
    }) {
      @Override
      protected long nextExecuteTime(long currentExecute) {
        return getNextTime(5);
      }
    };
  }

  /**
   * @param hour the requested hour
   * @return the next time it is the given hour
   *   in the JVM default time zone
   */
  private static long getNextTime(int hour) {
    Calendar cal = Calendar.getInstance();
    cal.set(Calendar.HOUR_OF_DAY, hour);
    cal.set(Calendar.MINUTE, 0);
    cal.set(Calendar.SECOND, 0);
    cal.set(Calendar.MILLISECOND, 0);
    if (cal.getTimeInMillis() < System.currentTimeMillis()) {
      cal.add(Calendar.DAY_OF_MONTH, 1);
    }
    return cal.getTimeInMillis();
  }

If you need more control on when to execute synchronously or not, take a look at method PeriodicExecutor#requestStart(boolean).

Download
Download PeriodicExecutor. PeriodicExecutor can also be used with Java 1.4 (and perhaps 1.3 as well).

Other options
Probably the most complete FOS library for task scheduling is Quartz.

Enjoy!

Java thread control

2008-08-02T09:50:00.008+02:00

As promised in my book review of Effective java 2nd edition, here is an article that shows a neat trick to control threads.

The recommended way to stop a thread according to the book is to use a volatile boolean field as a flag to properly stop a running thread:

public class StopThread {
  private static volatile boolean stopRequested;
  
  public StopThread() {
    Thread t = new Thread(new Runnable() {
      public void run() {
        int i = 0;
        while (!stopRequested) i++;
      }
    });
    t.start();

    TimeUnit.SECONDS.sleep(1);
    stopRequested = true;
  }
}

(See the book to understand why the volatile keyword is essential.) This is all nice, but this thread can only be started and stopped once. Here is a better implementation:

public class StopThread {
  private static volatile Thread currentThread;
  
  public StopThread() {
    Thread t = new Thread(new Runnable() {
      public void run() {
        int i = 0;
        while (currentThread == Thread.currentThread()) i++;
      }
    });
    currentThread = t;
    t.start();

    TimeUnit.SECONDS.sleep(1);
    currentThread = null;
  }
}

By storing a reference to the running thread, the running thread itself can easily see whether the rest of the system also thinks it should be running. Restarting the thread becomes easy now. Just reference a new thread instance and the old thread will die.

The previous program just demonstrates the principle. A better starting point is the following program:

public class RestartableThread {
  private AtomicInteger threadNumber = new AtomicInteger(0);
  private volatile Thread currentThread;

  public RestartableThread() {
    startThread();
  }

  private void startThread() {
    currentThread = new Thread(
      new RestartableRunnable(), "Restartable thread " + threadNumber.getAndIncrement());
    currentThread.setUncaughtExceptionHandler(new Thread.UncaughtExceptionHandler() {
      public void uncaughtException(Thread t, Throwable e) {
        startThread();
        logger.error(String.format("Thread '%s' has an uncaught exception and was restarted", t.getName()), e);
      }
    });
    currentThread.start();
  }

  public void shutdown() {
    currentThread = null;
  }

  private class RestartableRunnable implements Runnable {
    public void run() {
      while (true) {
        if (currentThread != Thread.currentThread()) {
          return;
        }

        try {
          // Do a bit of work. Make sure this takes a limited time.

        } catch (Exception ex) {
          logger.error("exception", ex);
        }
      }
    }
  }
}

You can stop the thread by calling the shutdown() method. Failure to do so on program exit will prevent the JVM from stopping.
The runnable contains a try/catch to prevent the thread from dying. As a double fail safety mechanism, the uncaught exception handler is triggered for unhandled Throwables such as out of memory errors. The exception handler then simply starts a new thread so that execution continues. There are subtle advantages to this double approach. Exceptions are normally not so catastrophic, the thread could just continue. Out of memory errors however can be better dealt with by letting the thread terminate and make all its used memory available for the garbage collector.
If for any other reason you want to (re-)start the thread (for example because it is programmed to run only for a limited time), simply call startThread().

Make sure that the run condition is checked regularly. Consumer threads that wait on something like a queue should wake up regularly. For example, if you are waiting on new entries in a BlockingQueue this is better done with poll() then with get().

Conclusion
Despite the presence of the new and shiny executor services, it is still sometimes necessary to write your own thread. I hope that this little article will get you started on a robust yet flexible implementation.

Next thread article: a helper for asynchronous cache updates.

Effective Java 2nd edition - Book review

2008-08-01T20:34:00.006+02:00

If you are even a bit serious about your java programming, this book is a must read. There is no other book that improved the quality of my java programming in such a dramatic way.

Now this was my opinion on the first edition. After having read the second edition, I can tell you that the same top quality is there, with lots and lots of big and tiny updates on top.

Ok, with that out of way, here are some comments:

In the introduction it says 'This book is not intended to be read from cover to cover'. Ignore that, read it from cover to cover anyway!
Initially I missed an overview of what changed since the 1st edition. But when I started reading I understood that such a list would be ridiculous. Every chapter has been improved in tiny, and sometimes big ways.

And now for some tiny counter noises:

Item 4: This item is about enforcing noninstantiability of classes that contain static methods only. However, except for saying that is nonsensical, it fails to provide a good reason for preventing this. I would say: don't bother. Let silly programmers instantiate the class. Firstly it can not do much wrong, secondly, any good IDE warns you about it.
Item 8: When overriding equals, one should use == instead of equals to compare fields of enum type. The item mentions that this should only be done for certain primitive fields.
Item 60: I disagree with the advice to throw NullPointerException in case an argument was null while the contract of the method specifies that this is not allowed. NPE is also thrown by the JVM and is therefore more suitable to indicate a programming error. IllegalArgumentException is in my opinion more suited to express that a method is used wrongly. In other words: a NPE indicates an error inside the method, an IAE indicates an error outside of the method.
Item 66: I recently saw a neater trick to control threads. More on this in the next post.
Item 68: One thing that still puzzles me is how one can write a consumer task with the new executor services. This item does not make this clear. Perhaps this use case still warrants the creation of your own thread.

Happy reading!

Update 2008-08-03: refined my critics on throwing a NPE vs IAE.

Spring message resource weirdness

2008-07-24T20:38:00.005+02:00

Today I got a bug report that e-mail subject appeared in Dutch while they should have been send in English. The first thing I check of course was whether the correct locale was used. It was.

Here is the situation:

Spring message source
2 files: email-subjects.properties and email-subjects_nl.properties
requested locale: en_US

After a bit of debugging I noticed that Spring looks for the following files (in order):

email-subjects_en_us.properties
email-subjects_en.properties
email-subjects_nl_NL.properties
email-subjects_nl.properties
email-subjects.properties

WTF?! It also looks for the default locale! Thank you Spring! This is really the first time you disappoint me (in 4 years, not bad). But of course I should have looked further, when will Spring ever let you down?

The solution? Well luckily the authors of the code realized this is not always what you want. An easy setFallbackToSystemLocale(false) overrides this funny behavior. I added <property name="fallbackToSystemLocale" value="false" /> to my config file and it all worked. Hurray for Spring!

Update 2008-08-04: adapted the article to the real solution.

JPA and bit operations (Hibernate/Postgres)

2008-06-28T14:22:00.005+02:00

In my current data model one of the entities has a column of integer type that contains a large set of boolean flags. Of course this is well hidden in the domain model objects, so you can just call things like event.isCanceled().

However, in the administration interface we need to search for this entity based on separate flags. The JPA query I found in our code base is select ... where flagBits = 1 (canceled is expressed in bit 0). Oops. What if the entity is not only canceled, but also sold out (bit 1)? Quite wrong, lets fix it.

So, I rewrote the where clause to flagBits & 1 = 1 (& is the binary and operation in Postgres). Hibernate, our JPA implementation parses the query to do its ORM magic. Unfortunately, Hibernate does not understand the & operation (though I heard it did in Hibernate 2) so it will throw an exception.

Looking through the Hibernate and Postgres documentations I found the get_bit operation. It does not except integer types as their value, so a cast was needed: get_bit(''||flagBits, 1) == 1. Another exception. And hopelessly wrong as well, ''||flagBits creates a string with the decimal representation of the number, the function get_bit expects a byte array. Maybe: get_bit((''||flagBits)::bytea, 1) == 1. Not really, the problem is still there. And furthermore, Hibernate also does not support the || and :: operations, more exceptions from Hibernate. (The :: could be replaced by a supported cast operation.)

Back to basics. How can you access a given bit with more basic, supported operations? Fact: division by 2x will shift the bits right by x. Fact: a number modulo 2 gives 1 if bit 0 is set.

What follows is the where clause (flagBits / bitmask) % 2 = 1. Where bitmask is 1 for bit 0, and 2 for bit 1, etc. Oops, % is not supported either! Last attempt: mod(flagBits / bitmask, 2) = 1. Success at last.

Conclusion

JPA/Hibernate give an extensive but still limited subset of SQL expressiveness.
It pays off to pay attention during math classes.

Ubuntu server 7.10 to 8.04 upgrade experiences

2008-06-13T20:32:00.010+02:00

Yesterday I upgraded my server at home (running 3 websites and a small e-mail server) from Ubuntu 7.10 to 8.04. My main system is a nice looking iMac, completed with an eeepc for traveling and kitchen table work. But I use Ubuntu in the closet and at work.
The upgrade went rather well. Of course, I had installed as little packages from outside the repositories as possible. I even did the upgrade over ssh while that is not recommended.

Unfortunately, I only just noticed that no e-mail was being received anymore. After some head scratching (I had also changed the firewall settings), I decided to compare my backup of /etc (yes, I had one!) to the current copy.

So how do you compare such a complex directory as /etc on a Mac? I tried Eclipse and IntelliJ. Despite their excellent compare tools, they have no such thing for complete directories. Searching the web for open source solutions did not give any conclusive evidence as well. Kdiff3 was mentioned most frequently, so I decided to try that one. Kdiff3 can be installed with MacPorts, but make sure you have plenty of time. It took more then more then 2 hours on this screaming iron!
Kdiff3 is not so intuitive as the commercial products Araxis and Beyond Compare (which I have used professionally), but once you get used to it, it is a nice and powerful merging tool.

And there it was: a small copy/paste error in my dovecot configuration. Postfix uses dovecot for authentication so every TLS connection failed. Copy, paste, restart, done!

Conclusions

Make sufficient backups of all your configuration files.
Use the packages from your distribution as much as possible. Do not install later versions of a package unless you are prepared to install it again and again and again.
Kdiff3 is an acceptable diff tool on the Mac.
Do not be afraid for an Ubuntu server upgrade!

Do not offer money!

2008-06-01T21:09:00.006+02:00

Money and open source, always an interesting combination. But how do we, simple developers (with no ambition but to make beautiful stuff and live from it), deal with this? Let me offer you my opinion.

Lets start with a real example on what you should not do. Here is a quote from an e-mail that was sent on the user list of well respected project xyz (which it is, is not important):

Hi!
I was wondering how I could make a contribution for xyz. I'm not talking about a code contribution but rather a small money contribution. I have got a lot of help here on this forum and in fact I don't think I have ever experienced this kind of help elsewhere!
Thanks to all that have helped and especially core xyz coders.

The author obviously has good intentions, so how can this be so disastrously wrong? Suppose you are celebrating Christmas (or any other traditional family event) at your mother-in-law´s house. She cooked really well and everything tasted just wonderfully. You stand up and say, "Mother, this was excellent. This must have been worth at least 400 euros. Here, this is for you." and you hand here 4 fresh 100 euros bills. You will probably understand that your mother-in-law will not be happy, and in fact she will likely not forget this for a long time. Will she have the same pleasure in cooking for you next year? I think not.

So what is the problem? As Dan Ariely, a professor of behavioral economics wrote in his excellent book Predictably Irrational (I read the Dutch translation "Waarom we altijd tijd te kort komen") there are 2 sets of norms, market norms and social norms. Market norms regulate how we interact with each other when money is involved. Social norms regulate all other interactions. So eating at a restaurant will fall under the market norms, eating at your mother-in-low falls under the social norms. It is both about eating, but if you think about it, there are vast differences in what you expect, how you deal with bad food, etc.

The coders of xyz do all of their hard work in their free time. They are not getting paid, they do this purely out of love for what they are creating, and perhaps because of the idea that it is useful to others, and perhaps because of the respect they will receive. Any social interaction therefore falls under the social norms. Now if you start to offer money, you will move the social interactions to market norms. Luckily the core coders of project xyz are wise and responded with a request to donate to a charity project. But suppose they accepted the money. Now why would they do their best next time when no money is involved? They might as well go drinking beer with friends. Or worse, why would non core contributors do their best if they do not get paid? If you know somebody else is getting paid for the same task, even if it only takes 1 minute, why would you spend any brain power on it?

What is even worse, after switching to market norms, it takes a lot of time before the social norms are reinstated. Here is quote from Eric Daugherty, author of the Java Email Server, on JES version 2.0:

I started down this path with passion and drive. After I'd completed the SMTP (I think) portion of the code base, I was contacted by Andrew Oliver about my interest in working on a JBoss mail server. He was working on a new project to build an enterprise class mail (and calender, etc) server build on JBoss (where he was employed at the time). This project, JBossMailServer at the time, seemed to eclipse what I was attempting, and given my already slowing momentum on the 2.0 branch, pretty much brought it to a screeching halt. I had a bit of interest in working on the JBoss version, but in truth some of the motivation for the projects that I work on is ego. Working on a project that wasn't 'mine' didn't quite have the same appeal, even if it was JBoss (remember, this was 4 years ago).
Apparently JBoss had found someone interested in the server, as they approached me with an offer to pay me for the task of completing the SMTP (or POP, I don't recall) functionality. Since I was a pretty easy task for me (I'd done it once already), and hey, they were paying me, I jumped on board. I worked through the deliverable they wanted and earned a little spending money.
However, a funny thing happened. Once I'd worked for pay on the project, it was really hard to get excited about working for free. Combine this with the lack of real ego payoff, and I drifted away from the project.

So when the previous state is restored (the money is gone), the market norms stay for quite a while. This is proven by Dan and colleagues and this is also Eric´s own observation. A troubling conclusion: by offering money for free services the result is less services.

Suppose you care for the success of an open source project, what can you do? My answer is simple: contribute by writing documentation, by answering questions, by writing patches, provide hosting and otherwise interact according to the social norms. If you are not able to do any of these things, it is better to stay out!

Acknowledgments
Thanks to Dan Ariely for his research. The Christmas dining example is from his book.

Everything about Wicket internationalization

2008-05-04T14:55:00.040+02:00

I have been using Wicket for about one and a half year (and followed it about a year longer), and only recently I realized the tremendous flexibility of Wicket. The core developers keep introducing major features by replacing large and small parts of the core, without changing much of the existing programming interface. It doesn't stop there though. Any good developer can do the same with relatively little effort. On occasion I replaced important parts of the core myself, just because I wanted it a little bit different.

Upcoming release 1.4 is all about introducing the long awaited generic models. But the release notes of Wicket 1.4-m1 contain another little gem. Created by non-core developer John Ray: child components within a wicket:message element. But before I explain what that means, lets first look at your internationalization options in Wicket.

This article requires you to have a very basic grasp of Wicket (what the application class is, how wicket:id is used, and simple components like Panel and Label). A bit of knowledge on Java properties helps as well.

Simple texts: wicket:message element, and wicket:message attribute
A simple web page easily contains dozens of texts that need i18n. It would be cumbersome to have to put a wicket:id on all of these, and then add a Label to each. Instead you can use the wicket:message element in the HTML file directly. For example in MyPanel.html:

<wicket:message key="helloworld">Hello</wicket:message>

The key (helloworld) is used to lookup the message in the property files. The default text (Hello) is used when the message could not be found.
The property files are (MyPanel.properties):

helloworld: Hello world

And for Dutch(MyPanel_nl.properties):

helloworld: Hallo wereld

The ouput with an English locale:

Hello world

You can do the same for HTML attributes. For example:

<input type="submit" value="Default text"
    wicket:message="value:helloworld"/>

Would result in:

<input type="submit" value="Hello world"/>

If there are multiple attributes to translate, add them prepended with a comma:

<input type="submit" value="Default text"
    wicket:message="value:helloworld,title:hellotitle"/>

Wicket's locale selection
As usual in Java i18n systems, messages are looked up by locale. The locale is automatically extracted from the HTTP request, but can also be explicitly set with getSession().setLocale(Locale.US).
If you need to override the locale for a specific component, override getLocale() on that component and it will be used by that component and its children.

If a client does not provide a preferred Locale, Java's default Locale is used. See Locale.getDefault() and Locale.setDefault(Locale)) for more information.

Finding the message
Another extremely powerful concept is how the message is looked up in the property files. Normally a properties file with the same name as the current component is used. E.g. if you are working within the component SummaryPanel, the message is looked up in the file SummaryPanel_nl_NL.properties (for Dutch, Netherlands locale). If that fails (either because the file does not exist, or because it does not contain the message), file SummaryPanel_nl.properties, and then SummaryPanel.properties are investigated. This is all just like Java's resource bundles work. But of course Wicket goes further. If the message is still not found, the property files of the parent class are investigated, and again, and again, all the way up to java.lang.Object. For each class all locale variants are searched for.

Components are reusable, but in order to make it really reusable you should be able to override the messages depending on where the component is used. This is facilitated by first looking up the message (following the algorithm above) for every parent in the component hierarchy (aka page hierarchy). Every component can override the messages of its child components, so the search starts at the page's properties and then trickles down to the component that uses it (yes, its top-down). In order to make overrides specific to a certain child component, you can prefix the message key with the component id of the child. See ComponentStringResourceLoader for more details.

If no message was found in the page hierarchy, another search starts which will look at your application class and its super classes. So Wicket first looks at MyApplication.properties (provided MyApplication is the name of your application) and then up the class hierarchy, passing org.apache.wicket.Application, up to java.lang.Object. This is how Wicket provides its many default i18n texts.

This might sound complicated, but in practice you simply have one properties file per page and some more for components that are reused over multiple pages. For smaller applications you can even put everything in one properties file. These rules work so well that you just do what you think is correct and it almost always just is.

One note on the location of the properties files: like HTML files, they must be in the same package (and same classloader) as the component they are associated with. In practice they live next to each other in the same directory.
If you want Wicket to get its resources from somewhere else (e.g. from a database), you can implement the interface org.apache.wicket.resource.loader.IStringResourceLoader and configure this in the init() method of your application class.

References: ExtensionResourceNameIterator, ComponentStringResourceLoader, and for real control freaks the new PackageStringResourceLoader.

Reloading and caching
When Wicket is started in development mode, changed properties files are detected and reloaded. To properly make use of this feature from an IDE, you should run Wicket directly from the compiled sources, for example with the Start file, included in every QuickStart. In Eclipse you just save the properties file, in IntelliJ you must do a make (Ctrl-F9) before changes are picked up. In production mode, resolved properties are heavily cached for performance. (Same applies to html files.)

Putting dynamic values in the messages
As soon as you need to add values to the messages, you also need to add some Java code. The java code provides the vales, but the rest of the text still comes from the properties file. One of the nice things here is how Wicket leverages java bean properties.

Here is a complete example. Not many frameworks make this so easy!

MyPanel.properties:
summ: You, and ${otherCount} others, reviewed '${title}' \
 and rated it ${rate}.

MyPanel.html:
<span wicket:id="summary">Text that will be replaced.</span>

MyPanel.java:
// Summary has getters for otherCount, title, etc.
Summary summary = ...;
add(new Label("summary", new StringResourceModel(
    "summ", this, new Model(summary))));

Resulting in something like:

<span>You, and 5 others, reviewed 'Wicket in Action'
and rated it excellent.</span>

Property based message key It goes further: do you know a framework that can do this?

MyPanel.properties:
summ.short: Thanks!
summ.long: You, and ${otherCount} others, reviewed '${title}' \
 and rated it ${rate}. Thanks!

MyPanel.html:
<span wicket:id="summary">Text that will be replaced.</span>

MyPanel.java:
// summary.getMsgPrefs().getStyle() returns "short" or "long"
add(new Label("summary", new StringResourceModel(
    "summ.${msgPrefs.style}", this, new Model(summary))));

And more! The Java property syntax is also still available (e.g. like {0,Date}, {2,number,###.##} etc.)? You can find all forms in the javadocs of StringResourceModel.

The trouble with property files
Now this is all nice, but there is one problem you will frequently encountered while using string resources. In this problem you have a sentence that contains one or more dynamically constructed parts (for example some links). The order of these dynamic parts is potentially different for each locale. In the previous example, we may want to link to an information page on the title, and on the word 'others' open a modal window with a list of people.

If you try this with Wicket 1.3 (or any other Java framework I know of that uses resource bundles) you'll find that there is actually no pure way to do this. I know of 3 workarounds, but neither is very attractive. Lets review them. The first way is to split the texts before, after and between the dynamic components. This works good enough, but varying the order of the components is not possible unless you so something clever on the Java side, or have multiple HTML files. The latter is also workaround number 2: for each locale use a separate HTML file with the translation directly included (note: Wicket uses the same rules to lookup HTML files as it does for property files, e.g. MyPanel_nl.html just works like you expect it to). However, having more then one HTML file per component often leads to maintenance horror as changes must be synchronized over many files. The third workaround is to add placeholders to the message text that later are replaced by some HTML. That HTML is either hand written (in which case you may wonder why you started with Wicket at all), or you need to do some serious Wicket hacking to get components to render to a StringBuffer.

Now lets do this with Wicket 1.4-m1:

MyPanel.properties:
summ: You, and ${othersLink}, reviewed ${titleLink} \
 and rated it ${rate}.
others: ${otherCount} others

MyPanel.html:
<wicket:message key="summ">Text that will be replaced.
  <span wicket:id="rate">rate</span>
  <a href="#" wicket:id="titleLink">
    <span wicket:id="titleLabel">label</span></a>
  <a href="#" wicket:id="othersLink">
    <span wicket:id="othersLabel">label<</span></a>
</wicket:message>

MyPanel.java:
// Note, we directly add the embedded components
// rate, othersLink and titleLink

add(new Label("rate", new PropertyModel(summary, "rate")));

Link othLink = new Link("othersLink") { .... }
add(othLink);
othLink.add(new Label("othersLabel", new StringResourceModel(
    "others", this, new Model(summary))));

ExternalLink titleLink = new ExternalLink(
    "titleLink", summary.getTitleUrl());
add(titleLink);
titleLink.add(new Label("titleLabel",
    new PropertyModel(summary, "title")));

The html file contains a wicket:message element with some embedded components. All text within the element is removed and replaced by the text from the properties file. Labels like ${othersLink} are replaced by the rendered component of the same wicket:id. That component must be embedded in the wicket:message element. Note that the order of the components is irrellevant. Also note that you will not see any reference to the wicket:message element in the java code.

If you want runnable code you can download the complete example code (a Maven 2 project based on Wicket's QuickStart). See below for instructions.

Getting string resources from code
Despite all of the above, there always remain some cases in which you need direct access to the messages. Luckily Wicket provides access through the Localizer. You can get the localizer from any component with getLocalizer().

Again, see the complete example code for an example.

Encoding troubles
Fairly unknown to beginning programmers is that you are only allowed to use ISO-8859-1 encoding in java properties files. If you live in Europe this is a fairly annoying as many languages have characters that are not known to ISO-8859-1 (for example the euro symbol €). The simple workaround is escaping: cree\u00EBr instead of creeër. (I always use this site to look up the ISO codepoint.)

But imagine you are making a site in Thai! Luckily Wicket can also read XML property files. Here is a fragment of the Thai properties that comes with Wicket:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE properties SYSTEM "http://java.sun.com/dtd/properties.dtd">
<properties>
  <entry key="Required">ข้อมูลใน ${label} เป็นที่ต้องการ.</entry>
</properties>

Nice!

Sample code
To test the code in this article I created a small test application. Unzip it, and run mvn jetty:run to start it. When it is started access it on http://localhost:8080/i18ntest.

Changing resource settings
Finally, if you need more power, you can change all of Wicket's settings in the init() method of your application. Call getResourceSettings() to get a IResourceSettings instance. Let look at some of the options.

ThrowExceptionOnMissingResource: this will make Wicket throw an exception when a resource is missing. At first this may seem a convenient way to test that you listed all messages in a properties file. However, many standard Wicket components use defaults as fall back, so this option is mostly useless. Alternatively, watch for warnings in the log.

UseDefaultOnMissingResource: this will make Wicket use the default value (e.g. the text within the wicket:message element) when the resource is not found in a properties file. Setting this to true (the default) may hide errors for a long time, but setting this false will make your site not work if you made an error. Choose carefully.

There are many more options that allow you to change what, when and how properties are loaded. I have never found a use for these, but they are there if you need them.

Conclusion
As you have seen in this article, Wicket provides simple ways to do everything around internationalization, and some more less simple ways to completely customize this for the rare case you need it. Furthermore, the new Wicket 1.4 release will make it a lot better with support for componenta embedded in a wicket:message element. You can download the example to see everything in action.

Update 20080526: Incorperated the comment from Stefan Fußenegger on what happens if the user gives no preferred locale.

Update 20080527: Scott Swank put this article on the Wicket Wiki. Feel free to edit it there!

Update 20091203: Corrected text after Satish_j pointed out that I made a glaring mistake in the order messages are looked up in the page hierarchy.

Announcement: radiantpostfixextension release 1.0

2008-04-15T22:10:00.008+02:00

I have just released a new extension for Radiant. From the description:

With this Radiant extension you can manage a small Postfix virtual domain. A Postfix virtual domain allows you to forward e-mail to another e-mail address. The extensions adds a tab to the admin screens of Radiant in which you to enter the redirects. It also supports crude but effective aliases and mail-lists. In addition the Postfix configuration is verified and instructions will be displayed if something is not right.

Here is an screendump to see the effect.

The first release, boldly named 1.0, should work under Radiant 0.6.3 and up.

Please visit the Radiant Postfix Extension project home.

Update 2008-04-21: just released version 1.1 as a file had the wrong name.

Installing Maven under Mac OSX

2008-03-19T13:11:00.010+01:00

Update 2011-09-12: Forget everything that's below here. Just install HomeBrew and type brew install maven.

Update 2008-03-28: Instead of following this article, you are probably better of with MacPorts. Thanks for the suggestion Nino!

I did not know it could be this hard to install Maven. I've installed Maven a bunch of times already but never before on a Mac. It took me about 1 hour!

Here is what you need to do:

Download the apache-maven.tar.gz. If you follow the link on the maven download page both Firefox and Safari will get an html page but save it as a tar file. Look in the downloaded file (search for the string "apache-maven") and extract a real download link. When in doubt run md5 apache-maven-version.tar.gz on the result and compare it to the MD5 that is posted on the download page.
Extract the tar somewhere. For example with tar xzf apache-maven-version.tar.gz. Here it gives an error (tar: A lone zero block at 3083) but so far it does not seem to affect the result.
Ignore all the babbling about environment.plist files, and just create a ~/.bash_profile and put something like the following in it:
export M2_HOME=/Users/erik/apps/apache-maven-2.0.8 export M2=$M2_HOME/bin export JAVA_HOME=/Library/Java/Home/ export PATH=$PATH:$M2

That's it. Good luck!

Repeated lesson: always distrust 3th party code generators

2008-02-20T04:06:00.008+01:00

Warning: rant follows!

I am usually try to avoid generation tools. But I fell for it again. I thought that using Wicket Web Beans would speed up development of the administration screens in my current project. Indeed, the first screens were shown very quickly. But a few weeks later, it did not turn out to be so smooth as expected.

Here are some of my pain points:

You must explicitly exclude properties you do not want to see. So when you add the method boolean isPersisted() in the base class of all your entities, you must visit the meta data for each entity and add "-persisted".
Writing new types of input fields for specific properties works, but is a little clumsy.
I have no clue how to add validation. Especially when you consider the excellent vlidation support in Wicket has for this, it is kind weird that it is not clear how to do this with Wicket Web Beans.
When you have a recursive data structure, Wicket Web Beans crashes during the render phase with a StackOverflowException!
Somehow I failed to get custom input fields for collection properties.

Some positive points:

Support is good though the wwb user e-mail list. Unfortunately, I failed to provide the correct input (which is source code) as we changed the code constantly in an attempt to work around to shortcomings mentioned above. Secondly I did not want to sent our domain objects to a public list.
For simple domain models you are probably quickly set up and done.

Okay, we could have worked on Wicket Web Beans itself. After all it is open source. The code quality seems quite alright though I expect you need some time to get into it. Due to project dynamics and considering the excellent Wicket form support, I decided to pull out.

Unfortunately its now 2 days before the end of the sprint. So in 2 days we will shame ourselves in the demo before the customer with almost nothing working. Hopefully we can do better next week with regular Wicket forms.

Note: 'me too' comments are likely to be removed. I am also very interested in success stories!

Update, a couple of hours later: I reworded the article a bit as I found the first version too harsh.

I finally made up my mind

2008-01-21T19:25:00.000+01:00

Those who read this blog more often may have noticed the minor change in the sub-title. It now reads:

Experiences from a hard core Java programmer that finally decided Ruby is a very very nice language, but rather stays on the JVM.

Compare it to the previous text:

Experiences from a hard core Java programmer that struggles with the question of switching to the world of Ruby and Ruby on Rails.

So how did I come to this conclusion? Here are some random considerations:

I have only once in the last 2 years used Ruby professionally (to do some counting in a site dump).
I do have 3 Ruby sites running (one Camping and two using Radiant). So I am not going to stop using Ruby anytime soon.
ActiveRecord totally kicks ass and is hugely impressive. Rails for the rest not so much (totally old school request response based, with like JSP code in the templates). It may be just my lack of experience, so don't base your opinion on this.
I am very addicted to correct refactoring and code completion.
As I wrote before, we now have Scala.
Update 2008-01-23: This one needs to be added. Since I recently became a father, it is simply too much to keep track of 2 worlds at the same time.

So if you expected me to be on amsterdam.rb right now, here is why.

GWT in action - Book review

2008-01-21T19:10:00.001+01:00

I recently joined JTeam, a self declared GWT specialist. Of course I do not want to stay behind so I started reading GWT in Action.

That was a mistake. There is nothing wrong with GWT, I think it is a very nice component oriented technology in the spirit of Swing and Wicket, which makes it a lot easier to build very complex applications.

But back to the book. Reading the book is a lot like looking at an extremely knowledgeable professor. The kind of professors that know so much that they don't know where to begin. And like with the professor you will eventually get it if you hang on and are able to sort it all out in your head.
Well, I was fed up with it in chapter 5, half way through the 17 chapter book. Here are my mischiefs: 1) the text was constantly referring to chapters further away in the book, 2) it was continuously repeating itself (I had to read at least 4 times that it is okay to use something else then Eclipse), but worse, 3) the text was almost continuously on a different conceptual level then the reader could be. For example implementation details are mentioned (for example the event_bits attribute in the generated HTML) while they have nothing to do with making you understand the API (writing an event callback class). Or the other way around, at places where a simple examples would do wonders there is only some broad goal description.

Since there are many other books on GWT, you are probably better of with another one.

JRuby on Rails - Book review

2007-12-27T19:42:00.001+01:00

So what do you put in a book on JRuby? After all, for the programmer JRuby is 'just' another Ruby implementation. The question must have come up when Ola Bini decided to write this book. The result is materialized in Practical JRuby on Rails Web 2.0 Projects - Bringing Ruby on Rails to the Java Platform. The title is well chosen and also answers the question: JRuby on Rails is a practical book; it guides you through the implementation of several Rails sites and along the way it shows you neat tricks that are only possible in JRuby.

Contents
The first half of the book is all about JRuby on Rails. If you already know Rails it is funny to consistently see jruby instead of ruby, but otherwise this is part is boring for the initiated. If you do not know Rails and like to learn from examples, this chapter is well worth your read.

Right after that is an interesting part for all us that want to mix Java and Ruby.

Later in the book, Ola goes through the trouble of explaining how to use JRuby with all the non interesting things in Java: session beans, message beans, JMX, XML processing and SOAP. I have been in Java business for 7 years and have been luckily able to avoid these for most of the time. The integration provided by Spring could have been a very nice replacement for these subjects.

A bit hidden between this stuff, the book again shows it practicality and goes through all the options of deploying JRuby on Rails applications.

The book ends with some convenient JRuby specific references.

Conclusion
I strongly recommend this book when you are a Java programmer (of any skill) that knows some Ruby and want to start working with (J)Ruby on Rails. If you know Rails well, but little Java and you want to start with JRuby on Rails, the book is probably too heavy and will teach you only a few useful things. Do not start on this book with zero Ruby knowledge. Though appendix A helps, for seriously learning Ruby, the old Pickaxe (dead tree version) is an excellent first read.

WebBeans, the JSF cover up

2007-12-16T18:00:00.002+01:00

At JavaPolis I saw very good presentation on WebBeans by Bob Lee. Bob did a very good job in explaining the concepts while thankfully still giving lots of code examples. WebBeans provides a way to use ordinary Java beans in the JSF environment. You do this with annotations.

Praise for WebBeans, and its main predecessor Seam; they really make developing JSF applications simpler. In particular, the conversation scope is a brilliant invention. But how simple does it get? Well lets see. For every aspect that can be managed there is an annotation. In one of the presentation examples I counted about 5 lines of ordinary Java code plus at least 12 annotations! Now there are ways to group annotations, but you do this by introducing more annotations! So instead of having to deal with JSF stuff, you now how to cook annotation soup.

I have always hated JSP programming, and now that I see that even talented people like Bob Lee and Gavin King are wrestling to fix JSF, I finally know why. The problem is that the whole idea of having contexts to store page data is flawed. It breaks all kinds of encapsulation rules and breaks the beloved type safety.

So what are the alternatives? Actually there are a few very nice web frameworks that do not use contexts. The one I am most familiar with is Wicket. Another often named contender is Tapestry, but of course there are many more. Wicket, like Swing, uses a component tree to build pages. Each component is completely responsible for its own markup (html) and data (the model). No encapsulation rules are broken. Another addition is GWT, not exactly a web framework, but nevertheless useful for making componentized web applications.

The demise of Java, long live the JVM!

2007-12-16T15:37:00.001+01:00

The Java language is dying. Most people do not yet realize this yet, but it is inevitable. What do I mean by dying? Why do I think it is dying, and why is there lots of hope?

Dead languages
So what do I mean by dead? First of all, languages do not die just like that. There are still people programming in Cobol, and likewise, there are still people that can read old Greek fluently. But just like Cobol and every other programming language, Java will at some time be left to the dinosaur programmers. I think that Java is currently at its peak (or will soon be), and will go downwards from here. It will take some time, but the signs are there.

The signs
First of all, there is the rise of dynamic languages. They have been around for quite some time, but all of sudden there was Ruby on Rails. Many Java programmers bit the bullet and switched to be rewarded with a up to 10 times productivity boost. There are those that say this can be attributed to dynamic typing. I believe this is not the case. I think the boost is possible because the Java language is quite weak compared to Ruby; the same problem can be expressed in a lot less lines of Ruby then lines of Java. This has nothing to do with typing, but everything with how the language is structured.

Secondly, there is the way Java is extended over time. The extensions have been done with great care. This has paid off: Java is still mostly a clean and simple language. Even with the tiny changes, each new version brought its problems. The largest change so far has been the introduction of generics. I see generics as an improvement, but as I wrote earlier, they can be a pain to use.
And now there is the closures debate. Some people (like James Gosling) want to have them at every price. Others (like Joshua Bloch) say that the complexity budget has been used and that Java is not ready for big changes like closures. I agree with the latter.

Despite the care with which Java was and is being extended, the possibilities for doing so are reaching zero rapidly.

There is hope!
If there is one gem coming out of the Java world, it is the Java Virtual Machine. With each release the JVM has brought performance improvements. And this is likely to continue for the next versions. The JVM is still climbing and is nowhere near its peak. This can be proved by looking at the many languages that now run on the JVM. Everything from PHP, Lisp, Cobol to Ruby and Python. Some of these (in particular Ruby) are very well supported by several IDEs.

The Java language does not need closures or other more advanced stuff, there are many languages on the JVM that already provide those things: Ruby, Python, Lisp and Scala.

Yes, now there is Scala! Scala is a language for both the JVM and the CLR. Scala's syntax is as concise as Ruby's. It provides a very smooth transition from the Java language, but nevertheless is a complete functional language while at the same time staying purely object oriented and statically typed. And, as I learned at Javapolis, its library is already excellent, and even better: without any particular optimizations Scala programs outperform Java programs on the same JVM! What is not to like about Scala?

Disclaimer
Many statements in this article are based on personal observations and anecdotal evidence.

Update 2007-12-24: Despite the tone of this article, I am still writing Java and foresee to do so for quite a bit longer. And note: having fun doing it!

Double-Checked Locking found in JVM

2007-11-22T10:34:00.002+01:00

I just found an implementation of the broken double checked locking in the java 6 runtime library!

Here is a snippet of the offending code:

package java.lang;
import java.util.Random;
public final class Math {
  private static Random randomNumberGenerator;
  private static synchronized void initRNG() {
    if (randomNumberGenerator == null) 
      randomNumberGenerator = new Random();
  }
  public static double random() {
    if (randomNumberGenerator == null) initRNG();
    return randomNumberGenerator.nextDouble();
  }
}

Amazingly, it used to work correctly, but the extra synchronization was removed in java 1.3. You can track the progress of this bug in report 6470700 and report 6633229.

Howto extend LDAP in java with JLDAP

2007-11-16T08:51:00.002+01:00

LDAP is a protocol that is wonderfully extensible. You can augment existing messages by adding 'controls', and you can define complete new messages. Extensions are identified by a universal OID, so that even code that does not know about an extensions can still work properly. For this each extension has a criticality flag to indicate whether the receiver may ignore unknown extensions. As a bonus, the content of controls and messages are all defined by a common syntax (ASN.1) and common encoding (which is BER with restrictions).

Writing you own controls and messages is a kind of under documented thing. In addition, not all LDAP libraries support all kinds of messages. In this small howto I show how to implement custom controls and messages based on my experiences with implementing a RFC4533 (synchronization) client. I used JLDAP as it is the only java library I could find that supports IntermediateMessages, a requirement for RFC4533. And before you ask: no sorry, I can not open source the results.

In case you actually want to start with JLDAP, you get a lot of knowledge from at the examples that are provided by Novel. You can find them through the JLDAP site. The javadoc is also useful at times.

Decoding a control

Lets take a look at the SyncDoneControl from RFC4533. The control's OID is 1.3.6.1.4.1.4203.1.9.1.3 and its content value is defined with ASN.1 as:

syncDoneValue ::= SEQUENCE {
  cookie          OCTET STRING OPTIONAL,
  refreshDeletes  BOOLEAN DEFAULT FALSE
}

Read this as: the value is a sequence that contains 2 other values. The first, named cookie is optional and has binary content. The second is named refreshDeletes and has boolean content. The default of refreshDeletes is false. See RFC4533 for the semantics.

Lets map this to java. All controls must have the same constructor signature so that JLDAP can instantiate it. We'll start with:

public class SyncDoneControl extends LDAPControl {
  public static final String OID = "1.3.6.1.4.1.4203.1.9.1.3";
   
  private byte[] cookie;
  private boolean refreshDeletes;
  // add getters for cookie and refresDeletes
   
  public SyncDoneControl(String oid, boolean critical, byte[] value) {
    super(oid, critical, value);
    ...see below
  }
}

The byte array value contains the BER encoded value of the control. The LDAP restriction put on the BER encoding mean that optional values and values that are equal to the default value must be omitted. With other words: when there is no cookie (allowed because it is declared OPTIONAL), and refreshDeletes is FALSE (which is the default), constructor argument value is null! Just to be robust we'll check for the empty array as well:

  if (value == null || value.lenght == 0) {
    cookie = null;
    refreshDeletes = false;
  } else {
    ...see below
  }

If it is not null/empty, we'll use the decoder as provided by JLDAP to decode the bytes. As the ASN.1 value is defined to start with a SEQUENCE (one of the native ASN.1 types), the LBERDecoder will instantiate an object of type ASN1Sequence:

ASN1Sequence asn1 = (ASN1Sequence) new LBERDecoder().decode(value);

The decoder can decode all native ASN.1 types. These native types are called "universal". Other important universal types are BOOLEAN, OCTET STRING, CHOICE and SET. Type information is available on every ASN1Object through the ASN1Object#getIdentifier() method.

We can examine the sequence further by calling the ASN1Sequence#size() and ASN1Sequence#get(int) methods. Again, we must take into account that each element may be omitted. You can do this by examining the type of ASN.1 value you get out of the sequence. First, extract a value from the sequence:

ASN1Object asn1Obj = asn1.get(0);

When this is the cookie, the value must be from the type-class UNIVERSAL, with as type OCTET STRING:

boolean isCookie =
   asn1Obj.getIdentifier().getASN1Class() == ASN1Identifier.UNIVERSAL &&
   asn1Obj.getIdentifier().getTag() == ASN1OctetString.TAG;

If it is, we can safely cast the object to an ASN1OctetString and extract the cookie:

cookie = ((ASN1OctetString) asn1Obj).byteValue()

We can do the same for the value refreshDeletes and JLPAP class ASN1Boolean. After we have moved this very verbose code to the utility class Asn1Util (exercise for the reader) we'll get the following code:

    ASN1Sequence asn1 = (ASN1Sequence) new LBERDecoder().decode(value);
    for (int i = 0; i < asn1.size(); i++) {
        ASN1Object asnSeqObj = asn1.get(i);
        if (i == 0 && Asn1Util.isOctetString(asnSeqObj)) {
            cookie = Asn1Util.getByteValue(asnSeqObj);
        } else if (i == (cookie == null ? 0 : 1) && Asn1Util.isBoolean(asnSeqObj)) {
            refreshDeletes = Asn1Util.getBooleanValue(asnSeqObj);
        } else {
            throw new IllegalArgumentException("Parse error at index " + i + ", parsing: " + asnSeqObj);
        }
    }

Tada! Your first JLDAP extension. All we have to do is make JLDAP aware of the extension and it will be parsed automatically when the control is present in a received LDAP message.

LDAPControl.register(SyncDoneControl.OID, SyncDoneControl.class);

One small warning: when there is an exception in the control's constructor, JLDAP will silently ignore your class and do its default thing.

Encoding a control

To start a sync operation, one must add a SyncRequestControl to the search constraints. Here is the ASN.1 definition of the control value:

syncRequestValue ::= SEQUENCE {
  mode ENUMERATED {
      refreshOnly       (1),
      refreshAndPersist (3)
  },
  cookie     OCTET STRING OPTIONAL,
  reloadHint BOOLEAN DEFAULT FALSE
}

First the ASN.1 enumeration is translated into a Java enumeration:

public enum SyncRequestMode { REFRESH_ONLY, REFRESH_AND_PERSIST }

The we'll start the control with

public class SyncRequestControl extends LDAPControl {
  public static final String OID = "1.3.6.1.4.1.4203.1.9.1.1";
  private SyncRequestMode mode;
  private byte cookie[];
  boolean reloadHint = false;

As we will construct this control ourself, and not JLDAP, we can give it any constructor we like. For example:

public SyncRequestControl(SyncRequestMode mode, byte cookie[], boolean reloadHint) {
  super(OID, true, null);
  this.mode = mode;
  this.cookie = cookie;
  this.reloadHint = reloadHint;
  setValue(encodedValue());
}

In the last line we set the BER encoded value. Here is a complete implementation of the encode method. Note how we follow the ASN.1 definition, but skip optional values and values that have the default value.

private byte[] encodedValue() throws IOException {
  ASN1Sequence asn1 = new ASN1Sequence();
  asn1.add(new ASN1Enumerated(mode == REFRESH_ONLY ? 1 : 3));
  if (cookie != null) {
    asn1.add(new ASN1OctetString(cookie));
  }
  if (reloadHint) {
    asn1.add(new ASN1Boolean(reloadHint));
  }
  ByteArrayOutputStream baos = new ByteArrayOutputStream();
  new LBEREncoder().encode(asn1, baos);
  return baos.toByteArray();
}

More complex example, decoding a message

The JLDAP BER decoder can only decode ASN.1 universal types. As soon as you define your own types, you must help the decoder. Lets look at the decoding of the SyncInfoMessage to see how this works. The value of SyncInfoMessage is defined with the following ASN.1:

syncInfoValue ::= CHOICE {
   newcookie      [0] OCTET STRING,
   refreshDelete  [1] SEQUENCE {
        cookie         OCTET STRING OPTIONAL,
        refreshDone    BOOLEAN DEFAULT TRUE
    },
    refreshPresent [2] SEQUENCE {
        cookie         OCTET STRING OPTIONAL,
        refreshDone    BOOLEAN DEFAULT TRUE
    },
    syncIdSet      [3] SEQUENCE {
        cookie         OCTET STRING OPTIONAL,
        refreshDeletes BOOLEAN DEFAULT FALSE,
        syncUUIDs      SET OF OCTET STRING (SIZE)16))
    }
}

The ASN.1 defines that the value can have one or four values. We'll represent the chosen value with a Java enumeration. By defining the enum values in order, we can abuse that the ordinal value of the enum values corresponds to the tag (defined between brackets []) value.

public static enum SyncInfoMessageChoiceType {
  // Note: order is important
  NEW_COOKIE, REFRESH_DELETE, REFRESH_PRESENT, SYNC_ID_SET
}

As SyncInfoMessage is an intermediate response, we'll start the message implementation as:

public class SyncInfoMessage extends LDAPIntermediateResponse {
  public static final String OID = "1.3.6.1.4.1.4203.1.9.1.4";
  private SyncInfoMessageChoiceType syncInfoMessageChoiceType;
  private byte[] cookie;
  private Boolean refreshDone;
  private Boolean refreshDeletes;
  private List syncUuids;
  // add getters ...

Not all fields will always get a value. For example field syncUuids will only be set when syncInfoMessageChoiceType == SYNC_ID_SET. This is the most simple implementation, and the user of this class must know about the CHOICE type anyway.

Intermediate messages must always have the same constructor, so that JLDAP can construct it for us:

public SyncInfoMessage(RfcLDAPMessage message) {
  super(message);
  ...

The choice is represented by an instance of type ASN1Tagged. The identifier of the tag indicates the choice. Instantiate field syncInfoMessageChoiceType so:

ASN1Tagged asn1Choice = (ASN1Tagged) new LBERDecoder().decode(getValue());
int tag = asn1Choice.getIdentifier().getTag();
syncInfoMessageChoiceType = SyncInfoMessageChoiceType.values()[tag];

Now comes the tricky part. As JLDAP has no clue about the ASN.1 definition, it does not know about the choice, and it can not decode any further. What we can do is get the contents of asn1Choice as an OCTET STRING, get its byte array, and decode that again with the JLDAP decoder.

So for most choice types we need to decode the tag's contents to a SEQUENCE. Here is a utility method we can add to the AsnUtil class:

public static ASN1Sequence parseContentAsSequence(ASN1Tagged asn1Choice) throws IOException {
  ASN1OctetString taggedValue = (ASN1OctetString) asn1Choice.taggedValue();
  byte[] taggedContent = taggedValue.byteValue();
  return new ASN1Sequence(new LBERDecoder(), new ByteArrayInputStream(taggedContent), taggedContent.length);
}

With this tool we'll decode the choice refreshPresent. Notice how we decode the contents of the tag, and how refreshDone is set to its default value when we did not see it in the sequence.

if (syncInfoMessageChoiceType == SyncInfoMessageChoiceType.REFRESH_PRESENT) {
  ASN1Sequence asn1Seq = Asn1Util.parseContentAsSequence(asn1Choice);
  for (int i = 0; i < asn1Seq.size(); i++) {
    ASN1Object asnSeqObj = asn1Seq.get(i);
    if (i == 0 && Asn1Util.isOctetString(asnSeqObj)) {
      cookie = Asn1Util.getByteValue(asnSeqObj);
    } else if ((i == (cookie == null ? 0 : 1)) && Asn1Util.isBoolean(asnSeqObj)) {
      refreshDone = Asn1Util.getBooleanValue(asnSeqObj);
    } else {
      throw new RuntimeException("Parse error");
    }
  }
  if (refreshDone == null) {
      refreshDone = Boolean.TRUE;
  }
}

When the choice is newCookie things are a bit simpler. The content of asnChoice is already an OCTET_STRING, so we can use that directly:

if (syncInfoMessageChoiceType == SyncInfoMessageChoiceType.NEW_COOKIE) {
  ASN1OctetString taggedValue = (ASN1OctetString) asn1Choice.taggedValue();
  cookie = taggedValue.byteValue();
}

Again, we have to make JLDAP aware of the new message. it will be parsed automatically when the control is present in a received LDAP message.

LDAPIntermediateResponse.register(SyncInfoMessage.OID, SyncInfoMessage.class);

Conclusion

In this article I showed how to get started with extending JLDAP. The example are functional but not always complete. Nor are they always according to best practices (for example, I would normally not declare so many exceptions). I shared some of the pitfalls you will encounter when encoding and decoding messages and controls.

Getting started with Ruby on Ubuntu 7.10

2007-10-29T20:57:00.010+01:00

Update 20100922: Apparently this article is so out of date that I even get e-mail to get this article down (or replace it).

Therefore: please do not use this article, in fact don't even rely on the Ubuntu packages as they are not all maintained properly.

One approach that may be fine (I did not try, don't blame me), is on http://krainboltgreene.github.com/l/3/. Good luck!

So now I have a new server, and a new 10Mbit/s internet connection. Its time for some applications! Here are my experiences with installing Rails, Radiant and Camping on a fresh Ubuntu 7.10 server installation.

First of all, if your server is running in a closet somewhere (like mine) or worse: in a remote data center, you need to disable apt-get asking for the installation CD. Edit /etc/apt/sources.list and comment the line that refers to the installation cdrom.

The next step of course it to install Ruby and Ruby Gems: sudo apt-get install ruby rubygems ruby1.8-dev Without the ruby1.8-dev you can not do much, don't forget it!

Where would you be without irb? The default install however does not link it! sudo ln -s /usr/bin/irb1.8 /usr/bin/irb

Any serious Ruby web application uses Mongrel. However, Mongrel compiles some of its stuff during the install. So you first need to get the compilation tools:
sudo apt-get install make gcc libc6

Mongrel is a trusted enterprise application nowadays, so you can install it with a certificate:

wget http://rubyforge.org/frs/download.php/25325/mongrel-public_cert.pem
gem cert --add mongrel-public_cert.pem
rm mongrel-public_cert.pem
gem install mongrel --include-dependencies -P HighSecurity

Select the highest version followed by (ruby).

Luckily, installing Rails is still a simple: sudo gem install rails --include-dependencies

If you want Camping, you'll first need to install sqlite: sudo apt-get install libsqlite3 libsqlite3-dev

Now you can:

sudo gem install camping --source http://code.whytheluckystiff.net
sudo gem install camping-omnibus --source http://code.whytheluckystiff.net

Radiant is a simple: sudo gem install radiant

For some reason the gem executables do not get added to the path. Make sure they do by adding /var/lib/gems/1.8/bin to your path. For example by adding the following line at the end of /etc/bash.bashrc: export PATH="$PATH:/var/lib/gems/1.8/bin".

The favorite database amongst Railers is of course MySQL. I assume you selected LAMP during the Ubuntu server installation, that means you already have MySQL installed. To use MySQL with ActiveRecord, its best to install the MySQL native adapter. Before you can install that, you first need to install all the compile stuff from above and some more: sudo apt-get install libmysqlclient15-dev sudo gem install mysql Again, for the gem select highest version number followed by (ruby).

Have fun!

How to really test RAM, or My search for system stability

2007-10-24T14:34:00.001+02:00

I recently bought a Mac because my previous system kept crashing. It would just beep, and reboot without a visible cause. Since I still wanted to use the old system for my Linux firewall, I needed to find out what the culprit was.

Over time I found out that:
- Reboots started intermittently after placing new memory. However, Memtest86 reported no problems whatsoever.
- Over time the reboots occurred more and more often.
- The problem occurred more often during heavy disk activity, sometimes after only 2 minutes. I could not even finish a long Cygwin install session.
- According to SpeedFan my processor heated up quite a bit (up to 65ºC), after putting a bit of heat sink paste between the CPU and the heat sink, that problem was solved.
- The same SpeedFan reported that my harddisk went to high temperatures as well, reaching 50ºC but still rising when the system went down. Some searches taught me that this is high but acceptable. A full copy of the harddisk (as a USB drive) did not give any problems.
- Replacing the power unit did not help.

When I had moved everything to another motherboard an interesting thing happened: once, just once out of many reboots I got a memory failure. Got you!

I finally was able to pin the wrong RAM module using an old memory test from Doug Ledford.

Since the shown script can not be used as is, here is what I did to make it work on Ubuntu 7.10:
- Download a Linux kernel from kernel.org (we are not going to compile a kernel, we just need a large zip file): wget http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.23.1.tar.bz2
- Transform it to a gzipped tar:
bunzip2 linux-2.6.23.1.tar.bz2 gzip linux-2.6.23.1.tar cp linux-2.6.23.1.tar.gz /tmp
- Download the adapted memtest.sh. My changes auto-detects files named linux-*.tar.gz, and uses the file name to predict the name of the root folder in the tar.
- One by one place a memory module in your computer and run memtest.sh for each configuration.

The original memtest.sh site has more information on how the script works and why Memtest86 is actually useless. The point is that a modern CPU can not put enough load on your memory. With concurrent DMA transfers more errors are detected.

Help! Looking for a Java LDAP client library!

2007-10-23T10:28:00.001+02:00

I often read that Java is very mature and that you can find Java libraries for everything. Wrong!

At least 5 complete working days I have spend investigating how to implement the LDAP synchronization protocol as supported by the OpenLdap server (RFC4533) in our Java product. Here are the libraries I investigated. None of them support RFC455 out of the box.

JNDI
Sun's implementation is alright for most things, and now that the JVM is being open sourced, you can actually see the com.sun classes you need to program with (BerEncoder and BerDecoder) for new LDAP controls. Unfortunately, JNDI is not actively developed anymore. As far as I can see the required LDAP Intermediate Response Messages (RFC4511, the most recent definition of LDAP) is not supported. No idea on how to add this to JNDI either.

OpenDS
Another Sun initiative, the open source directory server. The code shows support for the LDAP Intermediate Response Message however, all code is written from a server perspective. I did not see how this code could be used in a client.

ApacheDS
The code of this directory server has clearly separated code that could be used by both client and server. However, again, no support for the Intermediate Response Message.

JLDAP
I could not download the sources as our firewall does not allow CVS to go through. I'll investigate later. I doubt that Intermediate Response Messages are supported as there is not much development going on here. I think Novel's resources are all tied to their newer products.

So, still no go. What should I do? Any synchronization supported by OpenLdap will do. Help!

Update 2007-10-23: It seems that JLDAP has support for Intermediate Response Messages after all! Meanwhile a friendly colleague at another location has downloaded the code for me.

Announcement: Version 99 Does Not Exist

2007-10-10T13:09:00.012+02:00

Last update: 2011-12-19

Update 2011-08-15: DNS for Version 99 is offline.

In a previous post I announced no-commons-logging (a.k.a. commons-logging version 99.0-does-not-exist). After a request to add no-commons-logging-api I immediately realized that this can be generalized. So here it is: Version 99 Does Not Exist.

Features
Version 99 Does Not Exist emulates a Maven 2 repository and serves empty jars for any valid package that has version number 99.0-does-not-exist. It also generates poms, metadata files and of course the appropriate hashes.

For example the following links will give an empty jar, its pom and the maven metadata for commons-logging.

Why?
To get rid of dependencies that maven 2 insists on including on your classpath (like commons-logging while you want to use jcl-over-slf4j).

How do you use it?
First of all: if you were using no-commons-logging before, you do not need to change anything! Version 99 Does Not Exist is fully backward compatible with no-commons-logging. Otherwise, read on.

In your pom.xml declare the following 2 things: 1) the Version 99 Does Not Exist repository, and 2) for each jar that you get but do not want, declare a dependency with version 99.0-does-not-exist.

So, for example, if you do not want to be bothered with commons-logging, include the following in your pom.xml:

<repositories>
    <repository>
        <id>Version99</id>
        <name>Version 99 Does Not Exist Maven repository</name>
        <layout>default</layout>
        <url>http://no-commons-logging.zapto.org/mvn2</url>
    </repository>
</repositories>
<dependencies>
    <!-- get empty jar instead of commons-logging -->
    <dependency>
        <groupId>commons-logging</groupId>
        <artifactId>commons-logging</artifactId>
        <version>99.0-does-not-exist</version>
    </dependency>
</dependencies>

When? Right now. As I received questions on how stable this service will be, I hereby promise to keep this maven repository on-line for at least 5 years (or until no-ip stops offering their free dns service). Read further if you want to run Version 99 Does Not Exist yourself.

How?
Version 99 Does Not Exist is implemented in a single file as a Camping application.

Where is the code?
If you want to run it yourself you'll need the following: Version 99 Does Not Exist download (rb file, 4Kb, MIT license), Ruby, Ruby Gems and Camping. Version 99 Does Not Exist is started with a simple camping version99.rb. Good camping!

Update 2007-11-06: The empty jar is no longer completely empty as 'mvn site' fails on it. Thanks to Stefan Fußenegger for the report.

Update 2007-12-14: The repository was off line for a day or so because I goofed while switching internet provider. Now, it is not only working again, but reachability is better then ever: 10 Mbit/s up and down!

Update 2008-02-09: Version 1.2: another update to the empty jar. Sasha Ovsankin reported that the compiler could not open it. The jar now contains a valid manifest. Thanks Sacha!

Update 2009-05-01: During my move from Amsterdam to Haarlem the server has been off line for a day or so. I am now on ADSL so my internet connection is a lot slower. If anyone want to run a mirror, I am happy to set up a rotating DNS.

Update 2009-07-24: Version 1.3: artifacts with a groupid that contain a dot are now supported. Éric Vigeant, thanks for the bug report!

Update 2009-10-17: Version 2.0: Éric Vigeant's problems were still not over. Now removed metadata support, this will hopefully make some proxies behave better.

Update 2011-08-15: DNS for Version 99 is offline.

Update 2011-12-19: I just found an alternative with almost the same name: version99.qos.ch. It is a static maven repository with a limited number of version 99 jars.

Watch where you place those Camping constants!

2007-10-08T15:35:00.001+02:00

Finally got that bastard Camping application to work properly. I kept getting a ERROR: wrong number of arguments (0 for 3) on the first request. Subsequent requests would sometimes succeed, sometimes not. The funny thing was, the get method was not even called yet!

With deep gratitude for ruby guru Remco van 't Veer who found out that you can not use constants in a Camping Controller (actually, I already knew it from a previous camping app, but I had stupidly forgotten it). Move it up to the main module and you're good to go.

Wrong

Camping.goes :Nowhere
module Nowhere
  module Controllers
    MY_CONSTANT = 'wrong'   # Camping going nowhere
  end
end

Correct

Camping.goes :ForPresident
module ForPresident
  MY_CONSTANT = 'fine'
  module Controllers
  end
end

See my next post to see what this is all about.

Camping: optional group in regular expression

2007-10-01T20:48:00.000+02:00

Recently I finished a Camping application where I wanted to handle URIs with an optional part. Like so:

class Jar < R '/(.*)\.jar(\.sha1|\.md5)?'
  def get(jarname, hash)  # does not work
    ...
  end
end

Unfortunately this does not work. (Does this look familiar? See my next post!) In the end it turned out to be as simple as:

class Jar < R '/(.*)\.jar(\.sha1|\.md5)?'
  def get(jarname, hash = nil)
    ...
  end
end

At RailsconfEurope Manfred taught me another trick:

n = 4
class DigitParty < R "/" + ("(\\d)?" * n)
  def get(*args)
    args.length   # -> 4
    args[0]       # -> first digit or nil
    args[n - 1]   # -> last digit or nil
  end
end

This matches anything from 0 to n digits where a group is created for each digit. Using * in the parameter list will make method get work for any value of n. For URI's with less then n digits, the corresponding array elements are nil.

Multi user mac?

2007-09-05T14:30:00.000+02:00

Well who would have thought? I have become a Mac user. After seeing Windows Vista at my moms computer, I decided I had enough of MS. And I love it! No noise, very useful software included, very quick to start and stop, and my girl-friend and I are simply always logged in. Switching takes 2 clicks and 2 seconds. Then an idea struck me. I still have a spare monitor and attaching a second keyboard and mouse is not so hard either: why can we not work simultaneously? Unfortunately this is not supported. But why not? My question of the month: Why is the Mac still a single user system?

Having fun with Mule

2007-08-01T11:13:00.001+02:00

Mule, a very nice open source ESB implementation kept me busy in a not very nice way for 2 days.

In the Mule configuration you can inject properties that are created from another containers, in my case: Spring. You do so with the <container-property>. However, it seemed that these properties were not set on the services that need them. A full day of debugging Mule configuration parsing code gave no results. The properties were correctly loaded from the Spring container but somehow they were not used.

When we found out that another deployment unit of our project did work correctly we started comparing the configurations. It turned out that the configuration was not at all the problem. The problem was a rouge spring bean.

In my setup Mule is started by Spring. From the Spring configuration you simply instantiate Mule with some bootstrap code as follows:

<bean id="muleManager" class="org.mule.extras.spring.config.SpringMuleBootstrap">
  <property name="configResources" >
    <list>
      <value>classpath:mule-config.xml</value>
    </list>
  </property>
</bean>

Unfortunately, one of the other beans that are created by Spring instantiates a MuleClient in its constructor. The new MuleClient starts a Mule core (the MuleManager) if it can not find a running one. I guess you see the problem, a bit later the bootstrap code creates another MuleManager, but by now it is too late and somehow all the container properties get ignored.

Update 2007-08-01: More container property problems: the first transformer gets injected nicely. Somehow another instance is used later on. Not injected with container properties of course.

No more commons-logging

2007-07-19T10:32:00.003+02:00

Update 2007-10-10: No-commons-logging has been superseded by (backward compatible) Version 99 Does Not Exist.

Disclaimer
COMMONS-LOGGING VERSION 99.0-does-not-exist IS NOT IN ANY WAY AFFILIATED WITH THE ORIGINAL DEVELOPERS OF COMMONS-LOGGING NOR WITH APACHE.

Why no-commons-logging?
If you are using Maven you'll know it is practically impossible to move away from commons-logging (with its class-loading problems) and migrate to SLF4J. About every second pom declares it is dependent on commons-logging. Unfortunately Maven does not provide an easy way to exclude a certain package throughout your project. You will have to exclude commons-logging on each and every dependency you need (including transitive dependencies).

No-commons-logging is a Maven hack that allows you to exclude commons-logging from your application with a single piece of configuration.

How does no-commons-logging work?
No-commons-logging is a Maven2 package that mimics a commons-logging package with a high version number, but without any actual java code in the jar. This trick works because Maven allows you to specify a specific version for a dependency, and that version will then be used regardless of other dependency specifications.

Update 2007-07-22: Added package to mimic commons-logging-api as requested by Olivier Lamy.

How do you use no-commons-logging?
In your pom.xml include the following piece of xml:

<repositories>
    <repository>
        <id>no-commons-logging</id>
        <name>No-commons-logging Maven Repository</name>
        <layout>default</layout>
        <url>http://no-commons-logging.zapto.org/mvn2</url>
    </repository>
</repositories>
<dependencies>
    <!-- use no-commons-logging -->
    <dependency>
        <groupId>commons-logging</groupId>
        <artifactId>commons-logging</artifactId>
        <version>99.0-does-not-exist</version>
    </dependency>
    <!-- no-commons-logging-api, if you need it -->
    <dependency>
        <groupId>commons-logging</groupId>
        <artifactId>commons-logging-api</artifactId>
        <version>99.0-does-not-exist</version>
    </dependency>
    <!-- the slf4j commons-logging replacement -->
    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>jcl104-over-slf4j</artifactId>
        <version>1.4.2</version>
    </dependency>
    <!-- the other slf4j jars -->
    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-api</artifactId>
        <version>1.4.2</version>
    </dependency>
    <!-- using log4j as backend -->
    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-log4j12</artifactId>
        <version>1.4.2</version>
    </dependency>
    <dependency>
        <groupId>log4j</groupId>
        <artifactId>log4j</artifactId>
        <version>1.2.14</version>
    </dependency>
</dependencies>

Disclaimer
COMMONS-LOGGING VERSION 99.0-does-not-exist IS NOT IN ANY WAY AFFILIATED WITH THE ORIGINAL DEVELOPERS OF COMMONS-LOGGING NOR WITH APACHE.

(Pre-)announcement: transport-smtpin, an embedded e-mail server for Mule

2007-07-13T16:24:00.001+02:00

I am proud to (pre-)announce the open source project: transport-smtpin. Transport-smtpin acts as an embedded e-mail SMTP server for the Mule ESB implementation. Under the cover transport-smtpin uses SubEthaSMTP Mail Server (the last freaking java SMTP implementation).

I'll post a download location as soon as I am convinced the implementation is stable (should not take long). Please let me know if you are interested. It will be released under something like the Mozilla Public License Version 1.1.

Update 2007-07-16: This code was sponsored by Uzorg BV.

Update 2007-07-17: A project proposal was made on MuleForge.

Update 2007-8-12: The project has been accepted by MuleForge: mule-transport-smtpin project page. There is no code there yet. It seems to work for me, but not all features are thoroughly tested, nevertheless I'll try to upload the code this week so that more people can play with it.

A bit of history
Transport-smtpin came about when I was searching for a way to directly receive e-mails. In the Dutch health-care a much used standard (mis-)uses e-mail as a request-response protocol: the OZIS standard. The actual messages are in an attachment and are defined by one of many Edifact variants.

In the system I am currently working on, there is actually a person waiting for the answer in such a OZIS request-response cycle. So receiving the responses by polling a POP server would incur an unacceptable overhead.

In an earlier attempt to get e-mail directly we used Apache James. Our mailet installed in James passes received e-mails immediately to our ESB with a Hessian web service. Though writing a mailet is quite easy, this set-up has some serious disadvantages:

Installing and configuring James for this task is not easy nor straightforward.
In a cluster, we need to either install one James per cluster node, or devise some way to redirect e-mails to the right cluster node.

In an attempt to get rid of James I stumbled upon SubEthaSMTP Mail Server. SubEthaSMTP is a quite easy to use SMTP server implementation.

As we are using Mule, it would be neat if SubEthaSMTP could be configured as a transport. It took me 3 days (mostly to read Mule documentation), but the result is there: my very first open source project.

Making a site within 1 hour

2007-07-06T21:20:00.000+02:00

Pardon my little pause on this blog. I just got a son: Milo. Feel free to look at Milo's website. I build it with Radiant. It took me, including the first content, less then 1 hour!

Wicket for BSCs, part II

2007-05-24T22:13:00.000+02:00

So you are in a Big Slow Company (BSC) and want to switch to Wicket. I wrote about this before, but today's e-mail from Mark Stock on the Wicket user list backs up the arguments with some hard figures. Here is an abstract with some key phrases:

In my experience, it's taken me about two weeks to get up to speed on Wicket. ..... Now, the prototype that I have so far would have probably taken me at least two weeks in Struts 1 and I already know Struts 1 very well. The difference in productivity between the two frameworks is pretty dramatic.

So? What are you waiting for!

Comparing XML in a JUnit test

2007-05-11T11:33:00.001+02:00

Today I tried to compare 2 XML documents in a JUnit test. One was created with Altova's MapForce, the other was the result of a new XmlBeans document (BTW, both are nice products). Notice that these XML documents use a slightly different notation for the main namespace:

Document one:

<?xml version="1.0" encoding="UTF-8"?>
<Message xmlns="http://www.a.nl/a10.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="C:/longpath/a10.xsd">
<MessageHeader>....

Document two:

<?xml version="1.0" encoding="UTF-8"?>
<a:Message xmlns:a="http://www.a.nl/a10.xsd">
<a:MessageHeader>....

Here is what I tried:

1. org.w3c.dom.Document.equals. Well, that goes nowhere.

2. org.dom4j.Document.equals. Same.

3. XMLUtil's XMLAssert.assertXMLEqual. Bummer, works alright, but it says that Message and a:Message are different and they are not (they're in the same namespace!).

4. Juxy's XMLComparator.assertXMLEquals. No go, same result.

5. I took a short look at the site of XSLTunit. It says that XSLTunit is a proof of concept. Furthermore, this one is also targetted at XSLT testing. So I decided to skip it.

6. Reading a bit closer I noticed that XMLUtil 1.0 released in April 2003 (wow, that's old), has a followup: XMLUtil 1.1beta1 released in April 2007 (wow, that's new). The website says they fixed the namespace thing! Unfortunately they didn't (yet, I hope).

7. The final solution: with some String.replaces, I just removed the namespace stuff and the schema location from the documents. XMLUtil 1.0 now works nicely with very good diff messages.

Update 2007-05-24 I was quite wrong. XmlUnit does notice the differences in namespace usage (and puts a message in the exception), but it does not fail until it sees a real difference. The real difference turned out to be whitespace. By adding the code added below, the differences disappear.

XMLUnit.setControlParser("org.apache.xerces.jaxp.DocumentBuilderFactoryImpl");
XMLUnit.setTestParser("org.apache.xerces.jaxp.DocumentBuilderFactoryImpl");
XMLUnit.setSAXParserFactory("org.apache.xerces.jaxp.SAXParserFactoryImpl");
XMLUnit.setTransformerFactory("org.apache.xalan.processor.TransformerFactoryImpl");
XMLUnit.setIgnoreWhitespace(true);

Firewall gone crazy!

2007-05-08T17:30:00.001+02:00

Today I saw the solution to one of my weirdest problems ever. We have been searching for about a whole week why our JEE application would suddenly stop. The problem was that it took at least an hour to reproduce it (if at all), and even then we could find next to nothing in the logs. We only started to see the light when we noticed the difference between environments where the error was reproducable, and where is was not reproducable: the firewall.

Turns out the firewall between the application server and the database server would stop all trafic on the JDBC connection after an hour if idling, without actually killing the connection. Weblogic, and all other applications we tried, totally freak out when this happens. What on earth did the creators of that firewall think? When you kill a connection, then kill it! Do not just stop the data flow!

Well anyway, the solution? We now let Weblogic 'test' the connection every 10 minutes. Apparently, the firewall is happier when there is some traffic now and then.

Agile Web Development with Rails, 2nd edition - Book review

2007-05-05T17:34:00.001+02:00

About time for another book review: Agile Web Development with Rails, 2nd edition, written (mostly) by Dave Thomas.

The book shows you all facets of developing a Rails application. This is actually a big improvement over the first edition. That one did not even talk about migrations. There have been a lot of changes in Rails, and these are reflected in the second edition.

Just like the first edition, the second edition is well written and has the same structure: the first chapters walk you through creating a new application, the latter chapters provide more insight in the separate building blocks of Rails. Unfortunately the book never really goes to the bottom of things (though its pretty good in the Active Record area); there is simply too much to cover. So this book is probably not a good buy if you are already an experienced Rails developer and are prepared to find information about new Rails features on the internet.

So although it took me a couple of months to read this book from cover to cover, my conclusion on this book is short: this is a must-have for any new Rails developer.

Aslak Hellesøy to speak on RubyEnRails 2007!

2007-05-04T16:12:00.000+02:00

I just heard that Aslak Hellesøy will speak at the RubyEnRails 2007 conference! If want to know more about Behavior driven development (BDD), or love to hear more about RSpec, come to the conference!

RubyEnRails 2007 site is live! (Dutch)

2007-05-02T21:39:00.001+02:00

Vanaf vandaag kunnen mensen zich aanmelden voor RubyEnRails dag 2007! Op de website http://2007.rubyenrails.nl kan je je aanmelden met je OpenID.

Het sprekers gedeelte op de pagina is nog niet volledig. De lijst met sprekers zal zo spoedig mogelijk worden aangevuld. Er zullen diverse presentaties plaatsvinden over o.a. RESTful ontwikkeling, Behaviour Driven Development, Radiant en het Camping Framework.

Dus ben jij een Ruby On Rails specialist of ben je gewoon geïnteresseerd in Ruby of Ruby on Rails? Meldt je dan nu aan! Deelname is overigens geheel gratis!

Wicket article translated to French

2007-05-01T18:48:00.000+02:00

My article Backward compatible AJAX development with Wicket has just been tranlated to French by ZedroS! Nice work Joseph!

New layout

2007-05-01T14:15:00.000+02:00

I liked the little dots, but they were also a little too psychedelic. With 9 days to go for my blog's first anniversary, I decided it was time for a new layout: Tictac blue.

Spring integration test hacking

2007-04-26T09:01:00.001+02:00

Update 2007-05-01: You probably can not follow this article if you did not work with Spring tests yet. The key to this article is at the end, where I show how to call an injected object while it has been proxied by Spring.

A colleague (Levi Hoogenberg) showed me a working example of integration tests with Spring. The key is to make a JUnit test that inherits from AbstractAnnotationAwareTransactionalTests (they like long names at Spring). Simply override getConfigLocations() and a complete spring context will be loaded. Any setters in your test class will automatically be called with beans from the context (called auto-wiring). In addition, you can execute some SQL to initialize a database (for example by calling executeSqlScript() in onSetUpBeforeTransaction()). Each test gets a fresh view at the filled database and runs in a fresh new transaction.

My main spring configuration (in spring.xml) sets up a context with all the services and its Hibernate backend. The MySQL database connection is set up in a separate file (spring-db.xml). The test however, is not using MySQL but an in-memory H2 database so that it can easily be run from Continuum. This is configured in another file (spring-test.xml).

A big advantage of this setup is that it allows you to test most of the real application wiring. Disadvantage is that it uses Hibernate against H2 and not the MySQL target database. I am not too worried about this, I am not using advanced Hibernate features and I do not have to test Hibernate!

So I was happily writing tests until I found out that one integration test went much too far. The service I was trying to test (lets call it AbcService) did more then just save something in the database; it also called another service to schedule a task. In a real integration test I would need to assert that the task would have been scheduled. I realized that I was actually misusing the integration test to also do a unit test on AbcService. Instead of writing a proper unit test, I decided to leave it at that.

So how was I going to assert that the scheduler service was called? Here are my attempts:

Solution 1: Test specific config files
Since you can not override a small part of the configuration, this solution requires you to duplicate a lot of configuration files. Furthermore, you loose the ability to test the actual application configuration files. As soon as I realized this I gave up on the idea.

Solution 2: Override the configured service by changing the setter in the test class
The test class (AbcServiceImplTest) has a setter to inject the service under test like so:

public void setAbcService(AbcService abcService) {
  this.abcService = abcService;
}

Pretty standard. So my idea was to override the used scheduler service like this:

public void setAbcService(AbcService abcService) {
  this.abcService = abcService;

  // Override the scheduler service with a mock
  SchedulerService mockSchedulerService = ...
  ((AbcServiceImpl) abcSerive).setSchedulerService(
      mockSchedulerService);  // Cast fails!
}

Unfortunately, the passed in abcService is not the real thing. It has been proxied by Spring to add transaction support. The proxy that Spring uses (the standard JDK proxy) can only be casted to the implemented interfaces, and not to an actual class.

After a lot of searching and looking with the debugger, I finally found a solution. Be warned: this is a big hack. Do try this at home, but don't complain when it suddenly fails.

public void setAbcService(AbcService abcService) {
  this.abcService = abcService;

  // Override the scheduler service with a mock
  SchedulerService mockSchedulerService = ...
  InvocationHandler invocationHandler =
      Proxy.getInvocationHandler(abcService);
  try {
    invocationHandler.invoke(
      abcService,
      AbcServiceImpl.class.getMethod(
        "setSchedulerService",
        new Class[] {SchedulerService.class}),
      new Object[] {mockSchedulerService});
  } catch (Throwable e) {
    fail("setSchedulerService failed");
  }
}

Incredible, isn't it?

Ruby and Rails conference update

2007-03-23T09:16:00.001+01:00

The preparations for Ruby and Rails 2007 in Amsterdam are progressing quite well. We are proud to have Dr. Nic fly in to give a presentation. Also the rest of the conference is already well filled with many speakers. Mind you, the official announcement has not been done yet!

We are only searching for more people that want to present a quickie. A quickie is a presentation of 5 minutes sharp. You could use it for example to show some cool stuff. So if you are around in June, and you made something new, or saw a nice new feature somewhere, feel free to e-mail us at danny at rubyenrails dot nl.

Update 2007-04-23: Unfortunately Simon Willison canceled. He felt too much out of place on a Ruby and Rails conference.

Turmoil in Wicket land

2007-03-08T19:35:00.001+01:00

The impossible task faced by the Wicket team is finally coming to an end. For months and months work was done on 2 branches (1.3 and 2.0), with a third in maintenance mode (1.2). Three branches is realy too much for a group of volunteers. For the uninitiated: the 1.3 and 2.0 branches differ in one major big way: 2.0 has one extra argument to each and every component. Another large difference is that 2.0 supports generic models. For the rest, most new features and fixes were added to both 1.3 and 2.0.

So what are the options? It seems that the 'constructor change' will be dropped. This means that the 2.0 branch will be abandoned. All other 2.0 only features will be ported to 1.3. Users that use the 2.0 beta code from svn will have to migrate to 1.3.

Is this good news? I think so! If this turns out to be true, the excellent work of the Wicketeers will be much more focused again. Furthermore, all the good new features will be available for Wicket users of all branches!

Update 2007-03-19 And yes, it did turn out to be true:

I think it is time to close the vote and count our blessings. [...]
7 +1 binding votes, [...] 4 abstainees, and no non-binding votes.
This wraps it: we will drop the constructor change and migrate all features of trunk to branch 1.x in 2 releases: everything non-Java 5 goes into 1.3 and java 5 specifics into 1.4
Martijn

Vooraankondiging Ruby En Rails 2007 (Dutch)

2007-03-07T21:56:00.001+01:00

Binnenkort is het weer zover. Een nieuwe Ruby En Rails dag. Vorig jaar was het een zeer geslaagde, zonnige en gezellige dag waar veel Rails en Ruby enthousiastelingen aanwezig waren.

Ook voor dit jaar zijn we bezig om weer een dag te organiseren. Er wordt druk gezocht naar een lokatie (waarschijnlijk Amsterdam) en we zijn aan het inventariseren welke sprekers uitgenodigd worden. Waarschijnlijk zal het allemaal plaatsvinden op 31 mei, maar het is 7 juni geworden.

Heb je zelf nog goede ideeën of verzoeken m.b.t. sprekers op deze dag, surf dan naar Ruby en Rails en laat iets achter in de comments. Je kunt ook een e-mail sturen naar danny at rubyenrails.nl.

We zijn op zoek naar sprekers voor:

Presentaties van 45-50 minuten;
Lightning Quickies™ van 5-15 minuten.

Mogelijke onderwerpen:

Handige Ruby libraries (gems)
Rails plugins
Interessante real-world toepassingen van Rails
Tips & truuks, leuke vondsten, enzovoorts

Update 2007-03-20 De datum is intussen vast gezet op 7 juni. Ruby en Rails 2007 wordt gehouden in het gebouw van de Hogeschool van Amsterdam nabij het Amstel station.

Mule patch accepted

2007-03-06T22:38:00.001+01:00

Another product that accepted my patch. Till today I was proud to have patches in the following products:
- Xml2Db (specification for an aspect of the API, no code, long time ago)
- Struts (something with generating session ids, not long ago enough)
- Wicket (MixedUrlEncoding, other patches pending)
- Spring (a small documentation error)

Today this is extended with: - Mule (MailMessageAdapter potentially adds "null" string)

These were all small things, but I am still proud I was able to move open source further :)

For the future: I wrote a Camping application to manage a virtual e-mail domain for Postfix. My plan is to convert this to a Radiant extension and make it available under the MIT license.

Wicket to the recue!

2007-03-06T22:15:00.000+01:00

My previous site (for the municipality of Amsterdam) is barely finished, and I have already convinced my next client (where I work as a consultant) that Wicket is more suited for them then Spring MVC. Life is sweet! Spring MVC has a lot of Spring quality, but the programming model is just not what I want to work with. Wicket to the rescue!

More Ubuntu experiences - continued and discontinued

2007-01-30T22:24:00.001+01:00

Che promised and delivered. Almost.

The boot time was indeed fast. Che had compiled the wacom-linux kernel module so the Wacom tablet worked fine. Even dual head worked, well almost...

Problem 1: The screens were in the wrong order. The second screen is standing to the left of the main screen. No problem, just one one word changed in xorg.conf, reset and done.

Problem 2: The Wacom tablet driver still thinks that the second screen is to the right of the main screen. No idea how to solve this. Che thought this would require a code change and recompilation of the wacom-linux kernel module.

Problem 3: The external screen gets the wrong resolution. Whatever we tried, we could not get it to run on 1850x1050, the optimal Dell 2005fpw resolution. Bummer.

So again, after 2 additional hours of hacking, I am back with MS Windows. Perhaps, if time permits, I'll erase the Ubuntu partition and try familiar SuSE this week.

Note: fear not if you have one of the newer Dell Latitude D620s with nVidea graphics. Dual head works a lot better on these then on my older D620 with Intel graphics.

More Ubuntu experiences :(

2007-01-26T15:09:00.001+01:00

Having little to do for a couple of days, I decided it was time to move to a real OS on my work laptop, a Dell Latitude D620. One of my colleagues very enthusiastically recommended Ubuntu. Having read a whole lot of other enthusiastic stories about Ubuntu, it was time to give it another try. This time it was to be Ubuntu 6.10.

Stupid me.

First of all GParted just gave me an "Error" while resizing my Window's NTFS partition. That was it, just "Error", no details whatsoever. About an hour later I had tried to downloading the latest GParted iso. Luckily this did work.

Second problem was the line "BUG: soft lockup detected on CPU#0!" during startup of Ubuntu. After this the system froze up. No where to go for, my enthusiastic colleague did a google and find out that the wireless driver crashed when the hardware kill switch for the radio was on. Bummer. I flipped the switch, rebooted and Ubuntu now started. It did take a good 2 minutes longer, just to find out there is no access to a wireless network. (Why do you think the switch was off?)

Third problem was the mouse. Well, actually I do not have a mouse but a very nice Wacom Graphire3 tablet. The tablet kind of works (I could move the mouse pointer) but you can not always reach the edges of the screen. You know, like the place where the menu is located. This really annoyed the hell out of me. Luckily Firefox was already installed so I searched the internet for solutions. I found several and none of them worked. There was one guy with the same setup as me who had worked on it for 3 days! After 4 hours of messing around I found a workaround by setting the mouse acceleration a lot higher.

Today I gave it another try. The older Latitude D620s have a really bad completely sucking low contrast LCD screen that is totally unfit for anything serious. So I decided to configure my kick-ass 20" wide-screen Dell 2005FPW. I found a post on how to configure dual head but to no avail. Boy what a bummer.

Now, I am back on Windows XP and Cygwin. Don't get me wrong, I am a big Linux fan, running it for ages already, and usually avoid MS programs (even on Windows). I liked the looks of Ubuntu and I still do. But as long as it does not run with my hardware, I will not use it.

Backward compatible AJAX development with Wicket

2007-01-07T12:35:00.003+01:00

(Also published in Dutch and French, French translation by ZedroS).

Despite the general acceptance of AJAX, there is still some reticence in some sectors. The rich interaction provided by AJAX can not be used by people that, because of their handicap have to use a browser that does not support Javascript or CSS. For a sector like the government it is not acceptable to exclude these people and I believe this is an attitude that should be practiced more often.

Many developers today can or will only create web applications that work solely in Internet Explorer. Building an AJAX application that also works without Javascript is simply too much for many companies. To do this anyway, a number of techniques are known. A more general approach is given by Unobtrusive Javascript, but this article presents a different, more flexible approach that uses Wicket.

Wicket is one of the few fully component based web frameworks. In such a web framework one combines components to larger components until you have a web page. The advantage of components is that it is a lot more comfortable to develop and modify components separately. With a page oriented web framework one must usually develop the whole page simultaneously. Struts for example prescribes that you first collect all information for the whole page before a JSP page will render the (whole) page.

Composing Wicket components (a Wicket introduction)

Once creates a component in Wicket by writing an HTML fragment (the template) and by writing Java code that couples more components to the template. Creation and coupling of components happens during the construction phase. During the render phase the components can add or change the fragment or even completely replace it.

Lets look at an example. Here is an HTML template and the associated Java code:

<h1 wicket:id="title">_Template title</h1>

add(new Label("title", "The real title"));

The Label component is coupled to an h1 element. Label will put the real title in the HTML template during the render phase. The result is:

<h1>The real title</h1>

Composing components is just as simple. Suppose we want to use a title with subtitle on many places. We will create a component for that, a Panel to be precise:

<wicket:panel>

  <h1 wicket:id="title">_Template title</h1>

  <h2 wicket:id="subtitle">_Template subtitle</h2>

</wicket:panel>

class TitlePanel extends Panel {

  public TitlePanel(String id, String title, String subtitle) {

    super(id);

    add(new Label("title", title));

    add(new Label("subtitle", subtitle));

  }
}

The panel can now be used (for example in the template of another panel) with:

<span wicket:id="titlepanel"></span>

add(new TitlePanel(

  "titlepanel", "The real title", "with a subtitle"));

Linking between pages is done with the Link component:

<a href="#" wicket:id="detaillink">Book details</a>

add(new Link("detaillink") {

  void onClick() {

    setResponsePage(new DetailPage(bookId));

  }
});

The Link component will put a Wicket generated href attribute on the a element during the render phase. When the link is clicked the Wicket servlet will call the onClick method. In this example the response page is changed to a page that is constructed on the spot (pages are of course also components). After this the response page is rendered and sent to the browser. If the onClick method was left empty, the response page would not have changed and the current page is rendered again.

Dynamic pages in Wicket

Links are not only for jumping to other pages. In Wicket it is just as easy to change a part of the page by replacing a component by another one. Lets extend the example a bit:

final BookDetailPanel bookDetailPanel = ...;

add(bookDetailPanel);

add(new Link("detaillink") {

  void onClick() {

    bookDetailPanel.replaceWith(

      new BookDetailPanel(bookId));

  }
});

Clicking the link leads to a change in the current page. After this the current page is rendered again, and another book is displayed. Note that exceptionally little code is needed. In many other web frameworks all information of the complete page must be collected again.

The observant reader will have noticed that replacing a piece of a page is a trick that is nowadays mostly done with AJAX. In the example however, we have not used a single line of Javascript. Since the whole page is send to the browser again and again, lets change the example a bit more:

final Component bookDetailPanel = ...;

bookDetailPanel.setOutputMarkupId(true);

add(bookDetailPanel);

add(new AjaxFallbackLink("detaillink") {

  void onClick(AjaxRequestTarget target) {

    Component newBookDetailPanel =

      new BookDetailPanel(bookId);

    newBookDetailPanel.setOutputMarkupId(true);

    bookDetailPanel.replaceWith(newBookDetailPanel);

    if (target != null) {

      target.addComponent(newBookDetailPanel);

    }

  }

});

During the render phase the component AjaxFallbackLink generates both a href and an onclick attribute on the coupled a element. Furthermore it makes sure that a number of Wicket Javascript files are added to the HTML header. Depending on whether Javascript is supported, the browser will request either a normal URL, or it will do an AJAX call. In both cases the Wicket servlet will call the onClick method. In the first case the argument of onClick is null and everything will work exactly as in the previous example. When an AJAX call is done, Wicket will only render the components that were added to the AjaxRequestTarget and the result of that is sent to the browser. On the browser side, the Wicket Javascript searches for the element that needs to be replaced by using the id attribute. To make sure it is set, method setOutputMarkupId(true) is called.

With just a few lines of code we have created an AJAX application that even works in browsers without Javascript.

Conclusion

This article shows only a small piece of Wicket's AJAX capabilities. For example it is fairly easy to let the user know an AJAX call is in progress, to quickly execute some Javascript before an AJAX call (for example to disable a submit button), or to validate a form input when it changed.

Wicket is not only a fantastic framework to easily build modern and maintainable web applications, it is even possible to do it in such a way that older and special browsers can deal with them.

Why would you go for Wicket too?

2006-12-15T08:30:00.000+01:00

To answer that question Ryan Crumley investigated a couple of web frameworks for long term web development. He choose Wicket for a number of teams, even though there was lots of knowledge for older things like JSP. You can read his report to the wicket user list on nabble.

Wicket most active Java Web framework forum

2006-11-30T20:01:00.001+01:00

As you can see on this Nabble page, Wicket has the most active forum of all registered Java Web Frameworks.

Ruby on Rails and Django are the hot things in Ruby and Phyton land. So may we conclude Wicket is the hot thing in Java land?

Typo of the week

2006-11-30T19:24:00.000+01:00

Found on Tor Nobye's blog:

Hi Matt, I asked Roman Strobl the "Go to File" or "Open File" dialogs (equivalent to Idea's Ctrl+Shit+N). He assured me that this is planned to Netbeans 6.0.

SOA and Application/Integration databases

2006-11-22T20:24:00.002+01:00

Hmm, I was going to write about a new insight. SOA has been reported to give huge cost savings. I figured that must have been because they used an application database and prevented the complexity of an integration database.

But of course, our hero Martin Fowler has already thought of that. Its right there in the second link.

Lessons from giving a workshop Introduction Spring

2006-11-22T08:59:00.001+01:00

Yesterday I lectured a small group of students at the Hogeschool of Amsterdam to introduce them into the wonders of the Spring framework. To my surprise it was quite hard to convert the abstract ideas from Spring in to something lively and concrete.

Luckily one of the students suggested to explain the workings of dependency injection by working through the 'hello world' application from the book 'Spring in Action'. This I did using Eclipse with all keyboard shortcuts I known.
But even then, I am not sure I could really explain why Spring is useful. Simple examples never do just to Spring's power as they are more easily written directly in Java.

I am afraid that seeing why you need Spring only comes when you are building a real application.

Charles -JRuby- Nutter visits The Netherlands

2006-11-19T21:07:00.001+01:00

December 19, JRuby author Charles Nutter will talk about his work on JRuby in Rotterdam, The Netherlands. The meeting is organized by my colleagues at Finalist. The presentation is free and open for everybody. If you're interested (and of course you are), send an e-mail to seminar@finalist.com.

More information (Dutch)

Note: There is a rumor that Thomas Enebo, another JRuby author will also be there.

My first (bad) Ubuntu experience, long live SuSE

2006-11-19T15:39:00.001+01:00

SuSE 9.1 is no longer supported so it was time to update my home firewall. SuSE has been my choice since its 6.0 version many year ago. But now that there are so many hallelujah stories on Ubuntu, I thought lets give it a try. The installation went well but as soon as it was prime time the computer simply rebooted right after grub had loaded the kernel and the initial ramdisk. No error message or anything else to shed some light on the matter, just a friendly beep and the process started all over. I tried the 6.06 and the 6.10 versions, with the same results. The weird thing is, the hardware is not even that old (its an AMD-K6 at 400MHz) and has more than enough memory (almost 400Mb) and harddisk space (6 and 30 Gb).

After that I went through hell to get the firewall running again. My steps included:
- installing an ancient SuSE 7.0 from CD
- figuring out how to get that SuSE 10.0 installation DVD copy I accidentally still had on my MP3 player to the firewall's second harddisk
- trying to update SuSE 7.0 to 10.0 (it refused)
- figuring out how to make a SuSE 10 installation boot CD (see previous post)
- installing SuSE 10.0 from harddisk

Weirdly enough the hardest step was to copy the SuSE 10.0 installation files to the firewall's harddisk. First I tried WarFTP to serve it from my Windows machine. However, I kept getting access denied errors (configuring WarFTP is a science), and the ftp program is not really suitable for recursive retrieval. I finally succeeded by using the smbclient program.

Perhaps I'll try Ubuntu again when SuSE 10.0 is no longer supported. Unfortunately this is already next summer, sigh.

Update 2006-11-22: In my hurry to write this article I made a small mistake: my previous SuSE version was 9.2 and its support has been discontinued as of today. SuSE 10.0 is good to go until October 2007. See also SuSE lifetimes.

Creating a SuSE installation boot CD

2006-11-19T15:14:00.001+01:00

Suppose you have a copy of the SuSE 10.0 DVD, but you don't actually have a DVD player on your target computer. Here is a way to work around that:

Requisites:
- target computer must have a CD player and a BIOS that supports booting from CD
- target computer must have some means to get to the installation files, preferably these are put on a not-in-the-way partition on the target computer (for example on the second harddisk)
- a (unix) computer with package mkisofs installed
- some way to burn an iso image to CD

Steps:
- cd to installfolder/boot, where installfolder is the root of the installation source
- execute

mkisofs -R -b loader/isolinux.bin -no-emul-boot -boot-load-size 4 -boot-info-table -o suse10-installation.iso loader

- burn the result to a CD
- make sure the target computer has a means to get to the installation source
- let the target computer boot from the CD

When you are in the installation program you can do a ' back' to get in the installation program's menu. From there you can configure an alternate installation source.

Note that if you choose to install from a harddisk partition you should NOT format it, and you should NOT try to mount that partition during the installation. If you want you can mount the partition after the installation.

JRuby on Rails status

2006-10-29T14:52:00.001+01:00

JRuby 0.9.1 is out for a week. This is the first version in which JRuby really starts to perform and behave properly. Though Ruby on Rails has been reported to work on JRuby for a longer time, this was not repeated by many people; too many loopholes to jump through. However, with the advent of version 0.9.1 there are suddenly many people on the mailing list that are trying and achieving exactly this.

Another promising release was done yesterday in Japan: rails-asyncweb 0.1, a very fast web server for JRuby on Rails. TAKAI, Naoto's asyncweb uses AsyncWeb, a web server that prevents the synchronization pitfall that the Java Servlet API brings. AsyncWeb scales a lot better under load then traditional Java web servers. And the performance rocks! It is about 10 times faster then Webbrick under loads to 10 concurrent users. It will be very interesting to see what happens under much much larger loads. I am also curious how this will compare to Mongrel, Zed Shaw's successor for Webbrick in the C-Ruby world. Another interesting question is whether this will end Mongrel's Java port.

Another web framework that works under JRuby is Camping. Camping is smaller then 4K, but quite powerful. It is a creation of the well known Ruby virtuoso why the lucky stiff. If you want to get a feel for what Ruby can do but don't want to learn Ruby on Rails (which I found quite a large task) I recommend you try out Camping. Ola Bini has a nice tutorial on getting you started. (Note: you should not use Camping version 1.5 because JRuby does not support continuations.) Another very good Camping tutorial (Dutch) was written by Remco van 't Veer.

Update 2006-10-30: added Remco's tutorial.

Wicket autocompletion improvements

2006-10-20T21:07:00.001+02:00

Here is how I extended Wicket's (version 1.2.2) autocompletion with 2 new features:
- immediatate open of the drop-down as soon as the text field gets focus
- define the width of the drop-down independent of the text field

As the JavaScript is well hidden within Wicket and is not easy to extend, I tried another approach that worked well. I created the package wicket.extensions.ajax.markup.html.autocomplete and placed my own edit of wicket-autocomplete.js. Because the package is compiled into my WAR I am sure it is the first on the classpath.

First of all, in the initialize function I added the following fragment:

 if (obj.className.indexOf('immediateopen') != -1) {
     obj.onfocus=function(event){
         updateChoices();
         return null;
     }
 }

Secondly I changed function showAutoComplete to the following:

 function showAutoComplete(){
     var position=getPosition(wicketGet(elementId));
     var menu=getAutocompleteMenu();
     var input=wicketGet(elementId);
     var rem = input.className.match(/\bmenuwidth(\d+)\b/);
     var menuwidth = (rem == null) ? input.offsetWidth : rem[1];
     menu.show();
     menu.style.left=position[0]+'px'
     menu.style.top=(input.offsetHeight+position[1])+'px';
     menu.style.width=menuwidth+'px';
     visible=1;
     hideShowCovered();
 }

That's it. Its fully backward compatible, except that if you use the classes immediateopen or menuwidthXXX you'll get the new behavior. If you are lazy, you can download the complete file.

Example:

  <input class="inputcode immediateopen menuwidth300" type="text" />

If programming was as risky as parachute jumping...

2006-10-10T19:47:00.001+02:00

Last Friday my company organized a parachute jump. Unfortunately the clouds were, typically for the season, too low and the wind was too hard. But I learned some interesting things anyway as the morning was spend on a crash course testing by Kees Blokland from PolTeq.

One of the major things for testing is risk analysis; the amount of time spend on testing an aspect of the system is proportional to the risk of that aspect (where risk is chance of failure times impact). For fun we made a risk analysis of parachute tandem jumping. To make it more real the instructor and owner of the jumping company also joined the discussion. He told us what he does to keep jumping safe and the risk low. He named two aspects 1) attitude, and 2) an open culture. An open culture is important because incidents must always be reported so that they can be analyzed and prevented in the future.

The attitude aspect coincided well with a discussion we had earlier about the tendency to not test well. Sometimes the client thinks it is not necessary, sometimes developers don't know how to test properly, and sometimes they don't want to.
I asked what the instructor did to maintain a good attitude so that we might apply his lessons in the world of software. Here was his answer:
1. parachute jumping in itself already attracts the right type of people,
2. everybody understands that safety and risk control is paramount,
3. maintain maximum openness about incidents,
4. make an example, even a bruised toe is reported to the authorities, and
5. take disciplinary measures only when there is an attitude problem. These measures are usually drastic, for example a withdrawal of the jumping permit.

So what can we learn from this? Here are my ideas derived from each answer:
1. Attract people with the right mind set. Look out for people that are both result and quality driven.
2. Train you project managers, and require them to organize risk assessments. In software not everything is deadly dangerous and not everythings needs to be tested.
3. Use bug metrics. I am not so sure about this one. But you should at least use automated testing.
4. Even a technical team leader can make bugs. Be sure that everybody knows that.
5. Don't blame people when they do something wrong. But give the pink slip when they refuse to improve.

Even when you will apply these lessons, you will have to realize that only when programming will be as risky as parachute jumping, programs will become bug free.

We made the jump 2 day later. It was great and there were no bugs.

What makes a programmer?

2006-10-09T15:41:00.001+02:00

The answer to this rather philosophical question came up yesterday on the way home from my company's parachute jump event (more on this in a later post). My superior explained how he used to program in the past: "Try something, and if it works add more. Things that are used more then once are moved into methods." This approach will work for small projects, but will create a big mess over time. It will definitely create a mess immediately for larger projects. We are lucky that he is a good manager, but also lucky that he does not program for our customers.

So how does a good programmer work? In my opinion the sole work of a programmer is to map abstractions onto other abstractions, and to add new abstractions when this is not possible.

Lets give a simple example. XML documents are a well known abstraction for a hierarchical structure of labeled values. Files are a well known abstraction for a named sequence of bytes. To persist an XML document one writes code that maps the first abstraction to the second. How to do this is well described in the XML standard, but is definitely not the only way. I have written more performant mappings for specific cases.
A more complex example: to understand a voice command, one of the mappings might be from a sound sample to vowels and consonants.
Other types of abstractions are written in functional designs, test documents, etc.

So what makes an excellent programmer? An excellent programmer:
- Works with well known abstractions. There is not much to know about XML documents and Files. This makes them easy to understand and versatile. RDMSs provide an abstraction for storing tabular data. They are more complex, but well know.

- Will create small and consistent new abstractions.
An excellent programmer will create small levels of abstractions in his system with consistent terminology. For example, while processing sound samples it may be wise to create an intermediate level of abstraction that facilitates working with different sample rates and sample sizes. Consistency can be reached naming things carefully. Simply using the same names for the same things is a good step.

- Can work with many levels of abstraction at the same time.
At times it will be necessary to understand how the different abstraction mappings interact. For example, while persisting a phone book in an XML file, you need to know what each mapping does with character encoding.

So how do you do all this? Unfortunately I have no simple receipt as my manager has. I am afraid, excellent programming is still a craft.

Top3 reasons for using NetBeans

2006-10-04T09:25:00.001+02:00

Here is my top 3 reasons for using NetBeans:

3. You are masochistic, and like to change IDEs once every while
2. JPA support is apparently quite good
1. You are going to work for Sun

Wicket getting even more serious

2006-10-03T14:08:00.001+02:00

I just saw on the Wicket e-mail list that there is now a company that provides commercial Wicket support. Their name is, guess what: Wicket Support.

Their initial coverage is basically Europe and United States. But they are interested in extending it if the opportunity arrives.

Wicket getting serious, Pro Wicket - Book Review

2006-10-02T21:35:00.001+02:00

One of the interesting new web frameworks that saw the light recently is Wicket. Wicket is quite radical in that it relies solely on Java programming and does away with configuration and JSP/PHP/Rhtml type html generation. Although not unique in this approach (e.g. Barracuda, Tapestry) many web-developers will have a hard time getting up to full speed quickly. Luckily this has changed with the publication of the first Wicket book: Pro Wicket by Karthik Gurumurhty.

The book 'Pro Wicket' guides you through the development of a simple and sometimes not so simple web application. Page by page the application is extended and shows more and more Wicket features. Sometimes a 'mistake' is made. That mistake is then later 'corrected'. Unfortunately this makes the book less suitable as a reference book as you always have to flip through the chapter to see if you have the complete example.

The contents of the book varies nicely. The basics and way of thinking in Wicket are explained with lots of examples. Also the internationalization features are thoroughly explained (although I find it disturbing that the example mixes up languages and countries). There are well set up explanations and examples on how to integrate popular technologies like Spring and Hibernate. The AJAX examples cover the basics (which are quite good in Wicket) but a lot is left to be discovered. Testing is even less well covered, but at least here some good pointers are given to the manuals. A nice touch is to take a brief preview of Wicket 2.0.

The book is completely written in speaking language. The following sentence is typical: "As you might have guessed by now, this method returns false by default.". For some people, like me, this is a turn down, but most are probably okay. What is worse is that on every page you'll find pronouns (like 'it' and 'that') for which it is quite unclear what is referenced.

The Achilles' heel of the book is in the end of chapter 2 where there is a totally incomprehensible explanation on how to use repeating page elements in Wicket. I find it a shame that such an important subject is tucked away at the end of a chapter on a different subject (validation), and that the reader is suddenly confronted with a page full of abstract terms and such a small example.

I had a hard time reviewing this book. At many points I thought that the book was not finished enough to be published. Of course this was okay as I was reading the beta, but when the final book came out I noticed that only the code examples had been thoroughly updated. I am afraid that either the publisher pushed this one too hard, or the author just was out of time.

Conclusions
Despite some of the remarks above I am going to recommend this book anyway. First of all because it is the only Wicket book on the market, secondly because it does teach you Wicket. And that is a Good Thing.

Thanks to Apress for providing the book for this review.

Update 2006-10-05: Added link to Karthik Gurumurhty's blog, removed spelling mistake, deleted all comments related to the spelling mistake.

Back to Radiant

2006-09-29T22:07:00.001+02:00

In one of my first posts I talked about Radiant. Having tons of other stuff to do, it dropped below my horizon. Because I am still on the mailing list, I saw a presentation passing by from Nathan Wright. The presentation is a beauty for first time Radiant users. I can highly recommend it if you like Ruby and are searching for something to build a simple site.

Because I can't find a download, I've put it on my home server for now. So here it is:

Radius, Content Management Simplified (5.6Mb)
Denver Derailed meeting September 27, 2006
Covers Radiant version 0.5.2 By Nathan Wright

Update 2006-10-02: Thanks to Luis Lavena for providing a more solid server.

Update 2006-10-04: The link above is now pointing to Nathan's own blog.

Update 2007-03-06: All previous links are now dead. I placed the copy on my home server back.

Wicket for BSCs

2006-09-28T13:43:00.001+02:00

Since I am finally getting to do a Wicket project (more then a year after I first learned about it), I subscribed again to the Wicket user mailing list. Initiated by an e-mail from Erik Brakkee, I started a thread to find out how to convince Big Slow Companies (BSC) to use Wicket and finally leave Struts in peaceful history.

There was only one criterion for the items in the list: they must be one-liners that are irrefutable so that there is no place for any FUD.

Together we came to the following points:

- Wicket enables scalable development. It is easy to split the work over many developers, it is even easy to split the work for HTML people and Java people.

- Wicket provides a natural programming paradigm familiar to Java and other OO developers. There is no need to learn new languages, Java and HTML are sufficient.

- The Wicket user e-mail list provides excellent support.

- Wicket has excellent feedback messages in case something goes wrong. Furthermore, Wicket is robust and shows no weird or unexpected behavior.

- Wicket provides excellent clustering support. When required, it is possible to optimize session usage.

Although there was some strong support for including reusability and maintainability I did not do so. I have yet to find BSCs that deeply care about these issues.

Thanks go to Eelco Hillenius for his infinite ability to promote Wicket. Thanks to Che Schneider for coming up with the TLA BSC.

Java voor Excel Quiz

2006-09-22T16:05:00.001+02:00

Here are some projects:
- JExcel
- Java Excel API (aka JExcelApi)
- JXL
- POI

Which one does not belong in the list?

I'll post the answer in a few days with some more information on each project.

Update 2006-09-26

JExcel is a commercial library for for integrating Microsoft Excel into Swing applications. It will actually show an Excel sheet in your Java app (you do need Excel installed). From your Java code you can change the worksheets. Sounds neat, but I wonder who would want to use this. One thing I found is that the API seems to be a evolution of the following project: Java Excel API.

Java Excel API is an open source java library with which you can read, write, and modify Excel spreadsheets. I am currently using this one as it had some better blogs then POI. Although it does it works, I am not completely happy with it. The API is not consistent (for example: you can make something bold, but there is no obvious way to check whether a format is made bold) and the package structure and class names are kind of weird.

JXL is an umbrella for a couple of small open source java library with which you can build distributed Java programs with distributed class loading. I did not look much further into this. It seems that it tries to do JavaSpaces like things.

POI is again an open source project with which you can read, write and modify OLE2 documents including Excel and Word files. I have never used this, so I can not compare it to JExcelApi.

So the answer is: JXL.

More portlet woes

2006-09-07T21:09:00.001+02:00

I've got my portlet running for some time (see previous post). I even have a weird hack to enable file downloads. The Spring portlet MVC framework helped me a lot to structure the portlet really nice.
But now a colleague has started to write portlets as well. And guess what, he is as frustrated as I was. To make things worse, even with my previous Pluto experiences, there is nothing I can do for him! It is all so complicated with nondescript exceptions and configurations all over the place.

These 2 reasons (bad exceptions and configuration) are exactly the thing why Spring is such a joy to work with compared to using an old style EJB setup. And, you can even do better. The web framework Wicket gives you no configuration at all! And it still gets better: no more JSPs! Wicket to conquer the world!

Walking the Java Road

2006-08-31T20:51:00.001+02:00

I am walking the Java road. This week I told a junior programmer that learning Java is not something you start and then complete. Its a road; you learn as long as you are doing it. I am on the Java Road for 5 years now and I am still learning every week, almost every day even.

So what about this Ruby Road over there? It is a lot smaller road, but it appears that only fast vehicles are allowed. You really need to steer well, no mediocre developers here, they've crashed way behind you. And when you finally get it, boy, you get to go fast.

The major problem with walking on 2 roads is that you only have so much time in your life. I need time for the ones I love and then I need some sleep now and then. I would not mind switching, but then there should be enough work. Unfortunately, all the interesting stuff is still in Java. So for now its Java for me. But I'll keep looking...

Profict Java Summer Camp - AJAX

2006-08-24T23:53:00.001+02:00

I am just back from Profict's AJAX day, a very nice way of promoting one's company among developers. There were 2 speakers: Greg Murray, the AJAX architect from Sun and Bram Smeets, the DWR expert from Interface 21. In between there was a small product presentation from BackBase.

4 hours for Greg Murray
Four hours were filled by a jetlaged Greg. It seems that because he got so much time he could not chose what to tell from the many presentations he brought with him. It was a shame everything was so jumbled up because there were some really nice things in there. Without going into too much detail, here are some of the things he presented.

jMaki
Greg is the main author for jMaki. jMaki is a small toolkit that allows you to wrap other Javascript toolkits (among others Dojo and Zimbra). jMaki comes standard with some nice widgets. Besides the usual datepickers and autocomplete stuff, there were some very nifty widgets that allow you write Mashups with Flickr, Yahoo Geocoder and Google Maps. A Mashup is a mashed combination of several web applications. One of his examples was a reworked Petstore where each pet was indicated on a map. jMaki is available for JSP and JSF environments. But given the raw components (simply a html, css and js file) you can probably include this stuff in any other framework easily.

jMaki also has some other basic stuff. For example an eventing mechanism (modeled after Dojo) and XmlHttpProxy. XmlHttpProxy is an XmlHttpRequest like feature that will allow you to call services in any domain by proxying the call on a server. This circumvents the security mechanism of Javascript in which you can only do HTTP requests to the domain from which the page was served.

AJAX, Back button and Anchor links
The trick of the day was the use of anchor links (the part after the #). When the user triggers an AJAX action (for example loading a different product), the URL of the currently loaded document is changed, but only in the anchor part. Since only the anchor changes, the browser will not reload the current page, but will add an entry to the page history. This means that suddenly the AJAX enabled page has a bookmarkable URL. In addition, much better back button support becomes possible.

Bram Smeets on DWR
The 1 and half hour presentation from Bram was a very nicely done introduction to DWR. In contrast to many other Javascript toolkits, DWR only provides communication between JavaScript and Java. However, as this presentation showed, DWR does so very well. In short, the idea behind DWR is that you can call a Java method directly from Javascript, and then can use the returned objects as if they are plain Javascript objects. If so desired, some utility Javascript functions can easily convert these objects to HTML and put them into the current page.

First an example was presented for DWR with plain Java and XML configuration. The few lines of Java and Javascript code on the slides showed that DWR is really a convenient thing to use. One of the killer features is a debug view of all exposed classes and methods. After a pitch style introduction of Spring, Bram then showed how to configure DWR from Spring. Though the integration is indeed nice, it gives you no advantages when you are not using the Spring bean container for some other reason (which of course there are plenty).

Reverse AJAX
The audience really heated up with the topic of Reverse AJAX (a.k.a. Comet, Long poll and for some even as Pushlets). This experimental DWR feature allows you to execute Javascript from Java. A very very neat example showed a method call on a Java-Scriptaculous 'impersonator' on the server side, with as result that the Scriptaculous effect is applied in the browser. Unfortunately Reverse AJAX is still at an experimental stage. As Bram told me later, this is mainly because it has not been tested properly in many circumstances. In addition, you must be very aware of the load this brings to your server.

Barbecue
The day ended with a nice barbecue in the beautiful garden of Profict's castle.

Swing Date Pickers

2006-08-07T09:28:00.001+02:00

Back in April I looked around for a good open source Swing date picker for a big Swing project. I found 6 and I actually tried them all. But in the end we simply stayed with JSpinner components.

These were the must-have requirements:
1. support for typing dates directly (this is, without opening a pop-up)
2. ability to limit the selectable dates with a lower and/or upper boundary
3. good event mechanism

And some nice-to have requirements:
4. visual mouse selection
5. support for multiple date formats
6. look good
7. further limitation of selectable days, for example no weekend days

These were the date pickers I tried:
- JCommon
- MDateSelector
- SwingX's date picker
- UICollection's date picker
- JCalendar
- And there was a sixth, but I can not find it back anymore.

None of these met the must-haves. In particular requirement #1, keyboard support, is only present in the the SwingX component. In contrast good old JSpinner supports #1-#3 splendidly, but fails miserably on #4-#7.

However, my search did seem to have some result.

It started with comment on an introduction on the SwingX JXDatePicker by R.J. Lorimer on JavaLobby. This guy writes good stuff. Then I thought, why not ask on the SwingX forum directly. So I wrote this message in search of more features. The message was followed by a lively discussion in search of requirements.

It seems that the SwingX guys picked it up and are now working on the final killer Swing date picker component!

The Pragmatic Programmer - Book review

2006-08-06T12:54:00.001+02:00

A colleague from the company that currently hires me, proudly told that they have a copy of the The Pragmatic Programmer for every 3 employees. Having just finished the Spring MVC book, I thought lets see what this is about. So I read the book and here I will convince you to read it as well.

After only a few chapters 2 strong thoughts came up and remained there for most of the rest:
1. What a dull book, I am already doing this stuff. But more importantly:
2. This is Good Stuff, I wish more people would read this book.

Though the book is old (2000) it is still very relevant. It contains a large set of well described programming practices and attitudes. You can read the entire book in any order. From experience I can say that what this book says, will really help you build better code more quickly.

One point I loved, is when the authors write off the construction metaphor (which is so previous century) in favor of the gardening metaphor. 'Gardening' is much better for explaining to business people why you sometimes need to spend time with no visible improvements.

I wondered why I had never seen iContract before, which is I think a hidden gem of the book. Apparently it is only recently resurrected.

As a totally unrelated bonus, I finally understand the use of Linda like technologies (Rinda/JavaSpaces).

Some minor points:
- I only noticed one trend away from this book: things like Active Record makes the section "Code generators" a bit redundant. Of course it can still be useful. Just watch out for software in which you don't need it.
- Exercise 23 states that it is a good idea to set java references to null after you are finished with the object. I disagree. You can consider doing this when you are writing code for a particular (old) JVM version that you happen to know intimately. And even then I would say that your methods are too long.

Anyways, the book does justice to its title; all tips are practical and executable. The more people will use ideas from this book, the less bad software I will have to work with. So here is my request to all colleague programmers: please go and read this book!

Author blogs: Andrew Hunt and David Thomas