Breaking a Java HashSet

Can it be?

Set set1 = new HashSet(5); Set set2 = new HashSet(5); // add of bunch of strings to both sets assert set1.equals(set2) == false;

Yes it can!

We actually had this problem in an integration test. The cause was that the strings to one set were added concurrently. Interestingly, the sets seemed to be the same, when printed they contained the same strings, just in a different order (which is also interesting). However, the internals were apparently so damaged that an equals invocation return false.

We replaced the HashSet with a ConcurrentSkipSet, and all was fine again.

Conclusion: Okay, so this is a boring conclusion, but just be careful with using normal collections in concurrent situations.

DNS for Version 99 is off-line

To alleviate some pain with using the commons-logging framework, I created version 99. It was hosted at the hostname no-commons-logging.zapto.org courtesy of no-ip.com. Unfortunately, due to lack of traffic I had to affirm usage of the the hostname once per month. Though this was slightly annoying, a complete pain is that I missed the last deadline, and that recreation is out as dashes in hostnames are no longer allowed.

As of 2011-08-15 16:11:00 GMT no-ip did credit to its name; no ip for no-commons-logging.zapto.org.

As 1) the current situation is disruptive for any version-99 user anyway, 2) there is a better workaround using the provided scope, I decided to not put back version 99 online under another hostname.

If you really need to use version 99 for some more time, just add an entry to your /etc/hosts file:

83.163.41.27 no-commons-logging.zapto.org

As promised, I'll keep version 99 for 5 years, that is until October 2012. Please check back here for IP address changes.

Update 2011-12-19: I just found an alternative with almost the same name: version99.qos.ch. It is a static maven repository with a limited number of version 99 jars.

Lock-less singleton pattern

Although the singleton pattern is now know as an anti-pattern, I think it still is a valid choice when you only need one instance in a particular context. Anyway, in Java there are several ways to implement a singleton.

For example this one uses a synchronized block:

private Singleton singleton = null; protected Singleton createSingleton() { synchronized (this) { // locking on 'this' for simplicity if (singleton == null) { singleton = new Singleton(); } return singleton; } }

To prevent locking all the time you can use the double-check idiom on a modern JVM. But I just thought about another implementation that switches locking for preventing code re-ordering, making it a lock-less implementation:

private AtomicReference singletonRef = new AtomicReference(null); protected Singleton createSingleton() { Singleton singleton = singletonRef.get(); if (singleton == null) { singleton = new Singleton(); if (!singletonRef.weakCompareAndSet(null, singleton)) { singleton = singletonRef.get(); } } return singleton; }

There is only one catch to this implementation: although createSingleton() will always return the same instance, it must be allowed to call the constructor of Singleton twice without side-effects.

Update 2011-06-18: changed compareAndSet to weakCompareAndSet as suggested by Adriano.

Apache Wicket Cookbook — book review

Some time ago I reviewed the drafts of the new book from Wicket rockstar programmer Igor Vaynberg: Apache Wicket Cookbook. If you are serious about using Wicket, this book is for you. It is fast, to the point, has very clear code samples and teaches you all the relevant (both clean and dirty) stuff you need and which Wicket in Action could not cover.

Conclusion: this is the book to read after 'Wicket in Action'.

Virtualizing a PC

All hardware is eventually decommissioned. It may be broken, stolen, or just too old and slow. However, the software on it might still be needed. Not all software is easily transferable to a new PC with the latest OS on it. This quick guide helps you convert the old machine to a virtual machine, so that you can run it in VirtualBox. The process is called physical to virtual (P2V).

This article assumes you are at home in a Linux shell.

1. Boot the old hardware from a Ubuntu live CD or USB stick (or any other live cd, for example the gparted cd).
2. Prepare copying each file system with the following steps:
   1. Mount the filesystem
   2. Remove the contents of /tmp and trashcans in the home directories.
   3. Clear free space with the following commands:
      sudo -s dd if=/dev/zero of=/usr/bigfile; rm -f /usr/bigfile
3. Now copy each physical disk to the target host computer (you could also do this per partition). With nc (netcat) and dd this is easy. Netcat essentially opens a pipe between two computers. We then use dd to stream the entire disk to and from the pipe. One of the two netcat's are in listening mode, the other connects to that one. It doesn't matter which one does the listening, except I found that netcat in MacOSX does not properly support listening. To speed up the transfer, gzip is used. Because we cleared empty sections of the disk, gzip is a quite efficient addition. (Update 2013-04-15: Now I would recommend a compression program that compress/decompress faster at the cost of compression efficiency. For example lzop.)
   Here are the two examples of copying the disk /dev/sda. The first puts netcat in listen mode on the target:
   on target> nc -l 19001 | gzip -d -c | dd of=hd-copy.raw on source> dd if=/dev/sda | gzip -c --fast | nc [target] 19001
   The second example puts netcat in listen mode on the source system (execute in the given order):
   on source> dd if=/dev/sda | gzip -c --fast | nc -l 19001 on target> nc [source] 19001 | gzip -d -c | dd of=hd-copy.raw

   Make sure you have a decent network connection, 1 Gbit/s is fine.

4. The next step is to convert the raw image to a VirtualBox image. Do this with the following command (tested with VirtualBox 4.0.4):
   VBoxManage convertfromraw hd-copy.raw hd-copy.vdi --variant Standard
5. If there really was a lot of empty space on the disk, you can compact it with the following command:
   VBoxManage modifyhd hd-copy.vdi --compact

References

Chapter 8 of the VirtualBox manual
Another P2V technique which also works in VMWare.

On making a custom Ubuntu Bootable USB stick

In my upcoming Wicket course the students will use a rental laptop. How do we guarantee that they will be up and running in no time? Colleague Jason had the solution: burn a Ubuntu Live CD to a USB stick!

Though simple this may sound, in the end it took us almost 3 days to put it together. This article gives an overview of the steps I took to create the USB sticks and how I tested them with VirtualBox.

Preparations

For starters you need a fast PC with about 15 GB free diskspace. I used Ubuntu 10.10, but other version probably work similarly. You'll need sudo rights as well. If your PC is somewhat older building the image might take quite some time.

Secondly, install the Ubuntu Customization Kit (package name uck). If you are on Ubuntu 10.10, make sure package gfxboot-dev is installed as well.

Finally you will need a clean Ubuntu iso image. Any Ubuntu, Kubuntu or Xubuntu image will do. This will be your starting point so choose carefully. I took ubuntu-10.10-desktop-i386.iso as I wanted to be sure it ran on older (32 bit) PCs as well.

Be prepared to repeat the whole customization process: log carefully what you do and keep a copy of all content you change or add.

Customizing the CD, first build

Uck puts everything in $HOME/tmp so I created that directory first. Then I simply executed uck-gui. The questions that are asked are quite straight-forward to answer. Don't select too much; to make the resulting image fit on a 1Gb USB stick you will need all the space you have. The last question makes you choose between running a packet manager, running a command shell, or continuing the build. Choose the command shell as it makes it a lot easier to keep track of your changes. Tip: use the history command to see what you did.

Removing packages

In the command shell you can remove packages and make other customizations. The Ubuntu wiki gives much information. The shell is chrooted to the new filesystem, so everything you do in that shell is restricted to the image you are creating. To be extra clear: if you remove a package in the UCK shell, you remove it from the live cd image and not from the rest of your system.

I purged lots of packages I didn't expect my students would need. Comments with more large packages that are probably not needed in a course are appreciated.

apt-get purge ubuntu-docs openoffice.org-core openoffice.org-java-common openoffice.org-common openoffice.org-writer openoffice.org-help-en-us openoffice.org-calc evolution evolution-common libevolution evolution-data-server evolution-webcal gbrainy gnome-games-common gwibber-service telepathy-gabble telepathy-gabble libtelepathy-glib0 python-telepathy empathy-common hplip-data hpijs hplip-cups hplip

Installing packages

Then I installed Java and cleaned up:

sudo add-apt-repository "deb http://archive.canonical.com/ lucid partner" sudo apt-get update sudo apt-get install sun-java6-jdk apt-get clean

I also removed some example content:

rm -rf /usr/share/backgrounds/* rm -rf /usr/share/example-content/Ubuntu_Free_Culture_Showcase

and placed my own background. See the ubuntu wiki link above for more details.

In /etc/skel I removed the link to the example contents and I added the following to .profile:

# Java export JAVA_HOME=/usr/lib/jvm/java-6-sun # Maven export M2_HOME=/opt/maven3 export PATH=${PATH}:${M2_HOME}/bin export MAVEN_OPTS="-Xmx512m -XX:MaxPermSize=512m"

The next step was to unzip recent Maven, Eclipse and IntelliJ distributions in /opt. The easiest way I found to do this is open a new shell, switch to root with sudo -s and start a nautilus. This gives you convenient access to $HOME/tmp/remaster-root. Make sure all placed files have root:root as owner.

Close the shell with exit (if another shell opens, close this one as well), and the Uck menu will appear again. Select Continue build from the Uck menu. And within a few minutes you have your first $HOME/tmp/remaster-new-files/livecd.iso!

Using the live cd

To test the live cd I installed VirtualBox (I installed 4.0.4 from the Oracle repository). I created a new machine with 8Gb of virtual harddisk space and mounted the new live cd iso file as virtual cd. That's it, just start the machine and play with it.

Once the virtual machine was fully started I selected 'Try Ubuntu'. I used Maven to make sure all required dependencies were sucked in to $HOME/.m2. Hint: I used the following commands in the largest example project:

mvn install mvn eclipse:eclipse -DdownloadSources=true mvn jettty:run

To ease starting Eclipse and IntelliJ I created a launcher on the desktop. Just right-click the desktop and select 'create launcher'. This will create two .desktop files on the desktop.

Tip: check the settings of the background image.

Now open a new Nautilus window from the 'Places' menu and open a sftp connection to the guest system (do Ctrl-L and enter something like sftp://host/home/user). This allows you to copy all changed and new content to the guest system for inclusion in the next build. I used this to copy the seeded maven repository and the two new launchers.

Satisfied I had a record of all my changes, I shutdown the virtual host, and applied the changes during a second run of uck-gui.

Making a bootable USB stick

After a few builds, I was confident the live cd was ready. The next step is to create a bootable USB stick. This is made easy with the 'Startup Disk Creator' you will find in the 'System/Administration' menu. If you want the USB stick to be useful without installing Ubuntu to the guest computer, make sure there is plenty of space for writing changes. You will probably need a 2GB stick, unless you remove many more packages a 1Gb stick does not leave enough space for any serious development.

Test the USB stick by booting your computer from it.

Testing the installation from the USB stick turned out to be much hard then expected. I didn't have a spare laptop, and VirtualBox's BIOS does not support booting from a USB device. The easiest way I found to circumvent this was to create a vdi image from the stick in the following way:

sudo dd if=/dev/sdb of=raw-usb-image.bin sudo chown user:user raw-usb-image.bin VBoxManage convertfromraw raw-usb-image.bin usb-image.vdi

You can now mount the vdi file to your virtual machine (make sure its the first drive) and boot from it.

Cloning the USB

Once you have created the USB stick, you can clone it easily with the dd command.

Be careful! Selecting the wrong device may kill your harddisk! You have been warned.

# Copy the USB to an image on disk sudo dd if=/dev/sdb of=raw-usb-image.bin # Write it out the another USB image sudo dd if=raw-usb-image.bin of=/dev/sdc

A USB 2.0 hub with many ports helps because the throughput of the USB channel is much higher then the write speed of the sticks. For me, writing 8 USB sticks took the same time as a single stick.

USB cloning in action

Update 2011-03-27 One problem with cloning disks like this is that they all get the same UUID. In some situations this may give problems. I have yet to find a solution for this. I did not find a tools for just changing the UUID of a FAT file system.

Conclusions

With the help of the Ubuntu Customization Kit, creating a customized Ubuntu startup USB stick becomes a surprisingly straight forward -though lengthy- process. Make sure you can repeat your steps as you probably need to. I found VirtualBox to be an excellent tool to test both the Live CD and the bootable USB sticks.