Virtualizing a PC

All hardware is eventually decommissioned. It may be broken, stolen, or just too old and slow. However, the software on it might still be needed. Not all software is easily transferable to a new PC with the latest OS on it. This quick guide helps you convert the old machine to a virtual machine, so that you can run it in VirtualBox. The process is called physical to virtual (P2V).

This article assumes you are at home in a Linux shell.

1. Boot the old hardware from a Ubuntu live CD or USB stick (or any other live cd, for example the gparted cd).
2. Prepare copying each file system with the following steps:
   1. Mount the filesystem
   2. Remove the contents of /tmp and trashcans in the home directories.
   3. Clear free space with the following commands:
      sudo -s dd if=/dev/zero of=/usr/bigfile; rm -f /usr/bigfile
3. Now copy each physical disk to the target host computer (you could also do this per partition). With nc (netcat) and dd this is easy. Netcat essentially opens a pipe between two computers. We then use dd to stream the entire disk to and from the pipe. One of the two netcat's are in listening mode, the other connects to that one. It doesn't matter which one does the listening, except I found that netcat in MacOSX does not properly support listening. To speed up the transfer, gzip is used. Because we cleared empty sections of the disk, gzip is a quite efficient addition. (Update 2013-04-15: Now I would recommend a compression program that compress/decompress faster at the cost of compression efficiency. For example lzop.)
   Here are the two examples of copying the disk /dev/sda. The first puts netcat in listen mode on the target:
   on target> nc -l 19001 | gzip -d -c | dd of=hd-copy.raw on source> dd if=/dev/sda | gzip -c --fast | nc [target] 19001
   The second example puts netcat in listen mode on the source system (execute in the given order):
   on source> dd if=/dev/sda | gzip -c --fast | nc -l 19001 on target> nc [source] 19001 | gzip -d -c | dd of=hd-copy.raw

   Make sure you have a decent network connection, 1 Gbit/s is fine.

4. The next step is to convert the raw image to a VirtualBox image. Do this with the following command (tested with VirtualBox 4.0.4):
   VBoxManage convertfromraw hd-copy.raw hd-copy.vdi --variant Standard
5. If there really was a lot of empty space on the disk, you can compact it with the following command:
   VBoxManage modifyhd hd-copy.vdi --compact

References

Chapter 8 of the VirtualBox manual
Another P2V technique which also works in VMWare.

On making a custom Ubuntu Bootable USB stick

In my upcoming Wicket course the students will use a rental laptop. How do we guarantee that they will be up and running in no time? Colleague Jason had the solution: burn a Ubuntu Live CD to a USB stick!

Though simple this may sound, in the end it took us almost 3 days to put it together. This article gives an overview of the steps I took to create the USB sticks and how I tested them with VirtualBox.

Preparations

For starters you need a fast PC with about 15 GB free diskspace. I used Ubuntu 10.10, but other version probably work similarly. You'll need sudo rights as well. If your PC is somewhat older building the image might take quite some time.

Secondly, install the Ubuntu Customization Kit (package name uck). If you are on Ubuntu 10.10, make sure package gfxboot-dev is installed as well.

Finally you will need a clean Ubuntu iso image. Any Ubuntu, Kubuntu or Xubuntu image will do. This will be your starting point so choose carefully. I took ubuntu-10.10-desktop-i386.iso as I wanted to be sure it ran on older (32 bit) PCs as well.

Be prepared to repeat the whole customization process: log carefully what you do and keep a copy of all content you change or add.

Customizing the CD, first build

Uck puts everything in $HOME/tmp so I created that directory first. Then I simply executed uck-gui. The questions that are asked are quite straight-forward to answer. Don't select too much; to make the resulting image fit on a 1Gb USB stick you will need all the space you have. The last question makes you choose between running a packet manager, running a command shell, or continuing the build. Choose the command shell as it makes it a lot easier to keep track of your changes. Tip: use the history command to see what you did.

Removing packages

In the command shell you can remove packages and make other customizations. The Ubuntu wiki gives much information. The shell is chrooted to the new filesystem, so everything you do in that shell is restricted to the image you are creating. To be extra clear: if you remove a package in the UCK shell, you remove it from the live cd image and not from the rest of your system.

I purged lots of packages I didn't expect my students would need. Comments with more large packages that are probably not needed in a course are appreciated.

apt-get purge ubuntu-docs openoffice.org-core openoffice.org-java-common openoffice.org-common openoffice.org-writer openoffice.org-help-en-us openoffice.org-calc evolution evolution-common libevolution evolution-data-server evolution-webcal gbrainy gnome-games-common gwibber-service telepathy-gabble telepathy-gabble libtelepathy-glib0 python-telepathy empathy-common hplip-data hpijs hplip-cups hplip

Installing packages

Then I installed Java and cleaned up:

sudo add-apt-repository "deb http://archive.canonical.com/ lucid partner" sudo apt-get update sudo apt-get install sun-java6-jdk apt-get clean

I also removed some example content:

rm -rf /usr/share/backgrounds/* rm -rf /usr/share/example-content/Ubuntu_Free_Culture_Showcase

and placed my own background. See the ubuntu wiki link above for more details.

In /etc/skel I removed the link to the example contents and I added the following to .profile:

# Java export JAVA_HOME=/usr/lib/jvm/java-6-sun # Maven export M2_HOME=/opt/maven3 export PATH=${PATH}:${M2_HOME}/bin export MAVEN_OPTS="-Xmx512m -XX:MaxPermSize=512m"

The next step was to unzip recent Maven, Eclipse and IntelliJ distributions in /opt. The easiest way I found to do this is open a new shell, switch to root with sudo -s and start a nautilus. This gives you convenient access to $HOME/tmp/remaster-root. Make sure all placed files have root:root as owner.

Close the shell with exit (if another shell opens, close this one as well), and the Uck menu will appear again. Select Continue build from the Uck menu. And within a few minutes you have your first $HOME/tmp/remaster-new-files/livecd.iso!

Using the live cd

To test the live cd I installed VirtualBox (I installed 4.0.4 from the Oracle repository). I created a new machine with 8Gb of virtual harddisk space and mounted the new live cd iso file as virtual cd. That's it, just start the machine and play with it.

Once the virtual machine was fully started I selected 'Try Ubuntu'. I used Maven to make sure all required dependencies were sucked in to $HOME/.m2. Hint: I used the following commands in the largest example project:

mvn install mvn eclipse:eclipse -DdownloadSources=true mvn jettty:run

To ease starting Eclipse and IntelliJ I created a launcher on the desktop. Just right-click the desktop and select 'create launcher'. This will create two .desktop files on the desktop.

Tip: check the settings of the background image.

Now open a new Nautilus window from the 'Places' menu and open a sftp connection to the guest system (do Ctrl-L and enter something like sftp://host/home/user). This allows you to copy all changed and new content to the guest system for inclusion in the next build. I used this to copy the seeded maven repository and the two new launchers.

Satisfied I had a record of all my changes, I shutdown the virtual host, and applied the changes during a second run of uck-gui.

Making a bootable USB stick

After a few builds, I was confident the live cd was ready. The next step is to create a bootable USB stick. This is made easy with the 'Startup Disk Creator' you will find in the 'System/Administration' menu. If you want the USB stick to be useful without installing Ubuntu to the guest computer, make sure there is plenty of space for writing changes. You will probably need a 2GB stick, unless you remove many more packages a 1Gb stick does not leave enough space for any serious development.

Test the USB stick by booting your computer from it.

Testing the installation from the USB stick turned out to be much hard then expected. I didn't have a spare laptop, and VirtualBox's BIOS does not support booting from a USB device. The easiest way I found to circumvent this was to create a vdi image from the stick in the following way:

sudo dd if=/dev/sdb of=raw-usb-image.bin sudo chown user:user raw-usb-image.bin VBoxManage convertfromraw raw-usb-image.bin usb-image.vdi

You can now mount the vdi file to your virtual machine (make sure its the first drive) and boot from it.

Cloning the USB

Once you have created the USB stick, you can clone it easily with the dd command.

Be careful! Selecting the wrong device may kill your harddisk! You have been warned.

# Copy the USB to an image on disk sudo dd if=/dev/sdb of=raw-usb-image.bin # Write it out the another USB image sudo dd if=raw-usb-image.bin of=/dev/sdc

A USB 2.0 hub with many ports helps because the throughput of the USB channel is much higher then the write speed of the sticks. For me, writing 8 USB sticks took the same time as a single stick.

USB cloning in action

Update 2011-03-27 One problem with cloning disks like this is that they all get the same UUID. In some situations this may give problems. I have yet to find a solution for this. I did not find a tools for just changing the UUID of a FAT file system.

Conclusions

With the help of the Ubuntu Customization Kit, creating a customized Ubuntu startup USB stick becomes a surprisingly straight forward -though lengthy- process. Make sure you can repeat your steps as you probably need to. I found VirtualBox to be an excellent tool to test both the Live CD and the bootable USB sticks.