Monday, August 15, 2011

DNS for Version 99 is off-line

To alleviate some pain with using the commons-logging framework, I created version 99. It was hosted at the hostname no-commons-logging.zapto.org courtesy of no-ip.com. Unfortunately, due to lack of traffic I had to affirm usage of the the hostname once per month. Though this was slightly annoying, a complete pain is that I missed the last deadline, and that recreation is out as dashes in hostnames are no longer allowed.

As of 2011-08-15 16:11:00 GMT no-ip did credit to its name; no ip for no-commons-logging.zapto.org.

As 1) the current situation is disruptive for any version-99 user anyway, 2) there is a better workaround using the provided scope, I decided to not put back version 99 online under another hostname.

If you really need to use version 99 for some more time, just add an entry to your /etc/hosts file:

83.163.41.27 no-commons-logging.zapto.org

As promised, I'll keep version 99 for 5 years, that is until October 2012. Please check back here for IP address changes.

Update 2011-12-19: I just found an alternative with almost the same name: version99.qos.ch. It is a static maven repository with a limited number of version 99 jars.

Thursday, June 16, 2011

Lock-less singleton pattern

Although the singleton pattern is now know as an anti-pattern, I think it still is a valid choice when you only need one instance in a particular context. Anyway, in Java there are several ways to implement a singleton.

For example this one uses a synchronized block:

private Singleton singleton = null; protected Singleton createSingleton() { synchronized (this) { // locking on 'this' for simplicity if (singleton == null) { singleton = new Singleton(); } return singleton; } }

To prevent locking all the time you can use the double-check idiom on a modern JVM. But I just thought about another implementation that switches locking for preventing code re-ordering, making it a lock-less implementation:

private AtomicReference singletonRef = new AtomicReference(null); protected Singleton createSingleton() { Singleton singleton = singletonRef.get(); if (singleton == null) { singleton = new Singleton(); if (!singletonRef.weakCompareAndSet(null, singleton)) { singleton = singletonRef.get(); } } return singleton; }

There is only one catch to this implementation: although createSingleton() will always return the same instance, it must be allowed to call the constructor of Singleton twice without side-effects.

Update 2011-06-18: changed compareAndSet to weakCompareAndSet as suggested by Adriano.

Thursday, May 19, 2011

Apache Wicket Cookbook — book review

Some time ago I reviewed the drafts of the new book from Wicket rockstar programmer Igor Vaynberg: Apache Wicket Cookbook. If you are serious about using Wicket, this book is for you. It is fast, to the point, has very clear code samples and teaches you all the relevant (both clean and dirty) stuff you need and which Wicket in Action could not cover.

Conclusion: this is the book to read after 'Wicket in Action'.

Thursday, March 31, 2011

Virtualizing a PC

All hardware is eventually decommissioned. It may be broken, stolen, or just too old and slow. However, the software on it might still be needed. Not all software is easily transferable to a new PC with the latest OS on it. This quick guide helps you convert the old machine to a virtual machine, so that you can run it in VirtualBox. The process is called physical to virtual (P2V).

This article assumes you are at home in a Linux shell.

  1. Boot the old hardware from a Ubuntu live CD or USB stick (or any other live cd, for example the gparted cd).
  2. Prepare copying each file system with the following steps:
    1. Mount the filesystem
    2. Remove the contents of /tmp and trashcans in the home directories.
    3. Clear free space with the following commands:
      sudo -s dd if=/dev/zero of=/usr/bigfile; rm -f /usr/bigfile
  3. Now copy each physical disk to the target host computer (you could also do this per partition). With nc (netcat) and dd this is easy. Netcat essentially opens a pipe between two computers. We then use dd to stream the entire disk to and from the pipe. One of the two netcat's are in listening mode, the other connects to that one. It doesn't matter which one does the listening, except I found that netcat in MacOSX does not properly support listening. To speed up the transfer, gzip is used. Because we cleared empty sections of the disk, gzip is a quite efficient addition. (Update 2013-04-15: Now I would recommend a compression program that compress/decompress faster at the cost of compression efficiency. For example lzop.)
    Here are the two examples of copying the disk /dev/sda. The first puts netcat in listen mode on the target:
    on target> nc -l 19001 | gzip -d -c | dd of=hd-copy.raw on source> dd if=/dev/sda | gzip -c --fast | nc [target] 19001
    The second example puts netcat in listen mode on the source system (execute in the given order):
    on source> dd if=/dev/sda | gzip -c --fast | nc -l 19001 on target> nc [source] 19001 | gzip -d -c | dd of=hd-copy.raw

    Make sure you have a decent network connection, 1 Gbit/s is fine.

  4. The next step is to convert the raw image to a VirtualBox image. Do this with the following command (tested with VirtualBox 4.0.4):
    VBoxManage convertfromraw hd-copy.raw hd-copy.vdi --variant Standard
  5. If there really was a lot of empty space on the disk, you can compact it with the following command:
    VBoxManage modifyhd hd-copy.vdi --compact

References

Chapter 8 of the VirtualBox manual
Another P2V technique which also works in VMWare.

Thursday, March 24, 2011

On making a custom Ubuntu Bootable USB stick

In my upcoming Wicket course the students will use a rental laptop. How do we guarantee that they will be up and running in no time? Colleague Jason had the solution: burn a Ubuntu Live CD to a USB stick!

Though simple this may sound, in the end it took us almost 3 days to put it together. This article gives an overview of the steps I took to create the USB sticks and how I tested them with VirtualBox.

Preparations

For starters you need a fast PC with about 15 GB free diskspace. I used Ubuntu 10.10, but other version probably work similarly. You'll need sudo rights as well. If your PC is somewhat older building the image might take quite some time.

Secondly, install the Ubuntu Customization Kit (package name uck). If you are on Ubuntu 10.10, make sure package gfxboot-dev is installed as well.

Finally you will need a clean Ubuntu iso image. Any Ubuntu, Kubuntu or Xubuntu image will do. This will be your starting point so choose carefully. I took ubuntu-10.10-desktop-i386.iso as I wanted to be sure it ran on older (32 bit) PCs as well.

Be prepared to repeat the whole customization process: log carefully what you do and keep a copy of all content you change or add.

Customizing the CD, first build

Uck puts everything in $HOME/tmp so I created that directory first. Then I simply executed uck-gui. The questions that are asked are quite straight-forward to answer. Don't select too much; to make the resulting image fit on a 1Gb USB stick you will need all the space you have. The last question makes you choose between running a packet manager, running a command shell, or continuing the build. Choose the command shell as it makes it a lot easier to keep track of your changes. Tip: use the history command to see what you did.

Removing packages

In the command shell you can remove packages and make other customizations. The Ubuntu wiki gives much information. The shell is chrooted to the new filesystem, so everything you do in that shell is restricted to the image you are creating. To be extra clear: if you remove a package in the UCK shell, you remove it from the live cd image and not from the rest of your system.

I purged lots of packages I didn't expect my students would need. Comments with more large packages that are probably not needed in a course are appreciated.

apt-get purge ubuntu-docs openoffice.org-core openoffice.org-java-common openoffice.org-common openoffice.org-writer openoffice.org-help-en-us openoffice.org-calc evolution evolution-common libevolution evolution-data-server evolution-webcal gbrainy gnome-games-common gwibber-service telepathy-gabble telepathy-gabble libtelepathy-glib0 python-telepathy empathy-common hplip-data hpijs hplip-cups hplip

Installing packages

Then I installed Java and cleaned up:

sudo add-apt-repository "deb http://archive.canonical.com/ lucid partner" sudo apt-get update sudo apt-get install sun-java6-jdk apt-get clean

I also removed some example content:

rm -rf /usr/share/backgrounds/* rm -rf /usr/share/example-content/Ubuntu_Free_Culture_Showcase

and placed my own background. See the ubuntu wiki link above for more details.

In /etc/skel I removed the link to the example contents and I added the following to .profile:

# Java export JAVA_HOME=/usr/lib/jvm/java-6-sun # Maven export M2_HOME=/opt/maven3 export PATH=${PATH}:${M2_HOME}/bin export MAVEN_OPTS="-Xmx512m -XX:MaxPermSize=512m"

The next step was to unzip recent Maven, Eclipse and IntelliJ distributions in /opt. The easiest way I found to do this is open a new shell, switch to root with sudo -s and start a nautilus. This gives you convenient access to $HOME/tmp/remaster-root. Make sure all placed files have root:root as owner.

Close the shell with exit (if another shell opens, close this one as well), and the Uck menu will appear again. Select Continue build from the Uck menu. And within a few minutes you have your first $HOME/tmp/remaster-new-files/livecd.iso!

Using the live cd

To test the live cd I installed VirtualBox (I installed 4.0.4 from the Oracle repository). I created a new machine with 8Gb of virtual harddisk space and mounted the new live cd iso file as virtual cd. That's it, just start the machine and play with it.

Once the virtual machine was fully started I selected 'Try Ubuntu'. I used Maven to make sure all required dependencies were sucked in to $HOME/.m2. Hint: I used the following commands in the largest example project:

mvn install mvn eclipse:eclipse -DdownloadSources=true mvn jettty:run

To ease starting Eclipse and IntelliJ I created a launcher on the desktop. Just right-click the desktop and select 'create launcher'. This will create two .desktop files on the desktop.

Tip: check the settings of the background image.

Now open a new Nautilus window from the 'Places' menu and open a sftp connection to the guest system (do Ctrl-L and enter something like sftp://host/home/user). This allows you to copy all changed and new content to the guest system for inclusion in the next build. I used this to copy the seeded maven repository and the two new launchers.

Satisfied I had a record of all my changes, I shutdown the virtual host, and applied the changes during a second run of uck-gui.

Making a bootable USB stick

After a few builds, I was confident the live cd was ready. The next step is to create a bootable USB stick. This is made easy with the 'Startup Disk Creator' you will find in the 'System/Administration' menu. If you want the USB stick to be useful without installing Ubuntu to the guest computer, make sure there is plenty of space for writing changes. You will probably need a 2GB stick, unless you remove many more packages a 1Gb stick does not leave enough space for any serious development.

Test the USB stick by booting your computer from it.

Testing the installation from the USB stick turned out to be much hard then expected. I didn't have a spare laptop, and VirtualBox's BIOS does not support booting from a USB device. The easiest way I found to circumvent this was to create a vdi image from the stick in the following way:

sudo dd if=/dev/sdb of=raw-usb-image.bin sudo chown user:user raw-usb-image.bin VBoxManage convertfromraw raw-usb-image.bin usb-image.vdi

You can now mount the vdi file to your virtual machine (make sure its the first drive) and boot from it.

Cloning the USB

Once you have created the USB stick, you can clone it easily with the dd command.

Be careful! Selecting the wrong device may kill your harddisk! You have been warned.

# Copy the USB to an image on disk sudo dd if=/dev/sdb of=raw-usb-image.bin # Write it out the another USB image sudo dd if=raw-usb-image.bin of=/dev/sdc

A USB 2.0 hub with many ports helps because the throughput of the USB channel is much higher then the write speed of the sticks. For me, writing 8 USB sticks took the same time as a single stick.

USB cloning in action USB cloning in action

Update 2011-03-27 One problem with cloning disks like this is that they all get the same UUID. In some situations this may give problems. I have yet to find a solution for this. I did not find a tools for just changing the UUID of a FAT file system.

Conclusions

With the help of the Ubuntu Customization Kit, creating a customized Ubuntu startup USB stick becomes a surprisingly straight forward -though lengthy- process. Make sure you can repeat your steps as you probably need to. I found VirtualBox to be an excellent tool to test both the Live CD and the bootable USB sticks.

Tuesday, October 5, 2010

Why Functional Programming Matters - An exploration of functional Scala - Part 2

package nl.grons.whyfp
import scala.math.abs
/**

A translation of Miranda examples in chapter 4 of the paper Why Functional Programming Matters by John Hughes, 1984.

This article is the continuation of part 1 - Chapter 3 examples. I will show you lazy evaluation and how this can be useful.

@author Erik van Oosten
*/

object Chapter4_LazyList {

/*

Chapter 4’s examples make heavy use of Miranda’s lazy evaluation. In particular, it uses the list as defined in chapter 3 in a lazy way. This allows the creation of lists of unbounded length. Scala supports lazy lists as well. You can find an implementation in the Stream class. However, as I want to stay close to the original, I redefine the list from chapter 3 as follows:

*/
/** The list from chapter 3, redefined to make it a lazy list. */ sealed abstract class List[+A] { def head: A def tail: List[A] } /** Lazyly evaluate the argument of the constructor AND the value of tail. */ final class Cons[A](hd: A, tl: => List[A]) extends List[A] { override val head: A = hd override lazy val tail: List[A] = tl }
/*

If you compare this to the List implementation in the previous article, you note that parameter ‘tl’ of ‘Cons’ is passed as an by-name parameter (because of the =>). This means that the given expression is only evaluated when the parameter is used. Too prevent the tail expression to be evaluated again and again, the ‘tail’ value is declared as a val. Now ‘tl’ is evaluated exactly once. It is also declared as lazy so that the evaluation of the expression is delayed until the first time it is needed instead of during construction time.

These 3 things together allow the list to behave lazily.

You might also notice that ‘Cons’ is no longer a case class (not possible with by-name parameters). Therefore I define the following:

*/
object Cons { def apply[A](hd: A, tl: => List[A]) = new Cons(hd, tl) // Warning: using unapply leads to evaluation of the tail. def unapply[A](c: Cons[A]) = Some(c.head, c.tail) }
/*

‘Nil’ did not change:

*/
/** The empty list. */ case object Nil extends List[Nothing] { override def head: Nothing = { throw new NoSuchElementException("head of empty list") } override def tail: List[Nothing] = { throw new NoSuchElementException("tail of empty list") } }
/*

Just for fun: here is an example of how the list can be used to construct an infinite list.

*/
/** A list of all integer numbers starting at x and higher. */ // Miranda: nat x = cons x nat(x+1) def nat(x: Int): List[Int] = Cons(x, nat(x + 1))
/*

The Miranda version of ‘nat’ is limited by the available memory, the Scala version is limited to scala.Int.MaxValue.

Function ‘map’ from the previous chapter won’t work. It will try to completely iterate the list which will obviously fail for unbounded lists. Lets implement it again with some simple pattern matching:

*/
/** Create a new lazy list where each element a is replaced by f(a). */ def map_wrong[A, B](f: A => B)(l: => List[A]): List[B] = l match { case Nil => Nil case Cons(head, tail) => Cons(f(head), map(f)(tail)) }
/*

This implementation seemed to work for quite some time, even though everything was slow. If you look carefully, you will see that it evaluates ‘tail’ in ‘Cons.unapply’ which is invoked to match the pattern Cons(head, tail). The result is that exactly one more item in the list is evaluated then necessary. Though this is merely wasteful for most algorithms, it is catastrophic (as in stack overflow) when that evaluation contains another recursive call to ‘map’. This actually happens in the last example of this article.

Here is an implementation that does not have this problem:

*/
/** Create a new lazy list where each element a is replaced by f(a). */ def map[A, B](f: A => B)(l: => List[A]): List[B] = l match { case Nil => Nil case c: Cons[A] => Cons(f(c.head), map(f)(c.tail)) }
/*

Notice that the recursive expression, which includes c.tail, is now used in a place that is lazily evaluated.

On the console:

scala> import nl.grons.whyfp.Chapter4_LazyList._ scala> def printFirst[A](count: Int, l: List[A]) { if (count > 0 && l != Nil) { println(l.head); printFirst(count - 1, l.tail) } } printFirst: [A](count: Int,l: nl.grons.whyfp.Chapter4_LazyList.List[A])Unit scala> printFirst(3, map[Int,Int](_ * 2)(nat(3))) 6 8 10
*/
}

object Chapter4_1_NewtonRaphsonSquareRoots {

import Chapter4_LazyList._
/*

Here’s an imperative Scala implementation that does a numerical calculation of the square root of a number.

*/
/** Approximate the square root of n to within eps * starting with approximation a0. */ def imperative_squareroot(a0: Double, eps: Double, n: Double): Double = { // Warning: imperative scala! var x = a0 // The initial value of y does not matter so long as abs(x-y) > eps var y = a0 + 2 * eps while (abs(x - y) > eps) { y = x x = (x + n/x) / 2 } x }
/*

Here are my translations of the rest:

*/
/** Next approximation of the square root of n, * based on the previous approximation x. */ // Miranda: next N x = (x + N/x) / 2 def next(n: Double)(x: Double): Double = (x + n/x) / 2 /** Produce an infinite list that starts with a and where all other values * are the result of applying f to the previous value. */ // Miranda: repeat f a = cons a (repeat f (f a)) def repeat[A](f: A => A, a: A): List[A] = Cons(a, repeat(f, f(a))) /** Give the first value of the list that is within eps of its preceding value. */ // Miranda: // within eps (cons a (cons b rest)) = // = b, if abs(a-b) <= eps // = within eps (cons b rest), otherwise def within(eps: Double, l: List[Double]): Double = { val a = l.head if (a.isNaN) throw new RuntimeException("nan") val rest = l.tail val b = rest.head if(abs(a - b) <= eps) b else within(eps, rest) }
/*

When a calculation overflow or underflow occurs (not unlikely when working with approximations), ‘within’ goes amok and will loop forever. This is prevented by validating the head of the list against NaN, Not A Number.

*/
/** Approximate the square root of n to within eps * starting with approximation a0. */ // Miranda: sqrt a0 eps N = within eps (repeat (next N) a0) def sqrt(a0: Double, eps: Double, n: Double) = within(eps, repeat(next(n), a0)) /** Gives the first value of the list that of which the ratio of change * with its preceding value is lower then eps. */ // Miranda: // relative eps (cons a (cons b rest)) = // = b, if abs(a-b) <= eps*abs b // = relative eps (cons b rest), otherwise def relative(eps: Double, l: List[Double]): Double = { val a = l.head if (a.isNaN) throw new RuntimeException("nan") val rest = l.tail val b = rest.head if(abs(a - b) <= eps * abs(b)) b else relative(eps, rest) } /** Approximate the square root of n to within ratio eps starting with * approximation a0. */ // Miranda: relativesqrt a0 eps N = relative eps (repeat (next N) a0) def relativesqrt(a0: Double, eps: Double, n: Double) = relative(eps, repeat(next(n), a0))
/*

On the Console:

scala> import nl.grons.whyfp.Chapter4_LazyList._ scala> import nl.grons.whyfp.Chapter4_1_NewtonRaphsonSquareRoots._ scala> relativesqrt(0.1, 0.01, 4) res0: Double = 2.000010925778043
*/
}

object Chapter4_2_NumericalDifferentiation {

import Chapter4_LazyList._
import Chapter4_1_NewtonRaphsonSquareRoots._
/*

This chapter is about numerical differentiation.

*/
/** An approximation of the differentiation of f for x over h. */ // Miranda: easydiff f x h = (f(x+h)-f x) / h def easydiff(f: Double => Double, x: Double)(h: Double): Double = (f(x + h) - f(x)) / h // Miranda: // differentiate h0 f x = map (easydiff f x)(repeat halve h0) // halve x = x/2 def differentiate(h0: Double, f: Double => Double, x: Double) = map(easydiff(f, x))(repeat(halve, h0)) def halve(x: Double) = x / 2
/*

On the console:

scala> import nl.grons.whyfp.Chapter4_LazyList._ scala> import nl.grons.whyfp.Chapter4_1_NewtonRaphsonSquareRoots._ scala> import nl.grons.whyfp.Chapter4_2_NumericalDifferentiation._ scala> val f: Double => Double = math.pow(_, 3.0) f: (Double) => Double = <function1> scala> within(0.001, differentiate(1, f, 2)) res0: Double = 12.000732436776161

Some more examples:

*/
// Miranda: // elimerror n (cons a (cons b rest)) = // = cons ((b*(2**n)-a)/(2**n-1)) (elimerror n (cons b rest)) def elimerror(n: Double, l: List[Double]): List[Double] = { val a = l.head val bandrest = l.tail val b = bandrest.head Cons( (b * math.pow(2.0, n) - a) / (math.pow(2.0, n) - 1.0), elimerror(n, bandrest)) } /** Calculate the order of n to be used in elimerror. */ // Miranda: // order (cons a (cons b (cons c rest))) = // = round(log2( (a-c)/(b-c) - 1 )) // round x = x rounded to the nearest integer // log2 x = the logarithm of x to the base 2 def order(l: List[Double]): Double = { val LOG2: Double = math.log(2.0) def log2(x: Double) = math.log(x) / LOG2 val a = l.head val b = l.tail.head val c = l.tail.tail.head val o = math.round(log2((a-c)/(b-c) - 1)) if (o.isNaN || o == 0) 1 else o } // Miranda: improve s = elimerror (order s) s def improve(s: List[Double]): List[Double] = elimerror(order(s), s)
/*

Note that my implementation of ‘order’ is not only real (no vague descriptions of ‘round’ and ‘log2’), but it is more robust then the original. First, it protects against NaN for the case that b-c reaches 0. Secondly, it prevents a result of 0 as that leads to a division by 0 in ‘elimerror’.

On the console:

scala> within(0.001, improve(differentiate(1, f, 2))) res0: Double = 11.9998779296875 scala> // Improve can be improved recursively scala> within(0.001, improve(improve(improve(differentiate(1, f, 2))))) res1: Double = 12.0
*/
// Miranda: // super s = map second (repeat improve s) // second (cons a (cons b rest)) = b def superimprove(s: List[Double]): List[Double] = map(second)(repeat(improve, s)) def second(l: List[Double]): Double = l.tail.head def superdifferentiate(f: Double => Double, x: Double): Double = { val eps: Double = 0.00000001 within(eps, superimprove(differentiate(x + 2 * eps, f, x))) }
}

object Chapter4_3_NumericalIntegration {

import Chapter4_LazyList._
/*

This chapter is about numerical integration.

*/
// Miranda: easyintegrate f a b = (f a + f b)*(b-a)/2 def easyintegrate(f: Double => Double, a: Double, b: Double) = (f(a) + f(b)) * (b-a) / 2 // Miranda: integrate f a b = cons (easyintegrate f a b) // (map addpair (zip (integrate f a mid) // (integrate f mid b))) // where mid = (a+b)/2 def integrate_simple(f: Double => Double, a: Double, b: Double): List[Double] = { def addpair(pair: (Double, Double)) = pair._1 + pair._2 val mid = (a + b) / 2 Cons( easyintegrate(f, a, b), map(addpair)(zip(integrate(f, a, mid), integrate(f, mid, b))) ) } /** Convert two lists to a list of pairs. */ // Miranda: zip (cons a s) (cons b t) = cons (pair a b) (zip s t) def zip[A,B](as: => List[A], bs: => List[B]): List[(A,B)] = as match { case Nil => Nil case a: Cons[A] => bs match { case Nil => Nil case b: Cons[B] => Cons((a.head, b.head), zip(a.tail, b.tail)) } }
/*

Again note that ‘zip’ was defined such that a.tail and b.tail are defined as a lazily evaluated argument.

*/
// Miranda: // integrate f a b = integ f a b (f a) (f b) // integ f a b fa fb = cons ((fa+fb)*(b-a)/2) // (map addpair (zip (integ f a m fa fm) // (integ f m b fm fb))) // where m = (a+b)/2 // fm = f m def integrate(f: Double => Double, a: Double, b: Double): List[Double] = { def integ(a: Double, b: Double, fa: Double, fb: Double): List[Double] = { def addpair(pair: (Double, Double)) = pair._1 + pair._2 val mid = (a + b) / 2 val fm = f(mid) Cons( (fa + fb) * (b-a) / 2, map(addpair)(zip(integ(a, mid, fa, fm), integ(mid, b, fm, fb))) ) } integ(a, b, f(a), f(b)) }
/*

And of course, seeing is believing:

scala> import nl.grons.whyfp.Chapter4_LazyList._ scala> import nl.grons.whyfp.Chapter4_1_NewtonRaphsonSquareRoots._ scala> import nl.grons.whyfp.Chapter4_2_NumericalDifferentiation._ scala> import nl.grons.whyfp.Chapter4_3_NumericalIntegration._ scala> def f(x: Double): Double = 1/(1+x*x) f: (x: Double)Double scala> within(0.001, integrate(f, 1, 2)) res0: Double = 0.3218612363325556 scala> relative(0.001, integrate(f, 1, 2)) res1: Double = 0.3217782239721789 scala> relative(0.001, superimprove(integrate(f, 1, 2))) res2: Double = 0.3217525935931843
*/
}
/*

Conclusion

Scala is an excellent language for functional style programs and I really like that static typing prevents tons of errors. My only problem with Scala — in terms of the ‘Why Functional Programming matters’ paper — is that lazy evaluation is not the natural way of working. The way expressions are evaluation is more closely to the Java world. To make use of lazy evaluation you have to be very much aware of what your doing and carefully make use of the lazy keyword and by-name parameters. With Miranda this comes naturally though probably at the cost of some performance for the case laziness is not needed. Perhaps that the standard collections library (scala.collection.immutable.Stream) hides this sufficiently and makes life easy.

A more fundamental way to fix this problem would be to express the laziness nature of an expression in its type. I am out of the academic world for a looong time, so I have no idea if this is being researched at all. It will take some hard thoughts to make this practical and simple to use though.

The next challenge would be to repeat this experiment with the standard collections. But I’ll leave that to someone with more time...

*/

Tuesday, September 21, 2010

Why Functional Programming Matters - An exploration of functional Scala.

package nl.grons.whyfp
/**

A translation of Miranda examples in the paper Why Functional Programming Matters by John Hughes, 1984.

@author Erik van Oosten
*/

object WhyFp_Abstract {
/*

Scala is in the focus of attention because it allows a smooth migration from one of the most popular programming languages ever to a language that supports (among other things) a functional style of programming.

In this article I explore the functional aspects of Scala based on a very influential paper on functional programming: ‘Why Functional Programming matters’. To do so I will translate the Miranda examples of chapter 3 and 4 (next article) to Scala. To keep things pure, the standard Scala libraries are avoided.

I will conclude that Scala has sufficient features to live up to Hughes’ standards with the added bonus of being statically type safe.

Just like the paper I just assume you know a bit about functions, partial functions and partially applied functions. In addition I assume you know basic Scala syntax. However, as I am still a Scala newby myself, any comments that will improve this article are welcomed.

This article focusses on the Miranda -> Scala translation and will not repeat the message of the paper. So go ahead, and read the paper first.

All text in this article is properly commented, so the whole article can be copy pasted into your favourite text-editor.

*/
object Chapter3_GlueingFunctionsTogether {
import RichFunction2._
/*

The original article completely defines a list right there in just one line:

listof X ::= nil | cons X (listof X) (Miranda)

Scala is not so powerful and this mainly because we need to translate Miranda’s ‘|’ operator to a class hierarchy.

Lets define List more or less as the Scala library does it (as anonymous points out, it can be done much shorter).

*/
/** The base type for our list. */ sealed abstract class List[+A] { def head: A def tail: List[A] } /** Cons represents an element and the rest of the list. */ case class Cons[A](hd: A, tl: List[A]) extends List[A] { override def head: A = hd override def tail: List[A] = tl override def toString = "Cons(" + hd + "," + tl + ")" } /** Nil represents the empty list. */ case object Nil extends List[Nothing] { override def head: Nothing = { throw new NoSuchElementException("head of empty list") } override def tail: List[Nothing] = { throw new NoSuchElementException("tail of empty list") } override def toString = "Nil" }
/*

Note that Cons is a case class, therefore we can use Cons(x,y) as synonym for the constructor and also as extractor during pattern matching.

We can now use the list as follows:

[] => Nil [1] => Cons(1, Nil) [1,2,3] => Cons(1, Cons(2, Cons(3, Nil)))

As in the original article, what follows is a number of definitions of ‘sum’. I name each definition differently so that this whole article can be compiled as is.

My first approach mimics the Miranda approach with 2 partial functions. Scala does not automatically combine partial functions se we’ll need to do ourselves.

*/
// Miranda: // sum nil = 0 // sum (cons num list) = num + sum list // val sum_nil: PartialFunction[List[Int], Int] = { case Nil => 0 } val sum_cons: PartialFunction[List[Int], Int] = { case Cons(num, list) => num + sum_combinedpartials(list) } def sum_combinedpartials = sum_nil orElse sum_cons
/*

The second, more idiomatic Scala, version wraps this into a single method:

*/
def sum_idiomatic(li: List[Int]): Int = li match { case Nil => 0 case Cons(num, list) => num + sum_idiomatic(list) }
/*

Note that both approaches use pattern matching, just like the Miranda original.

The following definitions of ‘sum’ are more interesting. They are very close to the original example. Note that this actually defines an immutable reference (a val) to a function. In case you are wondering, List[Int] => Int is the type of the function: it accepts one argument, List[Int], and has Int as return type.

*/
// Miranda: sum = reduce add 0 val sum: List[Int] => Int = reduce(add, 0)
/*

Note that nowadays most languages instead of ‘reduce’, use the term ‘foldr’ (fold-right).

Scala has type inference, so if we let the compiler derive the return type, its even closer. Note that we need to tuck an underscore to the end to let the compiler know we really want a partially applied function.

*/
val sum_noTypes = reduce(add, 0) _
/*

And of course we need a definition for ‘add’:

*/
// Miranda: add x y = x + y def add(x: Int, y: Int): Int = x + y
/*

For function ‘reduce’, I again use pattern matching just like the Miranda original. Notice that ‘reduce’ defines 2 parameter lists. One with parameters f and x, and the second with parameter l. This allows for easy creation of partially applied functions. And this is exactly what happened in the definitions of ‘sum’ and ‘sum_noTypes’ above.

*/
// Miranda // (reduce f x) nil = x // (reduce f x) (cons a l) = f a ((reduce f x) l) // /** * Right fold a list with a function f and a constant x. * * @param f a function to apply * @param x the result for an empty list * @param l the list to fold * @param [A] the type of list elements * @param [B] the result type * @return the result of the expression where in l, * each Cons is replaced by f, and Nil by x */ def reduce[A, B](f: (A, B) => B, x: B)(l: List[A]): B = l match { case Nil => x case Cons(head, tail) => f(head, reduce(f, x)(tail)) }
/*

More examples:

*/
// Miranda: product = reduce multiply 1 def product: List[Int] => Int = reduce(multiply, 1)
/*

Just for fun I define ‘multiply’ a little bit different then ‘add’. Where ‘add’ is a method (which can be used as a function), ‘multiply’ is directly defined as a function. Each underscore declares an anonymous parameter. To avoid the ugly (_: Int) syntax, the type of these anonymous parameters are derived form the return type of ‘multiply’.

*/
val multiply: (Int, Int) => Int = _ * _
/*

We can also inline these functions. If you do so the compiler still needs some hints to infer the types of the anonymous parameters. In the following definitions of ‘anytrue’ and ‘alltrue’ I solve that by adding the generic types of ‘reduce’ (the [Boolean,Boolean] part).

With ‘alltrue’ the return type is inferred by the compiler (again, this requires an underscore to confirm we want partial application of ‘reduce’). In the definition of ‘alltrue’, the return type is defined and therefore the underscore is not needed.

Personally I prefer the explicit return type as the function is too complex to just see what it should be. My rule is omit the return type only when it is blindingly obvious (as in ‘add’ above).

*/
// Miranda // anytrue = reduce or false // alltrue = reduce and true // def anytrue: List[Boolean] => Boolean = reduce[Boolean, Boolean](_ || _, false) def alltrue = reduce[Boolean, Boolean](_ && _, true) _
/*

Lets also explore how ‘reduce’ works and calculate the sum of [1,2,3]. Note how we replace Cons with the given function ‘add’ and Nil with the given constant ‘0’.

Starting point: Cons(1, Cons(2, Cons(3, Nil))) Reduce applied: add(1, add(2, add(3, 0 ))) == 6 And for multiply: multiply(1, multiply(2, multiply(3, 1 ))) == 6

Here is my translation of ‘append’.

*/
// Miranda: append a b = reduce cons b a def append[A](a: List[A], b: List[A]): List[A] = reduce[A, List[A]](Cons[A], b)(a)
/*

Note that the first parameter we pass to ‘reduce’ is actually the Cons object (with generic type parameter [A]). This is possible because anything with an ‘apply’ method is recognized as a Scala function.

The generic type ‘[A]’ on Cons is essential; without it the compiler generates nothing but misleading error messages. It took me many hours to figure this one out.

If you are new to this complex kind of type declarations (like I am), you’ll find that careful reading is in order. I am afraid this is just something you need to practice. Also, it helps greatly when you wrote this kind of stuff yourself.

Lets skip ahead to ‘doubleall’.

*/
// Miranda: // doubleall = reduce doubleandcons nil // where doubleandcons num list = cons (2*num) list def doubleall_1: List[Int] => List[Int] = { def doubleandcons(num: Int, list: List[Int]) = Cons(2 * num, list) reduce(doubleandcons, Nil) }
/*

As you can see Miranda’s where keyword nicely translates to nested methods.

Next version:

*/
// Miranda: // doubleandcons = fandcons double // where double n = 2*n // fandcons f el list = cons (f el) list def doubleandcons: (Int, List[Int]) => List[Int] = { def double(n: Int) = 2 * n def fandcons[A, B](f: A => B)(el: A, list: List[B]) = Cons(f(el), list) fandcons(double) }
/*

So far, I had little problem with translating this to Scala. However, the next line needed quite a bit more thought:

fandcons f = cons . f

Scala (Miranda as well) only defines the ‘compose’ method on functions of arity 1. For Miranda this is no problem as all functions written down with multiple arguments are actually syntactic sugar for functions with a single argument that produce further functions with 1 argument. So the type of ‘add’ in

add x y = x + y

is not (Int, Int) => Int as it would be in Scala, but Int => Int => Int.

In our Scala code Cons.apply has 2 arguments and thus arity 2. To be precise, the type of Cons[B] _ is (B, List[B]) => List[B]. (Note in Scala we write Cons[B] _ to indicate we need it as a partially applied function).

In order to apply ‘compose’ we first need to convert Cons[B].apply to something of type B => List[B] => List[B] (a unary function that accepts a B, and gives another unary function that accepts a List[B], and gives a List[B]). This is called currying and is performed with the ‘curried’ function.

The result of currying is a unary function that we can compose with ‘f’. The next step is to uncurry back to a function of arity 2. This is done with ‘uncurried’. Not lets put this in code:

*/
// Miranda: fandcons f = cons . f def fandcons[A, B](f: A => B): (A, List[B]) => List[B] = Function.uncurried( (Cons[B] _).curried.compose(f) )
/*

However, this is very a unwieldy syntax and I am going to need it a lot in the upcoming examples. Therefore, I’ll pimp the Scala library to include ‘compose’ also on functions of arity 2 (see RichFunction2 below).
We can now condense the code to:

*/
// Miranda: fandcons f = cons . f def fandcons_short[A, B](f: A => B): (A, List[B]) => List[B] = (Cons[B] _).compose(f)
/*

And then we try it out with some arguments to see it is correct:

fandcons(f, el)(list) = Cons.compose(f)(el)(list) = Cons(f(el), list)

And finally (no longer defining ‘double’ as a nested function):

*/
// Miranda: doubleall = reduce (cons . double) nil def double(n: Int) = 2 * n def doubleall: List[Int] => List[Int] = reduce((Cons[Int] _).compose(double), Nil) _
/*

The next step is to extract ‘map’:

*/
// Miranda: // doubleall = map double // map f = reduce (cons . f) nil def doubleall_map: List[Int] => List[Int] = map(double) def map[A, B](f: A => B): List[A] => List[B] = reduce((Cons[B] _).compose(f), Nil) _
/*

Map could for example be used to calculate the sum in a matrix, a list of list of int.

*/
// Miranda: summatrix = sum . map sum val summatrix: List[List[Int]] => Int = sum.compose(map(sum))
/*

Here is a fragment of the scala console in which I try out summatrix:

scala> import nl.grons.whyfp.Chapter3_GlueingFunctionsTogether._ scala> val a: List[List[Int]] = Cons(Nil, Cons(Cons(1, Cons(2, Nil)), Cons(Cons(4, Nil), Nil))) a: nl.grons.whyfp.Chapter3_GlueingFunctionsTogether.List[nl.grons.whyfp.Chapter3_GlueingFunctionsTogether.List[Int]] = Cons(Nil,Cons(Cons(1,Cons(2,Nil)),Cons(Cons(4,Nil),Nil))) scala> summatrix(a) res0: Int = 7
/*

Lets move on and create a tree.

*/
//Miranda: treeof X ::= node X (listof (treeof X)) sealed abstract class Tree[+A] { def label: A def children: List[Tree[A]] } case class Node[A](l: A, c: List[Tree[A]]) extends Tree[A] { override def label: A = l override def children: List[Tree[A]] = c }
/*

The tree 1 o / \ / \ 2 o o 3 | | o 4 in Scala would look like:

Node(1, Cons( Node(2, Nil), Cons(Node(3, Cons(Node(4, Nil), Nil)), Nil)))

Here is my version of ‘redtree’:

*/
// Miranda: // redtree f g a (node label subtrees) = // f label (redtree' f g a subtrees) // redtree' f g a (cons subtree rest) = // g (redtree f g a subtree) (redtree' f g a rest) // redtree' f g a nil = a /** * Reduce a tree with functions f and g and constant a. * @param f a function to apply to a node label and * the result of g on the node's subtree * @param g a function to apply to a subtree and * the result of g on the following sibling subtrees * @param a the result for an empty tree * @param tree the tree to reduce * @param [F] the return type of f and redtree itself * @param [G] the return type of g * @param [A] the type of the tree labels * @return the result of the expression where in tree, * each Node is replaced by f, each Cons by g, and Nil by x */ def redtree[F, G, A](f: (A, G) => F, g: (F, G) => G, a: G)(tree: Tree[A]): F = { def redchildren(children: List[Tree[A]]): G = children match { case Cons(subtree, rest) => g(redtree(f, g, a)(subtree), redchildren(rest)) case Nil => a } f(tree.label, redchildren(tree.children)) }
/*

Notice how Scala’s nested functions are more elegant then Miranda’s top level function ‘redtree' ’ (renamed to ‘redchildren’ here); in Scala the arguments of ‘redtree’ are not passed to ‘redchildren’ as the nested function can access them directly.

Lets use redtree to sum all integer labels in a tree of type Tree[Int]:

*/
// Miranda: sumtree = add add 0 val sumtree: Tree[Int] => Int = redtree(add, add, 0)
/*

Next: collecting all the labels in a list:

*/
// Miranda: labels = redtree cons append nil def labels[A]: Tree[A] => List[A] = redtree(Cons[A], append[A], Nil)
/*

And the last example: create a new tree by applying a function to each label:

*/
// Miranda: maptree f = redtree (node . f) cons nil def maptree[A, B](f: A => B): Tree[A] => Tree[B] = redtree((Node[B] _).compose(f), Cons[Tree[B]], Nil)
} */

Parting thoughts

Although it took quite some time to write the code and then translate this to my 100th article, I had great fun doing so. In addition, I learned quite a bit on writing more complex Scala code.

The code for chapter 4 has already been translated, so expect another article soon.

P.S. RichFunction2

As was written above, Scala does not define ‘compose’ on Function2 (the class representing functions with 2 parameters), only on Function1 ‘compose’ is available. To remedy this, we’ll define it ourselves with the pimp-my-library pattern.

*/
class RichFunction2[T1, T2, R](f: Function2[T1, T2, R]) { /** * (f compose g)(x, y) == f(g(x), y) */ def compose[A](g: (A) => T1): (A, T2) => R = { (x: A, y: T2) => f(g(x), y) } } object RichFunction2 { // Note: complete return type is required here because the method // is used before it is declared. implicit def richFunction2[T1, T2, R](f: Function2[T1, T2, R]): RichFunction2[T1, T2, R] = new RichFunction2(f) }