Spring message resource weirdness

Today I got a bug report that e-mail subject appeared in Dutch while they should have been send in English. The first thing I check of course was whether the correct locale was used. It was.

Here is the situation:

• Spring message source
• 2 files: email-subjects.properties and email-subjects_nl.properties
• requested locale: en_US

After a bit of debugging I noticed that Spring looks for the following files (in order):

1. email-subjects_en_us.properties
2. email-subjects_en.properties
3. email-subjects_nl_NL.properties
4. email-subjects_nl.properties
5. email-subjects.properties

WTF?! It also looks for the default locale! Thank you Spring! This is really the first time you disappoint me (in 4 years, not bad). But of course I should have looked further, when will Spring ever let you down?

The solution? Well luckily the authors of the code realized this is not always what you want. An easy setFallbackToSystemLocale(false) overrides this funny behavior. I added <property name="fallbackToSystemLocale" value="false" /> to my config file and it all worked. Hurray for Spring!

Update 2008-08-04: adapted the article to the real solution.

JPA and bit operations (Hibernate/Postgres)

Update: Don't use the code in this article as it is flawed. See the comments for more tips.

In my current data model one of the entities has a column of integer type that contains a large set of boolean flags. Of course this is well hidden in the domain model objects, so you can just call things like event.isCanceled().

However, in the administration interface we need to search for this entity based on separate flags. The JPA query I found in our code base is select ... where flagBits = 1 (canceled is expressed in bit 0). Oops. What if the entity is not only canceled, but also sold out (bit 1)? Quite wrong, lets fix it.

So, I rewrote the where clause to flagBits & 1 = 1 (& is the binary and operation in Postgres). Hibernate, our JPA implementation parses the query to do its ORM magic. Unfortunately, Hibernate does not understand the & operation (though I heard it did in Hibernate 2) so it will throw an exception.

Looking through the Hibernate and Postgres documentations I found the get_bit operation. It does not except integer types as their value, so a cast was needed: get_bit(''||flagBits, 1) == 1. Another exception. And hopelessly wrong as well, ''||flagBits creates a string with the decimal representation of the number, the function get_bit expects a byte array. Maybe: get_bit((''||flagBits)::bytea, 1) == 1. Not really, the problem is still there. And furthermore, Hibernate also does not support the || and :: operations, more exceptions from Hibernate. (The :: could be replaced by a supported cast operation.)

Back to basics. How can you access a given bit with more basic, supported operations? Fact: division by 2x will shift the bits right by x. Fact: a number modulo 2 gives 1 if bit 0 is set.

What follows is the where clause (flagBits / bitmask) % 2 = 1. Where bitmask is 1 for bit 0, and 2 for bit 1, etc. Oops, % is not supported either! Last attempt: mod(flagBits / bitmask, 2) = 1. Success at last.

Conclusion

• JPA/Hibernate give an extensive but still limited subset of SQL expressiveness.
• It pays off to pay attention during math classes.

Ubuntu server 7.10 to 8.04 upgrade experiences

Yesterday I upgraded my server at home (running 3 websites and a small e-mail server) from Ubuntu 7.10 to 8.04. My main system is a nice looking iMac, completed with an eeepc for traveling and kitchen table work. But I use Ubuntu in the closet and at work.
The upgrade went rather well. Of course, I had installed as little packages from outside the repositories as possible. I even did the upgrade over ssh while that is not recommended.

Unfortunately, I only just noticed that no e-mail was being received anymore. After some head scratching (I had also changed the firewall settings), I decided to compare my backup of /etc (yes, I had one!) to the current copy.

So how do you compare such a complex directory as /etc on a Mac? I tried Eclipse and IntelliJ. Despite their excellent compare tools, they have no such thing for complete directories. Searching the web for open source solutions did not give any conclusive evidence as well. Kdiff3 was mentioned most frequently, so I decided to try that one. Kdiff3 can be installed with MacPorts, but make sure you have plenty of time. It took more then more then 2 hours on this screaming iron!
Kdiff3 is not so intuitive as the commercial products Araxis and Beyond Compare (which I have used professionally), but once you get used to it, it is a nice and powerful merging tool.

And there it was: a small copy/paste error in my dovecot configuration. Postfix uses dovecot for authentication so every TLS connection failed. Copy, paste, restart, done!

Conclusions

• Make sufficient backups of all your configuration files.
• Use the packages from your distribution as much as possible. Do not install later versions of a package unless you are prepared to install it again and again and again.
• Kdiff3 is an acceptable diff tool on the Mac.
• Do not be afraid for an Ubuntu server upgrade!

Do not offer money!

Money and open source, always an interesting combination. But how do we, simple developers (with no ambition but to make beautiful stuff and live from it), deal with this? Let me offer you my opinion.

Lets start with a real example on what you should not do. Here is a quote from an e-mail that was sent on the user list of well respected project xyz (which it is, is not important):

Hi!

I was wondering how I could make a contribution for xyz. I'm not talking about a code contribution but rather a small money contribution. I have got a lot of help here on this forum and in fact I don't think I have ever experienced this kind of help elsewhere!

Thanks to all that have helped and especially core xyz coders.

The author obviously has good intentions, so how can this be so disastrously wrong? Suppose you are celebrating Christmas (or any other traditional family event) at your mother-in-law´s house. She cooked really well and everything tasted just wonderfully. You stand up and say, "Mother, this was excellent. This must have been worth at least 400 euros. Here, this is for you." and you hand here 4 fresh 100 euros bills. You will probably understand that your mother-in-law will not be happy, and in fact she will likely not forget this for a long time. Will she have the same pleasure in cooking for you next year? I think not.

So what is the problem? As Dan Ariely, a professor of behavioral economics wrote in his excellent book Predictably Irrational (I read the Dutch translation "Waarom we altijd tijd te kort komen") there are 2 sets of norms, market norms and social norms. Market norms regulate how we interact with each other when money is involved. Social norms regulate all other interactions. So eating at a restaurant will fall under the market norms, eating at your mother-in-low falls under the social norms. It is both about eating, but if you think about it, there are vast differences in what you expect, how you deal with bad food, etc.

The coders of xyz do all of their hard work in their free time. They are not getting paid, they do this purely out of love for what they are creating, and perhaps because of the idea that it is useful to others, and perhaps because of the respect they will receive. Any social interaction therefore falls under the social norms. Now if you start to offer money, you will move the social interactions to market norms. Luckily the core coders of project xyz are wise and responded with a request to donate to a charity project. But suppose they accepted the money. Now why would they do their best next time when no money is involved? They might as well go drinking beer with friends. Or worse, why would non core contributors do their best if they do not get paid? If you know somebody else is getting paid for the same task, even if it only takes 1 minute, why would you spend any brain power on it?

What is even worse, after switching to market norms, it takes a lot of time before the social norms are reinstated. Here is quote from Eric Daugherty, author of the Java Email Server, on JES version 2.0:

I started down this path with passion and drive. After I'd completed the SMTP (I think) portion of the code base, I was contacted by Andrew Oliver about my interest in working on a JBoss mail server. He was working on a new project to build an enterprise class mail (and calender, etc) server build on JBoss (where he was employed at the time). This project, JBossMailServer at the time, seemed to eclipse what I was attempting, and given my already slowing momentum on the 2.0 branch, pretty much brought it to a screeching halt. I had a bit of interest in working on the JBoss version, but in truth some of the motivation for the projects that I work on is ego. Working on a project that wasn't 'mine' didn't quite have the same appeal, even if it was JBoss (remember, this was 4 years ago).

Apparently JBoss had found someone interested in the server, as they approached me with an offer to pay me for the task of completing the SMTP (or POP, I don't recall) functionality. Since I was a pretty easy task for me (I'd done it once already), and hey, they were paying me, I jumped on board. I worked through the deliverable they wanted and earned a little spending money.

However, a funny thing happened. Once I'd worked for pay on the project, it was really hard to get excited about working for free. Combine this with the lack of real ego payoff, and I drifted away from the project.

So when the previous state is restored (the money is gone), the market norms stay for quite a while. This is proven by Dan and colleagues and this is also Eric´s own observation. A troubling conclusion: by offering money for free services the result is less services.

Suppose you care for the success of an open source project, what can you do? My answer is simple: contribute by writing documentation, by answering questions, by writing patches, provide hosting and otherwise interact according to the social norms. If you are not able to do any of these things, it is better to stay out!

Acknowledgments
Thanks to Dan Ariely for his research. The Christmas dining example is from his book.

Everything about Wicket internationalization

I have been using Wicket for about one and a half year (and followed it about a year longer), and only recently I realized the tremendous flexibility of Wicket. The core developers keep introducing major features by replacing large and small parts of the core, without changing much of the existing programming interface. It doesn't stop there though. Any good developer can do the same with relatively little effort. On occasion I replaced important parts of the core myself, just because I wanted it a little bit different.

Upcoming release 1.4 is all about introducing the long awaited generic models. But the release notes of Wicket 1.4-m1 contain another little gem. Created by non-core developer John Ray: child components within a wicket:message element. But before I explain what that means, lets first look at your internationalization options in Wicket.

This article requires you to have a very basic grasp of Wicket (what the application class is, how wicket:id is used, and simple components like Panel and Label). A bit of knowledge on Java properties helps as well.

Simple texts: wicket:message element, and wicket:message attribute
A simple web page easily contains dozens of texts that need i18n. It would be cumbersome to have to put a wicket:id on all of these, and then add a Label to each. Instead you can use the wicket:message element in the HTML file directly. For example in MyPanel.html:

<wicket:message key="helloworld">Hello</wicket:message>

The key (helloworld) is used to lookup the message in the property files. The default text (Hello) is used when the message could not be found.
The property files are (MyPanel.properties):

helloworld: Hello world

And for Dutch(MyPanel_nl.properties):

helloworld: Hallo wereld

The ouput with an English locale:

Hello world

You can do the same for HTML attributes. For example:

<input type="submit" value="Default text" wicket:message="value:helloworld"/>

Would result in:

<input type="submit" value="Hello world"/>

If there are multiple attributes to translate, add them prepended with a comma:

<input type="submit" value="Default text" wicket:message="value:helloworld,title:hellotitle"/>

Wicket's locale selection
As usual in Java i18n systems, messages are looked up by locale. The locale is automatically extracted from the HTTP request, but can also be explicitly set with getSession().setLocale(Locale.US).
If you need to override the locale for a specific component, override getLocale() on that component and it will be used by that component and its children.

If a client does not provide a preferred Locale, Java's default Locale is used. See Locale.getDefault() and Locale.setDefault(Locale)) for more information.

Finding the message
Another extremely powerful concept is how the message is looked up in the property files. Normally a properties file with the same name as the current component is used. E.g. if you are working within the component SummaryPanel, the message is looked up in the file SummaryPanel_nl_NL.properties (for Dutch, Netherlands locale). If that fails (either because the file does not exist, or because it does not contain the message), file SummaryPanel_nl.properties, and then SummaryPanel.properties are investigated. This is all just like Java's resource bundles work. But of course Wicket goes further. If the message is still not found, the property files of the parent class are investigated, and again, and again, all the way up to java.lang.Object. For each class all locale variants are searched for.

Components are reusable, but in order to make it really reusable you should be able to override the messages depending on where the component is used. This is facilitated by first looking up the message (following the algorithm above) for every parent in the component hierarchy (aka page hierarchy). Every component can override the messages of its child components, so the search starts at the page's properties and then trickles down to the component that uses it (yes, its top-down). In order to make overrides specific to a certain child component, you can prefix the message key with the component id of the child. See ComponentStringResourceLoader for more details.

If no message was found in the page hierarchy, another search starts which will look at your application class and its super classes. So Wicket first looks at MyApplication.properties (provided MyApplication is the name of your application) and then up the class hierarchy, passing org.apache.wicket.Application, up to java.lang.Object. This is how Wicket provides its many default i18n texts.

This might sound complicated, but in practice you simply have one properties file per page and some more for components that are reused over multiple pages. For smaller applications you can even put everything in one properties file. These rules work so well that you just do what you think is correct and it almost always just is.

One note on the location of the properties files: like HTML files, they must be in the same package (and same classloader) as the component they are associated with. In practice they live next to each other in the same directory.
If you want Wicket to get its resources from somewhere else (e.g. from a database), you can implement the interface org.apache.wicket.resource.loader.IStringResourceLoader and configure this in the init() method of your application class.

References: ExtensionResourceNameIterator, ComponentStringResourceLoader, and for real control freaks the new PackageStringResourceLoader.

Reloading and caching
When Wicket is started in development mode, changed properties files are detected and reloaded. To properly make use of this feature from an IDE, you should run Wicket directly from the compiled sources, for example with the Start file, included in every QuickStart. In Eclipse you just save the properties file, in IntelliJ you must do a make (Ctrl-F9) before changes are picked up. In production mode, resolved properties are heavily cached for performance. (Same applies to html files.)

Putting dynamic values in the messages
As soon as you need to add values to the messages, you also need to add some Java code. The java code provides the vales, but the rest of the text still comes from the properties file. One of the nice things here is how Wicket leverages java bean properties.

Here is a complete example. Not many frameworks make this so easy!

MyPanel.properties: summ: You, and ${otherCount} others, reviewed '${title}' \ and rated it ${rate}. MyPanel.html: <span wicket:id="summary">Text that will be replaced.</span> MyPanel.java: // Summary has getters for otherCount, title, etc. Summary summary = ...; add(new Label("summary", new StringResourceModel( "summ", this, new Model(summary))));

Resulting in something like:

<span>You, and 5 others, reviewed 'Wicket in Action' and rated it excellent.</span>

Property based message key It goes further: do you know a framework that can do this?

MyPanel.properties: summ.short: Thanks! summ.long: You, and ${otherCount} others, reviewed '${title}' \ and rated it ${rate}. Thanks! MyPanel.html: <span wicket:id="summary">Text that will be replaced.</span> MyPanel.java: // summary.getMsgPrefs().getStyle() returns "short" or "long" add(new Label("summary", new StringResourceModel( "summ.${msgPrefs.style}", this, new Model(summary))));

And more! The Java property syntax is also still available (e.g. like {0,Date}, {2,number,###.##} etc.)? You can find all forms in the javadocs of StringResourceModel.

The trouble with property files
Now this is all nice, but there is one problem you will frequently encountered while using string resources. In this problem you have a sentence that contains one or more dynamically constructed parts (for example some links). The order of these dynamic parts is potentially different for each locale. In the previous example, we may want to link to an information page on the title, and on the word 'others' open a modal window with a list of people.

If you try this with Wicket 1.3 (or any other Java framework I know of that uses resource bundles) you'll find that there is actually no pure way to do this. I know of 3 workarounds, but neither is very attractive. Lets review them. The first way is to split the texts before, after and between the dynamic components. This works good enough, but varying the order of the components is not possible unless you so something clever on the Java side, or have multiple HTML files. The latter is also workaround number 2: for each locale use a separate HTML file with the translation directly included (note: Wicket uses the same rules to lookup HTML files as it does for property files, e.g. MyPanel_nl.html just works like you expect it to). However, having more then one HTML file per component often leads to maintenance horror as changes must be synchronized over many files. The third workaround is to add placeholders to the message text that later are replaced by some HTML. That HTML is either hand written (in which case you may wonder why you started with Wicket at all), or you need to do some serious Wicket hacking to get components to render to a StringBuffer.

Now lets do this with Wicket 1.4-m1:

MyPanel.properties: summ: You, and ${othersLink}, reviewed ${titleLink} \ and rated it ${rate}. others: ${otherCount} others MyPanel.html: <wicket:message key="summ">Text that will be replaced. <span wicket:id="rate">rate</span> <a href="#" wicket:id="titleLink"> <span wicket:id="titleLabel">label</span></a> <a href="#" wicket:id="othersLink"> <span wicket:id="othersLabel">label<</span></a> </wicket:message> MyPanel.java: // Note, we directly add the embedded components // rate, othersLink and titleLink add(new Label("rate", new PropertyModel(summary, "rate"))); Link othLink = new Link("othersLink") { .... } add(othLink); othLink.add(new Label("othersLabel", new StringResourceModel( "others", this, new Model(summary)))); ExternalLink titleLink = new ExternalLink( "titleLink", summary.getTitleUrl()); add(titleLink); titleLink.add(new Label("titleLabel", new PropertyModel(summary, "title")));

The html file contains a wicket:message element with some embedded components. All text within the element is removed and replaced by the text from the properties file. Labels like ${othersLink} are replaced by the rendered component of the same wicket:id. That component must be embedded in the wicket:message element. Note that the order of the components is irrellevant. Also note that you will not see any reference to the wicket:message element in the java code.

If you want runnable code you can download the complete example code (a Maven 2 project based on Wicket's QuickStart). See below for instructions.

Getting string resources from code
Despite all of the above, there always remain some cases in which you need direct access to the messages. Luckily Wicket provides access through the Localizer. You can get the localizer from any component with getLocalizer().

Again, see the complete example code for an example.

Encoding troubles
Fairly unknown to beginning programmers is that you are only allowed to use ISO-8859-1 encoding in java properties files. If you live in Europe this is a fairly annoying as many languages have characters that are not known to ISO-8859-1 (for example the euro symbol €). The simple workaround is escaping: cree\u00EBr instead of creeër. (I always use this site to look up the ISO codepoint.)

But imagine you are making a site in Thai! Luckily Wicket can also read XML property files. Here is a fragment of the Thai properties that comes with Wicket:

<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE properties SYSTEM "http://java.sun.com/dtd/properties.dtd"> <properties> <entry key="Required">ข้อมูลใน ${label} เป็นที่ต้องการ.</entry> </properties>

Nice!

Sample code
To test the code in this article I created a small test application. Unzip it, and run mvn jetty:run to start it. When it is started access it on http://localhost:8080/i18ntest.

Changing resource settings
Finally, if you need more power, you can change all of Wicket's settings in the init() method of your application. Call getResourceSettings() to get a IResourceSettings instance. Let look at some of the options.

ThrowExceptionOnMissingResource: this will make Wicket throw an exception when a resource is missing. At first this may seem a convenient way to test that you listed all messages in a properties file. However, many standard Wicket components use defaults as fall back, so this option is mostly useless. Alternatively, watch for warnings in the log.

UseDefaultOnMissingResource: this will make Wicket use the default value (e.g. the text within the wicket:message element) when the resource is not found in a properties file. Setting this to true (the default) may hide errors for a long time, but setting this false will make your site not work if you made an error. Choose carefully.

There are many more options that allow you to change what, when and how properties are loaded. I have never found a use for these, but they are there if you need them.

Conclusion
As you have seen in this article, Wicket provides simple ways to do everything around internationalization, and some more less simple ways to completely customize this for the rare case you need it. Furthermore, the new Wicket 1.4 release will make it a lot better with support for componenta embedded in a wicket:message element. You can download the example to see everything in action.

Update 20080526: Incorperated the comment from Stefan Fußenegger on what happens if the user gives no preferred locale.

Update 20080527: Scott Swank put this article on the Wicket Wiki. Feel free to edit it there!

Update 20091203: Corrected text after Satish_j pointed out that I made a glaring mistake in the order messages are looked up in the page hierarchy.

Announcement: radiantpostfixextension release 1.0

I have just released a new extension for Radiant. From the description:

With this Radiant extension you can manage a small Postfix virtual domain. A Postfix virtual domain allows you to forward e-mail to another e-mail address. The extensions adds a tab to the admin screens of Radiant in which you to enter the redirects. It also supports crude but effective aliases and mail-lists. In addition the Postfix configuration is verified and instructions will be displayed if something is not right.

Here is an screendump to see the effect.

The first release, boldly named 1.0, should work under Radiant 0.6.3 and up.

Please visit the Radiant Postfix Extension project home.

Update 2008-04-21: just released version 1.1 as a file had the wrong name.

Installing Maven under Mac OSX

Update 2011-09-12: Forget everything that's below here. Just install HomeBrew and type brew install maven.

Update 2008-03-28: Instead of following this article, you are probably better of with MacPorts. Thanks for the suggestion Nino!

I did not know it could be this hard to install Maven. I've installed Maven a bunch of times already but never before on a Mac. It took me about 1 hour!

Here is what you need to do:

• Download the apache-maven.tar.gz. If you follow the link on the maven download page both Firefox and Safari will get an html page but save it as a tar file. Look in the downloaded file (search for the string "apache-maven") and extract a real download link. When in doubt run md5 apache-maven-version.tar.gz on the result and compare it to the MD5 that is posted on the download page.
• Extract the tar somewhere. For example with tar xzf apache-maven-version.tar.gz. Here it gives an error (tar: A lone zero block at 3083) but so far it does not seem to affect the result.
• Ignore all the babbling about environment.plist files, and just create a ~/.bash_profile and put something like the following in it:
  export M2_HOME=/Users/erik/apps/apache-maven-2.0.8 export M2=$M2_HOME/bin export JAVA_HOME=/Library/Java/Home/ export PATH=$PATH:$M2

That's it. Good luck!