JPA and bit operations (Hibernate/Postgres)

Update: Don't use the code in this article as it is flawed. See the comments for more tips.

In my current data model one of the entities has a column of integer type that contains a large set of boolean flags. Of course this is well hidden in the domain model objects, so you can just call things like event.isCanceled().

However, in the administration interface we need to search for this entity based on separate flags. The JPA query I found in our code base is select ... where flagBits = 1 (canceled is expressed in bit 0). Oops. What if the entity is not only canceled, but also sold out (bit 1)? Quite wrong, lets fix it.

So, I rewrote the where clause to flagBits & 1 = 1 (& is the binary and operation in Postgres). Hibernate, our JPA implementation parses the query to do its ORM magic. Unfortunately, Hibernate does not understand the & operation (though I heard it did in Hibernate 2) so it will throw an exception.

Looking through the Hibernate and Postgres documentations I found the get_bit operation. It does not except integer types as their value, so a cast was needed: get_bit(''||flagBits, 1) == 1. Another exception. And hopelessly wrong as well, ''||flagBits creates a string with the decimal representation of the number, the function get_bit expects a byte array. Maybe: get_bit((''||flagBits)::bytea, 1) == 1. Not really, the problem is still there. And furthermore, Hibernate also does not support the || and :: operations, more exceptions from Hibernate. (The :: could be replaced by a supported cast operation.)

Back to basics. How can you access a given bit with more basic, supported operations? Fact: division by 2x will shift the bits right by x. Fact: a number modulo 2 gives 1 if bit 0 is set.

What follows is the where clause (flagBits / bitmask) % 2 = 1. Where bitmask is 1 for bit 0, and 2 for bit 1, etc. Oops, % is not supported either! Last attempt: mod(flagBits / bitmask, 2) = 1. Success at last.

Conclusion

• JPA/Hibernate give an extensive but still limited subset of SQL expressiveness.
• It pays off to pay attention during math classes.

Ubuntu server 7.10 to 8.04 upgrade experiences

Yesterday I upgraded my server at home (running 3 websites and a small e-mail server) from Ubuntu 7.10 to 8.04. My main system is a nice looking iMac, completed with an eeepc for traveling and kitchen table work. But I use Ubuntu in the closet and at work.
The upgrade went rather well. Of course, I had installed as little packages from outside the repositories as possible. I even did the upgrade over ssh while that is not recommended.

Unfortunately, I only just noticed that no e-mail was being received anymore. After some head scratching (I had also changed the firewall settings), I decided to compare my backup of /etc (yes, I had one!) to the current copy.

So how do you compare such a complex directory as /etc on a Mac? I tried Eclipse and IntelliJ. Despite their excellent compare tools, they have no such thing for complete directories. Searching the web for open source solutions did not give any conclusive evidence as well. Kdiff3 was mentioned most frequently, so I decided to try that one. Kdiff3 can be installed with MacPorts, but make sure you have plenty of time. It took more then more then 2 hours on this screaming iron!
Kdiff3 is not so intuitive as the commercial products Araxis and Beyond Compare (which I have used professionally), but once you get used to it, it is a nice and powerful merging tool.

And there it was: a small copy/paste error in my dovecot configuration. Postfix uses dovecot for authentication so every TLS connection failed. Copy, paste, restart, done!

Conclusions

• Make sufficient backups of all your configuration files.
• Use the packages from your distribution as much as possible. Do not install later versions of a package unless you are prepared to install it again and again and again.
• Kdiff3 is an acceptable diff tool on the Mac.
• Do not be afraid for an Ubuntu server upgrade!

Do not offer money!

Money and open source, always an interesting combination. But how do we, simple developers (with no ambition but to make beautiful stuff and live from it), deal with this? Let me offer you my opinion.

Lets start with a real example on what you should not do. Here is a quote from an e-mail that was sent on the user list of well respected project xyz (which it is, is not important):

Hi!

I was wondering how I could make a contribution for xyz. I'm not talking about a code contribution but rather a small money contribution. I have got a lot of help here on this forum and in fact I don't think I have ever experienced this kind of help elsewhere!

Thanks to all that have helped and especially core xyz coders.

The author obviously has good intentions, so how can this be so disastrously wrong? Suppose you are celebrating Christmas (or any other traditional family event) at your mother-in-law´s house. She cooked really well and everything tasted just wonderfully. You stand up and say, "Mother, this was excellent. This must have been worth at least 400 euros. Here, this is for you." and you hand here 4 fresh 100 euros bills. You will probably understand that your mother-in-law will not be happy, and in fact she will likely not forget this for a long time. Will she have the same pleasure in cooking for you next year? I think not.

So what is the problem? As Dan Ariely, a professor of behavioral economics wrote in his excellent book Predictably Irrational (I read the Dutch translation "Waarom we altijd tijd te kort komen") there are 2 sets of norms, market norms and social norms. Market norms regulate how we interact with each other when money is involved. Social norms regulate all other interactions. So eating at a restaurant will fall under the market norms, eating at your mother-in-low falls under the social norms. It is both about eating, but if you think about it, there are vast differences in what you expect, how you deal with bad food, etc.

The coders of xyz do all of their hard work in their free time. They are not getting paid, they do this purely out of love for what they are creating, and perhaps because of the idea that it is useful to others, and perhaps because of the respect they will receive. Any social interaction therefore falls under the social norms. Now if you start to offer money, you will move the social interactions to market norms. Luckily the core coders of project xyz are wise and responded with a request to donate to a charity project. But suppose they accepted the money. Now why would they do their best next time when no money is involved? They might as well go drinking beer with friends. Or worse, why would non core contributors do their best if they do not get paid? If you know somebody else is getting paid for the same task, even if it only takes 1 minute, why would you spend any brain power on it?

What is even worse, after switching to market norms, it takes a lot of time before the social norms are reinstated. Here is quote from Eric Daugherty, author of the Java Email Server, on JES version 2.0:

I started down this path with passion and drive. After I'd completed the SMTP (I think) portion of the code base, I was contacted by Andrew Oliver about my interest in working on a JBoss mail server. He was working on a new project to build an enterprise class mail (and calender, etc) server build on JBoss (where he was employed at the time). This project, JBossMailServer at the time, seemed to eclipse what I was attempting, and given my already slowing momentum on the 2.0 branch, pretty much brought it to a screeching halt. I had a bit of interest in working on the JBoss version, but in truth some of the motivation for the projects that I work on is ego. Working on a project that wasn't 'mine' didn't quite have the same appeal, even if it was JBoss (remember, this was 4 years ago).

Apparently JBoss had found someone interested in the server, as they approached me with an offer to pay me for the task of completing the SMTP (or POP, I don't recall) functionality. Since I was a pretty easy task for me (I'd done it once already), and hey, they were paying me, I jumped on board. I worked through the deliverable they wanted and earned a little spending money.

However, a funny thing happened. Once I'd worked for pay on the project, it was really hard to get excited about working for free. Combine this with the lack of real ego payoff, and I drifted away from the project.

So when the previous state is restored (the money is gone), the market norms stay for quite a while. This is proven by Dan and colleagues and this is also Eric´s own observation. A troubling conclusion: by offering money for free services the result is less services.

Suppose you care for the success of an open source project, what can you do? My answer is simple: contribute by writing documentation, by answering questions, by writing patches, provide hosting and otherwise interact according to the social norms. If you are not able to do any of these things, it is better to stay out!

Acknowledgments
Thanks to Dan Ariely for his research. The Christmas dining example is from his book.