System hardening: Migrating from docker compose to podman-compose

TL;DR: Podman-Compose provides essential security features for homelabbers, albeit with a few inconveniences.

Many applications that appeal to homelabbers can be installed with a docker-compose file. Docker-compose files are awesome! They provide a nice uniform way to install wildly different applications.

The most used runtime is Docker. Unfortunately, docker runs as root and container-escape vulnerabilities are abundant. Therefore, a hacked application immediately leads to a fully compromised system. Luckily, there are viable alternatives. For example, podman explicitly allows running containers without root.

This article summarizes what we learned when we migrated our seven homelab services from docker compose to podman-compose on a Ubuntu 24.04 server.

Preparations

Install Podman and Podman-compose

This is by far the simplest section of this article. As root, run:

apt install podman podman-compose

How to prepare non-root user(s)

Since we want to run our services without root access (rootless), we need to create a non-root user. To maximize isolation, we have chosen to create one user per application. However, some applications share a mounted directory, so they run as the same user. For example, we run Syncthing and Apache HTTPD as the same user because Apache serves files from a Syncthing directory.

These are the steps:

1. Create a non-root user without a shell. This prevents remote logins.

2. Enable linger. This allows the user to run processes, even when it is not logged in.

3. By default Podman does not start containers after server boot. Most documentation will tell you to use podman generate systemd to generate a systemd unit file. However, a much simpler approach is to enable the existing podman-restart systemd service.

Below are the steps to perform this process for the user named "immich":

APPUSER=immich
# 1. create user:
sudo useradd -m -s /usr/sbin/nologin $APPUSER
# 2. allow user to run services even when it is not logged in:
loginctl enable-linger $APPUSER
# 3, Enable restart after boot:
sudo -u $APPUSER mkdir -p /home/$APPUSER/.config/systemd/user/
sudo -u $APPUSER cp /lib/systemd/system/podman-restart.service /home/$APPUSER/.config/systemd/user/
systemctl --user --machine ${APPUSER}@ enable podman-restart.service

How to run podman as the non-root user

In the previous section we created the users with the nologin pseudo-shell. It is therefore not possible to login as that user. It might be tempting to use su or sudo to switch to the non-root user. However, do not do this! Neither command creates the required 'login session'. If you try anyway, it may appear to work, but you will get problems in the future. For more details see sudo rootless podman.

One way to create a shell with a login session is to use machinectl:

APPUSER=immich
machinectl -q shell ${APPUSER}@ /bin/bash

In this shell it is safe to run commands like podman ps.

How to prepare the docker-compose file

Prepare a directory in the user's home directly, and place the docker-compose.yaml file you downloaded from the service's website in it. However, there are a few things that need to be changed to work with podman-compose and rootless.

These are the changes we found:

1. Make image names fully specified. In particular you will need to prepend docker.io/ when the container registry is missing from the name. For example, image: wallabag:1.41 becomes image: docker.io/wallabag:1.41. Podman has a catalog of some short names. For example, image: ubuntu works fine.

2. Configure your application to bind to ports above 1024 (unprivileged ports). Even if the application thinks it is running as root inside the container, and there is a port mapping to a higher port, it is still not allowed to bind privileged ports.

   You may also need to adjust the application configurations within the container. See below for some tips.

   Another solution is to change the lowest privileged port on the host system. For example with: sysctl -w net.ipv4.ip_unprivileged_port_start=80. This should be safe as long as you have a firewall in place (which you do, right?!).

3. Replace restart: unless-stopped with restart: always. The systemd restart service only supports always.

4. Disable health checks. We have yet to find a more better way to reduce the enormous amount of garbage logging that podmam produces.

Start the application as the non-root user

With the above done, we are ready to start the containers with podman-compose. For example, as root, run:

cd /home/wallabag/podman-wallabag
machinectl -q shell wallabag@ podman-compose pull
machinectl -q shell wallabag@ podman-compose up -d

To make things repeatable, you should create a script. Here is the script that we use to run Wallabag:

#!/bin/bash
set -euo pipefail
IFS=$'\n\t'
cd "$(dirname "$0")"

podman-compose pull
podman-compose down
podman-compose --podman-run-args=--log-driver=none up -d
sleep 2
podman-compose exec wallabag /var/www/wallabag/bin/console doctrine:migrations:migrate --env=prod --no-interaction
podman-compose exec db psql --user=postgres -c 'ALTER DATABASE wallabag REFRESH COLLATION VERSION'
podman image prune --force

Which root can run with:

machinectl -q shell wallabag@ /home/wallabag/podman-wallabag/pull-and-start.sh

User ID mapping and running Apache HTTPD

Apache HTTPD requires more attention than most services. It insists on running as root and then stepping down to another user. While this is a good security measure, it is also annoying because the user which it steps down to in the container (on Ubuntu, this defaults to www-data with user ID 33) gets mapped to a completely unique user ID (something like 100032) on the host. This makes it more difficult to share a mounted directory.

Fortunately, podmap has an option to map some users (inside the container) to the user that started the container on the host. For example, when podman-compose is run by user mainsite, the parameter --userns=keep-id:uid=0,uid=33 maps the in-container users 0 and 33 to the mainsite user on the host. (Note that user 0 (root) is already mapped as such by default.)

Here is the list of changes we had to make while building the apache container:

• Add USER root somewhere to the end of the Dockerfile (but before ENTRYPOINT or CMD).
• The file /etc/apache2/ports.conf should contain the text Listen 8080
• Update the VirtualHost tags in all /etc/apache2/sites-enabled files so they look like this: <VirtualHost *:8080>.
• Run podman like this: podman-compose --podman-run-args=--userns=keep-id:uid=0,uid=33 up -d

Why this is great!

Although Docker is very easy to use, we are happy that we no longer have to deal with these annoyances:

Solved Docker annoyance #1 — root access

This was the whole point of this exercise! Applications no longer have root access on the host system, which makes full system takeovers after a breach less likely.

Solved Docker annoyance #2 — network misery

Docker's networking setup collides very hard with the firewalls we have used (Shorewall and UFW). You have to jump through hoops to get everything working reliably. These issues are all gone with podman. The slirp4nets network mode simply opens a port without trying to change iptables.

(Hopefully) solved Docker annoyance #3 — poor image cleanup

Even running docker image prune often does not prevent an ever growing /var/lib/docker directory. Hopefully, podman does not have this problem. At least, the images are cached per user and are therefore easier to clean up.

Podman is not perfect

Here are some issues we encountered with podman:

Podman annoyance #1 — logging

For some reason, podman and podman-compose like to log every little detail. The result is that syslog becomes so cluttered with useless data that it becomes unusable. Moreover, this increases wear on our SSDs. We have yet to find a good way to deal with this. For now, we have disabled health checks, and added

[engine]
events_logger = "none"

to each user's .config/containers/containers.conf. You may have also noticed the --podman-run-args=--log-driver=none argument in the start script above.

Perhaps the solution lies in logging directly to syslog (which supports filters) instead of via journald (which doesn't).

Podman-compose annoyance #2 — no incremental changes

Docker compose is smart, it detects and applies only the necessary changes. Podman-compose, however, is not so advanced. Our workaround is to always run a podman-compose down before running a podman-compose up -d.

Removing Docker

Once all services have been migrated, docker can be removed. This is not just a matter of running apt remove docker.io docker-buildx docker-compose-v2 (in case you have Ubuntu's stock docker installed). You have to actively search for remnants. For example with find / -iname '*docker*' 2>/dev/null. In particular you should delete /var/lib/docker (for fun, first do du -h -s /var/lib/docker to see how much disk space Docker needed).

Aside: podman-compose or docker compose on top of podman?

It is possible to run docker compose over podman. This should give you the best of both worlds: the sleek and complete support from docker compose, and the rootless safety of podman.

We did not go this route because:

• Even though podman-compose is a bit rough, it does what we need.
• Using tools from the same family feels more future-proof. Good luck when you have interoperability issues!
• For docker-compose to work, you need to set up a docker-socket. More moving parts mean less reliability.

Conclusions

• With minor changes you can migrate from docker compose to podman-compose.
• Podman is not as polished as Docker.
• Some docker annoyances disappear, but are replaced by podman annoyances.
• Using docker for internet-facing applications is irresponsible. Using rootless podman fixes that.

MavenGate gets it all wrong and hurts open source

MavenGate claims that some Maven namespaces (for example nl.grons, the namespace I control) are vulnerable to hijacking. If I understand it correctly, the idea is that hackers can place a package with the existing or newer Maven coordinates in the same, or different Maven repository, thereby luring users into using a hacked version of your package. Sounds serious, and it probably is.

However, they then went on to create a list of Maven namespaces that are vulnerable. Unfortunately, they do not say what criteria were used to put namespaces on this list. Is it because the associated DNS domain expired? Because the DNS domain moved to a different owner, or only to another DNS registrar? Is it because the PGP key used to sign packages is not on a known server? Or something else entirely? For some reason my namespace ended up on the list, even though I never lost control of the DNS domain and strictly follow all their recommendations.

Even more unfortunately, this is not even the right way to look at the problem. It is not the namespaces that are vulnerable, it is the Maven repositories themselves! It is the Maven repositories that are responsible for checking the namespace against ownership of the associated DNS domain and link that to a PGP key. Once the key is linked to the namespace, packages signed with a different PGP key should not be accepted. Any exceptions to this rule should be considered very carefully.

Now to my second point, how does this hurt open source? Since my Maven Central account was blocked after MavenGate, I contacted Sonatype, the owners of Maven Central. Luckily, I use Keybase and was therefore easily able to assert I am still owner of the DNS domain and the PGP key that has been used to sign packages. But then Sonatype also wrote this:

It is important to note that, even if we are able to verify your publisher authorization, security software may flag components published under this namespace. It may be worth considering registering a separate, new namespace with a clean-slate reputation.

I am just an individual publishing open source packages in my free time. IMHO it is totally unreasonable to ask people to switch to another domain because some random company on the internet suspects you might be vulnerable! Switching to a new DNS domain is a lot of work and in addition, not everyone is willing or able to bear the costs. I suspect that many people, including me, will give up rather than join a race against 'security software'.

To summarize:

• MavenGate declares Maven namespaces to be vulnerable based on unclear and probably wrong criteria.
• If this is taken seriously, the bar to publishing open source becomes so high that many will give up instead.

Note: I have tried to contact the MavenGate authors, but unfortunately did not receive a reply yet.

Zio-kafka hacking day

Not long ago I contacted Steven (committer of the zio-kafka library) to get some better understanding of how the library works. April 12, not more than 2 months later I am a committer, and I was sitting in a room together with Steven, Jules Ivanic (another committer) and wildcard Pierangelo Cecchetto (contributor), hacking on zio-kafka.

The meeting was an idea of Jules who was ‘in the neighborhood’. He was traveling from Australia for his company (Conduktor). We were able to get a nice room in the Amsterdam office of my employer (Adevinta). Amsterdam turned out to be a nice middle ground for Steven, me and Pierangelo. (Special thanks to Go Data Driven who also had place for us.)

In the morning we spoke about current and new ideas on how to improve the library. Also, we shared detailed knowledge on ZIO and what Kafka expects from its users. After lunch we started hacking. Having someone to start an ad hoc discussion turned out to be very productive; we were able to move some tough issues forward.

Here are some highlights.

PR #788 — Wait for stream end in rebalance listener is important to prevent duplicates during a rebalance process. This PR was mostly finished for quite some time, but many details made the extensive test suite fail. We were able to solve many of these issues.

In the area of performance we implemented an idea to replace buffering (pre-fetching a fixed number of polls), with pre-fetching based on the stream’s queue size. This resulted in PR #803 — Alternative backpressure mechanism.

We also laid the seeds for another performance improvement implementation: PR #908 — Optimistically resume partitions early.

These last two PRs showed great performance improvements bringing us much closer to direct usage of the Java Kafka client. All 3 PRs are now in review.

All in all it was a lot of fun to meet fellow enthusiasts and hack on the complex machinery that is inside zio-kafka.

Kafka is good for transport, not for system boundaries

In the last years I have learned that you should not run Kafka as a system boundary. A system boundary in this article is the place where messages are passed from one autonomy domain to another.

Now why is that? Let’s look at two classes of problems: connecting to Kafka and the long feedback loop. To prove my points, I am going to bore you with long stories from my personal experience. You may be in a different situation, YMMV!

Problem 1: Connecting to Kafka is hard

Compared to calling an HTTP endpoint, sending messages to Kafka is much much harder.

Don’t agree? Watch out for observation bias! During my holiday we often have long high-way drives through unknown countries. After looking at a highway for several hours non-stop, you might be inclined to believe that the entire country is covered by a dense highway network. In reality though, the next highway might be 200km away. A similar thing can happen at work. My part of the company offers Kafka as a service. We also run several services that invariable use Kafka in some way. We have deep knowledge and experience. It would be easy to think that Kafka is simple for everyone. However, for the rest of the company this Kafka thing is just another far away system that they have to integrate with and knowledge will be spotty and incomplete.

Let’s look at some of the problems that you have to deal with.

Partitioning is hard

It is easier to deal with partitioning problems when you control both the producer and the broker. We once had a problem where our systems could not keep up with the inflow of Kafka messages for one of the producers. The weird thing is that most of the machines were just idling. The problem grew slowly, so it took us some time before we realized it was caused by some partitions having most of the traffic. Producers of Kafka events do not always realize the effect of wrongly chosen key values. When many messages have the same key they end up in the same partition. It took some time before we got across that they needed to change the message key.

When you run an HTTP endpoint, spreading traffic and partitioning is handled by the load-balancer and is therefore under control of the receiver and not the sender.

Cross network connections are hard

Producers and the Kafka brokers need to have the same view of the network. This is because the brokers will tell a producer to which broker (by DNS name or IP address) it needs to connect to for each partition. This might go wrong when the producers and brokers use a different DNS server, or when they are on networks with colliding IP address ranges. Getting this right is a lot easier when you’re running everything in a single network you control.

This is not a problem with HTTP endpoints. Producers only need 1 hostname and optionally an HTTP proxy.

We didn’t talk about authentication and encryption yet. Kafka is very flexible; it has many knobs and settings in this area and the producers have to be configured exactly right or else it just won’t work. And don’t expect good error messages. Good documentation and cooperation is required to make this work across different teams.

With HTTP endpoints, encryption is very well-supported through https. Authentication is straight forward with HTTP’s basic authentication.

Problems that have been solved

Just for completeness here are some problems from around 2019 that have since been solved.

Around 2019 Kafka did not support authentication and TLS out of the box. Crossing untrusted networks was quite cumbersome.

Also around that time you had to be very careful about versioning. The client and server had to be upgraded in a very controlled order. Today this looks much better; you can combine almost any client and server version.

The default partitioner would give slow brokers more, instead of less work. This has been solved a few months ago.

Problem 2: Long feedback loop

When messages are being given to you via Kafka, you can not reject them. They are send and forget, the producer no longer cares. Dealing with invalid messages is now your responsibility.

In one of our projects we used to set invalid messages apart and offer Slack alerts so that the producers knew they had to look at the validation errors. Unfortunately, it didn’t work well. The feedback loop was simply too long and the number of invalid messages stayed high.

Later we introduced an HTTP endpoint in which we reject invalid messages with a 400 response. This simple change was nothing less than a miracle. For every producer that switched the vast majority of invalid messages disappeared. The number of invalid messages has remained very low since then.

Because we were able to reject invalid messages the feedback loop shortened and became much more effective.

Conclusions

Kafka within your own autonomy domain can be a great solution for message transport. However, Kafka as a boundary between autonomy domains will hurt.

Footnotes

1. Though at high enough volume, HTTP is not easy either; you’ll need proper connection pooling and an endpoint that accepts batches or else deploy a huge server park. ↩
2. Many load balancers offer sticky sessions which is a weak form of partitioning. ↩
3. We suffered both. ↩
4. When your authentication settings are wrong, the Kafka command line tools tell you that by showing an OutOfMemoryError. My head still hurts from this one. ↩
5. Though unfortunately, many architects will make this complex by using oauth or other such systems. ↩
6. Most invalid messages could be fixed with a few minutes of coding time. ↩

Don’t call your state ‘state’

In the OO world you are frowned upon if you call something object. Its time to extend this principle: don’t call your state state. This post is about why this is a bad idea and what you can do about it.

Recently we sat down to discuss the new data model for our messaging system at eBay’s Classifieds Group. One of the things we inherited from the past is the entity called conversation with a field called state. Possible values were Ok, On hold and Blocked.

So what was the problem?
A field called ‘state’ almost always has a very intuitive meaning. Unfortunately, the word is so vague that the meaning can easily warp, depending on the problem at hand. I noticed this in a couple of projects: the state field started to collect more and more possible values. With more values came increasingly difficult state transitions. This lead to code that was way more messy then necessary.

For example, in our conversation entity we could introduce the state Closed to indicate that a participant wants to stop the conversation. Then we continue by adding the state Archived to indicate that the conversation should be hidden until a new messages arrive.

What can we do?
The key observation is that each state value represents multiple behaviors. Think about it, what behavior is needed in each state? How do these behaviors change for each state? These questions will lead you to multiple fields that can represent the entire state of your entities.

Within a couple of minutes we found three behaviors we wanted to have for our conversations: a conversation is either visible or not (field visibility with values Displayed and Hidden), it will accept new messages or not (field acceptNew with values Accept and Reject) and we want to notify the recipient of a new message (or not) (field notifyOnNew with values Notify and Mute).Not only did our code become easier to extend and reason about, as a bonus we found a feature that would have been really hard with the old model: muting a conversation.

Conclusion
Don’t call your state ‘state’, instead, think about the behavior each state represents and model that instead.

Breaking the Circuit Breaker

The circuit breaker is this wonderful pattern to protect your application against resources that fail slowly. The idea is that you stop trying to use a resource when it has too many failures. Regular retries test the resource and will make the resource available again. The benefit is that your application can react quickly to a failed resource instead of hogging CPU, threads, network, etc. while you are waiting to find out the resource is unavailable.

So what's wrong?

Its the metaphor. In the classical description a circuit breaker has 3 states: the open state, the closed state and the half-open state. So what does it mean when the circuit breaker is open? When is a bridge open? When you can drive over it, or when you can sail through it? Only when you look at the first image you may see that a traditional open circuit breaker stops flow of electricity. To us that translates to no usage of the resource. In the 'closed' state electricity flows, which translates to having access to our resource. Now read that again and see if you can remember that!

Then we have a half-open state? Again, look at the first image. For such a switch half-open is still open. (A half-open bridge lets no traffic trough at all but that is another topic.) Why do we need the half-open state anyway? In the classical description we attempt to use the resource once while in this state. If it fails just once, we go back to the open state. This seems like a good idea, but let us think of modern networked applications. In such applications many requests are done simultaneously. So as soon as we switch to the half-open state for a retry, many, maybe hundreds of request will immediately try to use the resource, even if it is still down. This is exactly what we were trying to prevent!

Stop!

Although the circuit breaker is a great invention, I think we need a new metaphor, or at least some new terminology.

No more half-open

The first thing we can do is get rid of the half-open state. Instead, when its time to retry, we just let 1 client through to the resource. While that check is in progress we keep denying access to the resource for other clients; we stay in the same state. Only when the single check succeeds, we switch to the state in which we allow full access to the resource.

No more open

The second thing we need to do is to end the confusion on what it means to be 'open'. Instead I propose we call this state the broken state. No further explanation required. Good. In this state we do the regular retries.

Finally, to make things symmetric, I propose to rename the 'closed' state to flow state as all requests are granted.

Metaphor

Above I proposed new terminology but I failed to provide a new metaphor. Unfortunately metaphors are hard to find and too easy to get wrong. Perhaps a good metaphor should be related to the fact that we are limiting the number of errors we tolerate from a resource. If you have an idea, please let me know in a comment. I hope you liked my little rant. Any comments are always welcome.

—   ❧   —

Postscript: Sentries and the circuit breaker

The Sentries library contains a highly optimized circuit breaker implementation for Scala programs. The ideas in this article developed while writing Sentries. Feel free to have a look. As you can see there are only 2 states, the FlowState and the BrokenState. Note that the retryAt in BrokenState is a val; it can not be changed after initialization. When it is time to retry we replace the broken state with a new instance (in method attemptResetBrokenState).

Apache Wicket Cookbook — book review

Some time ago I reviewed the drafts of the new book from Wicket rockstar programmer Igor Vaynberg: Apache Wicket Cookbook. If you are serious about using Wicket, this book is for you. It is fast, to the point, has very clear code samples and teaches you all the relevant (both clean and dirty) stuff you need and which Wicket in Action could not cover.

Conclusion: this is the book to read after 'Wicket in Action'.

Open source - why bother with anything else?

Since I work in a fine small company where we are breathing open source for at least a decade, it is sometimes weird to be confronted again by open source adversaries or agonists. For example, a colleague wrote a fine technical design based on Mule. All of a sudden we're asked to compare this to BEA AquaLogic and see whether we could implement the project with that. Now this is probably possible, and AquaLogic is probably a fine product family, but why bother?

Since there was already a Mule prototype, I found it a cumbersome idea. To get up to speed with AquaLogic you first have to find out what products in the AquaLogic family you need, and of course you only need a tiny bit of most of them. Secondly you have to go into a trajectory to get the software, including development licenses. Somehow money is not always a problem, but the time to get the products and start working always is, and the deadline won't move. Did I already mention I really hate bureaucracy?

Starting with open source often just takes 5 lines in a pom.xml and a few minutes of download time.

Okay, well suppose this was all taken care of and you are happily underway with development. You then run into a problem. Yes, you will, this is no different from open source. Now lets see how I typically deal with problems with using open source and see how this applies to closed source software.

Finding a solution to a problem with an open source product
1) (Re-)read the documentation
The first step is always to read the documentation. With the source jars attached in your favorite IDE, the javadoc is one key-press away.

2) Debugging
In step 2 we'll do some debugging. Again, with the source jars attached, tracing through your own code is as easy as tracing the open source code. I may not understand all the code, but I know what I need and I can read the JavaDoc of code encountered underway.
More often then not tracing leads to a deeper understanding of the used products, and I frequently find things that are useful for other parts of the project. The deeper understanding helps to form a solution. This can be changing your code, or patching the open source product. Otherwise it helps you formulate a more precise problem statement for the next steps.

3) Read more documentation
As I have now seen the code, I can search the available documentation more efficiently. So I do this first before going into the next step. Documentation in this phase can be anything, from manuals to blogs and forums.

4) Post questions
The last step is to post a question on a forum or mailing list. If you get this far, you are either lazy, you just missed something, or you found out that the library has a bug. The results of this step are not always satisfactory; it really depends on the community around the product. At least you can always change the open source product yourself.

Finding a solution to a problem with a closed source product

1) (Re-)read the documentation
Again, the first step is to read the documentation. However, javadoc is not always available. Rarely is it available in the form of a source jar.

2) Debugging
Oops, we can only trace our own code. Perhaps you can use de-compilation. Note that this might actually be illegal in your country (not in The Netherlands luckily). Secondly, the source might be obfuscated. And of course, I am not even talking about products that run as a complete separate program.

3) Read more documentation
As it is unlikely that debugging gave us more insight, we skip this step.

4) Post questions
In rare cases there are user communities around closed source. These are very valuable! However, usually you just have to ask the manufacturer. By lack of insight, the question won't be as detailed as with open source. And then we rely on the manufacturer. Some react quick and accurate, some don't even give you the the ability to ask questions. More often then not it costs lots of money and time. And of course meanwhile you will have to code workarounds yourself.

Conclusions

Despite the title I am not at all against closed and commercial software. Its just that as a programmer I find it hardly ever worth the bother.

Don't use Intellij

IntelliJ is okay, but it has one bug (for more then 4 years) that absolutely drives me nuts: its steals focus, big time. And not just once either. It can steal the focus for at least 4 times within 30 seconds!

If are thinking about using IntelliJ: don't do it!

If you are already hooked up: please vote for this bug!

Do not offer money!

Money and open source, always an interesting combination. But how do we, simple developers (with no ambition but to make beautiful stuff and live from it), deal with this? Let me offer you my opinion.

Lets start with a real example on what you should not do. Here is a quote from an e-mail that was sent on the user list of well respected project xyz (which it is, is not important):

Hi!

I was wondering how I could make a contribution for xyz. I'm not talking about a code contribution but rather a small money contribution. I have got a lot of help here on this forum and in fact I don't think I have ever experienced this kind of help elsewhere!

Thanks to all that have helped and especially core xyz coders.

The author obviously has good intentions, so how can this be so disastrously wrong? Suppose you are celebrating Christmas (or any other traditional family event) at your mother-in-law´s house. She cooked really well and everything tasted just wonderfully. You stand up and say, "Mother, this was excellent. This must have been worth at least 400 euros. Here, this is for you." and you hand here 4 fresh 100 euros bills. You will probably understand that your mother-in-law will not be happy, and in fact she will likely not forget this for a long time. Will she have the same pleasure in cooking for you next year? I think not.

So what is the problem? As Dan Ariely, a professor of behavioral economics wrote in his excellent book Predictably Irrational (I read the Dutch translation "Waarom we altijd tijd te kort komen") there are 2 sets of norms, market norms and social norms. Market norms regulate how we interact with each other when money is involved. Social norms regulate all other interactions. So eating at a restaurant will fall under the market norms, eating at your mother-in-low falls under the social norms. It is both about eating, but if you think about it, there are vast differences in what you expect, how you deal with bad food, etc.

The coders of xyz do all of their hard work in their free time. They are not getting paid, they do this purely out of love for what they are creating, and perhaps because of the idea that it is useful to others, and perhaps because of the respect they will receive. Any social interaction therefore falls under the social norms. Now if you start to offer money, you will move the social interactions to market norms. Luckily the core coders of project xyz are wise and responded with a request to donate to a charity project. But suppose they accepted the money. Now why would they do their best next time when no money is involved? They might as well go drinking beer with friends. Or worse, why would non core contributors do their best if they do not get paid? If you know somebody else is getting paid for the same task, even if it only takes 1 minute, why would you spend any brain power on it?

What is even worse, after switching to market norms, it takes a lot of time before the social norms are reinstated. Here is quote from Eric Daugherty, author of the Java Email Server, on JES version 2.0:

I started down this path with passion and drive. After I'd completed the SMTP (I think) portion of the code base, I was contacted by Andrew Oliver about my interest in working on a JBoss mail server. He was working on a new project to build an enterprise class mail (and calender, etc) server build on JBoss (where he was employed at the time). This project, JBossMailServer at the time, seemed to eclipse what I was attempting, and given my already slowing momentum on the 2.0 branch, pretty much brought it to a screeching halt. I had a bit of interest in working on the JBoss version, but in truth some of the motivation for the projects that I work on is ego. Working on a project that wasn't 'mine' didn't quite have the same appeal, even if it was JBoss (remember, this was 4 years ago).

Apparently JBoss had found someone interested in the server, as they approached me with an offer to pay me for the task of completing the SMTP (or POP, I don't recall) functionality. Since I was a pretty easy task for me (I'd done it once already), and hey, they were paying me, I jumped on board. I worked through the deliverable they wanted and earned a little spending money.

However, a funny thing happened. Once I'd worked for pay on the project, it was really hard to get excited about working for free. Combine this with the lack of real ego payoff, and I drifted away from the project.

So when the previous state is restored (the money is gone), the market norms stay for quite a while. This is proven by Dan and colleagues and this is also Eric´s own observation. A troubling conclusion: by offering money for free services the result is less services.

Suppose you care for the success of an open source project, what can you do? My answer is simple: contribute by writing documentation, by answering questions, by writing patches, provide hosting and otherwise interact according to the social norms. If you are not able to do any of these things, it is better to stay out!

Acknowledgments
Thanks to Dan Ariely for his research. The Christmas dining example is from his book.