System hardening: Migrating from docker compose to podman-compose

TL;DR: Podman-Compose provides essential security features for homelabbers, albeit with a few inconveniences.

Many applications that appeal to homelabbers can be installed with a docker-compose file. Docker-compose files are awesome! They provide a nice uniform way to install wildly different applications.

The most used runtime is Docker. Unfortunately, docker runs as root and container-escape vulnerabilities are abundant. Therefore, a hacked application immediately leads to a fully compromised system. Luckily, there are viable alternatives. For example, podman explicitly allows running containers without root.

This article summarizes what we learned when we migrated our seven homelab services from docker compose to podman-compose on a Ubuntu 24.04 server.

Preparations

Install Podman and Podman-compose

This is by far the simplest section of this article. As root, run:

apt install podman podman-compose

How to prepare non-root user(s)

Since we want to run our services without root access (rootless), we need to create a non-root user. To maximize isolation, we have chosen to create one user per application. However, some applications share a mounted directory, so they run as the same user. For example, we run Syncthing and Apache HTTPD as the same user because Apache serves files from a Syncthing directory.

These are the steps:

1. Create a non-root user without a shell. This prevents remote logins.

2. Enable linger. This allows the user to run processes, even when it is not logged in.

3. By default Podman does not start containers after server boot. Most documentation will tell you to use podman generate systemd to generate a systemd unit file. However, a much simpler approach is to enable the existing podman-restart systemd service.

Below are the steps to perform this process for the user named "immich":

APPUSER=immich
# 1. create user:
sudo useradd -m -s /usr/sbin/nologin $APPUSER
# 2. allow user to run services even when it is not logged in:
loginctl enable-linger $APPUSER
# 3, Enable restart after boot:
sudo -u $APPUSER mkdir -p /home/$APPUSER/.config/systemd/user/
sudo -u $APPUSER cp /lib/systemd/system/podman-restart.service /home/$APPUSER/.config/systemd/user/
systemctl --user --machine ${APPUSER}@ enable podman-restart.service

How to run podman as the non-root user

In the previous section we created the users with the nologin pseudo-shell. It is therefore not possible to login as that user. It might be tempting to use su or sudo to switch to the non-root user. However, do not do this! Neither command creates the required 'login session'. If you try anyway, it may appear to work, but you will get problems in the future. For more details see sudo rootless podman.

One way to create a shell with a login session is to use machinectl:

APPUSER=immich
machinectl -q shell ${APPUSER}@ /bin/bash

In this shell it is safe to run commands like podman ps.

How to prepare the docker-compose file

Prepare a directory in the user's home directly, and place the docker-compose.yaml file you downloaded from the service's website in it. However, there are a few things that need to be changed to work with podman-compose and rootless.

These are the changes we found:

1. Make image names fully specified. In particular you will need to prepend docker.io/ when the container registry is missing from the name. For example, image: wallabag:1.41 becomes image: docker.io/wallabag:1.41. Podman has a catalog of some short names. For example, image: ubuntu works fine.

2. Configure your application to bind to ports above 1024 (unprivileged ports). Even if the application thinks it is running as root inside the container, and there is a port mapping to a higher port, it is still not allowed to bind privileged ports.

   You may also need to adjust the application configurations within the container. See below for some tips.

   Another solution is to change the lowest privileged port on the host system. For example with: sysctl -w net.ipv4.ip_unprivileged_port_start=80. This should be safe as long as you have a firewall in place (which you do, right?!).

3. Replace restart: unless-stopped with restart: always. The systemd restart service only supports always.

4. Disable health checks. We have yet to find a more better way to reduce the enormous amount of garbage logging that podmam produces.

Start the application as the non-root user

With the above done, we are ready to start the containers with podman-compose. For example, as root, run:

cd /home/wallabag/podman-wallabag
machinectl -q shell wallabag@ podman-compose pull
machinectl -q shell wallabag@ podman-compose up -d

To make things repeatable, you should create a script. Here is the script that we use to run Wallabag:

#!/bin/bash
set -euo pipefail
IFS=$'\n\t'
cd "$(dirname "$0")"

podman-compose pull
podman-compose down
podman-compose --podman-run-args=--log-driver=none up -d
sleep 2
podman-compose exec wallabag /var/www/wallabag/bin/console doctrine:migrations:migrate --env=prod --no-interaction
podman-compose exec db psql --user=postgres -c 'ALTER DATABASE wallabag REFRESH COLLATION VERSION'
podman image prune --force

Which root can run with:

machinectl -q shell wallabag@ /home/wallabag/podman-wallabag/pull-and-start.sh

User ID mapping and running Apache HTTPD

Apache HTTPD requires more attention than most services. It insists on running as root and then stepping down to another user. While this is a good security measure, it is also annoying because the user which it steps down to in the container (on Ubuntu, this defaults to www-data with user ID 33) gets mapped to a completely unique user ID (something like 100032) on the host. This makes it more difficult to share a mounted directory.

Fortunately, podmap has an option to map some users (inside the container) to the user that started the container on the host. For example, when podman-compose is run by user mainsite, the parameter --userns=keep-id:uid=0,uid=33 maps the in-container users 0 and 33 to the mainsite user on the host. (Note that user 0 (root) is already mapped as such by default.)

Here is the list of changes we had to make while building the apache container:

• Add USER root somewhere to the end of the Dockerfile (but before ENTRYPOINT or CMD).
• The file /etc/apache2/ports.conf should contain the text Listen 8080
• Update the VirtualHost tags in all /etc/apache2/sites-enabled files so they look like this: <VirtualHost *:8080>.
• Run podman like this: podman-compose --podman-run-args=--userns=keep-id:uid=0,uid=33 up -d

Why this is great!

Although Docker is very easy to use, we are happy that we no longer have to deal with these annoyances:

Solved Docker annoyance #1 — root access

This was the whole point of this exercise! Applications no longer have root access on the host system, which makes full system takeovers after a breach less likely.

Solved Docker annoyance #2 — network misery

Docker's networking setup collides very hard with the firewalls we have used (Shorewall and UFW). You have to jump through hoops to get everything working reliably. These issues are all gone with podman. The slirp4nets network mode simply opens a port without trying to change iptables.

(Hopefully) solved Docker annoyance #3 — poor image cleanup

Even running docker image prune often does not prevent an ever growing /var/lib/docker directory. Hopefully, podman does not have this problem. At least, the images are cached per user and are therefore easier to clean up.

Podman is not perfect

Here are some issues we encountered with podman:

Podman annoyance #1 — logging

For some reason, podman and podman-compose like to log every little detail. The result is that syslog becomes so cluttered with useless data that it becomes unusable. Moreover, this increases wear on our SSDs. We have yet to find a good way to deal with this. For now, we have disabled health checks, and added

[engine]
events_logger = "none"

to each user's .config/containers/containers.conf. You may have also noticed the --podman-run-args=--log-driver=none argument in the start script above.

Perhaps the solution lies in logging directly to syslog (which supports filters) instead of via journald (which doesn't).

Podman-compose annoyance #2 — no incremental changes

Docker compose is smart, it detects and applies only the necessary changes. Podman-compose, however, is not so advanced. Our workaround is to always run a podman-compose down before running a podman-compose up -d.

Removing Docker

Once all services have been migrated, docker can be removed. This is not just a matter of running apt remove docker.io docker-buildx docker-compose-v2 (in case you have Ubuntu's stock docker installed). You have to actively search for remnants. For example with find / -iname '*docker*' 2>/dev/null. In particular you should delete /var/lib/docker (for fun, first do du -h -s /var/lib/docker to see how much disk space Docker needed).

Aside: podman-compose or docker compose on top of podman?

It is possible to run docker compose over podman. This should give you the best of both worlds: the sleek and complete support from docker compose, and the rootless safety of podman.

We did not go this route because:

• Even though podman-compose is a bit rough, it does what we need.
• Using tools from the same family feels more future-proof. Good luck when you have interoperability issues!
• For docker-compose to work, you need to set up a docker-socket. More moving parts mean less reliability.

Conclusions

• With minor changes you can migrate from docker compose to podman-compose.
• Podman is not as polished as Docker.
• Some docker annoyances disappear, but are replaced by podman annoyances.
• Using docker for internet-facing applications is irresponsible. Using rootless podman fixes that.