TL;DR: Podman-Compose provides essential security features for homelabbers, albeit with a few inconveniences.
Many applications that appeal to homelabbers can be installed with a docker-compose file. Docker-compose files are awesome! They provide a nice uniform way to install wildly different applications.
The most used runtime is Docker. Unfortunately, docker runs as root and container-escape vulnerabilities are abundant. Therefore, a hacked application immediately leads to a fully compromised system. Luckily, there are viable alternatives. For example, podman explicitly allows running containers without root.
This article summarizes what we learned when we migrated our seven homelab services from docker compose
to podman-compose on a Ubuntu 24.04 server.
Preparations
Install Podman and Podman-compose
This is by far the simplest section of this article. As root, run:
apt install podman podman-compose
How to prepare non-root user(s)
Since we want to run our services without root access (rootless), we need to create a non-root user. To maximize isolation, we have chosen to create one user per application. However, some applications share a mounted directory, so they run as the same user. For example, we run Syncthing and Apache HTTPD as the same user because Apache serves files from a Syncthing directory.
These are the steps:
Create a non-root user without a shell. This prevents remote logins.
Enable linger. This allows the user to run processes, even when it is not logged in.
By default Podman does not start containers after server boot. Most documentation will tell you to use
podman generate systemdto generate a systemd unit file. However, a much simpler approach is to enable the existing podman-restart systemd service.
Below are the steps to perform this process for the user named "immich":
APPUSER=immich
# 1. create user:
sudo useradd -m -s /usr/sbin/nologin $APPUSER
# 2. allow user to run services even when it is not logged in:
loginctl enable-linger $APPUSER
# 3, Enable restart after boot:
sudo -u $APPUSER mkdir -p /home/$APPUSER/.config/systemd/user/
sudo -u $APPUSER cp /lib/systemd/system/podman-restart.service /home/$APPUSER/.config/systemd/user/
systemctl --user --machine ${APPUSER}@ enable podman-restart.service
How to run podman as the non-root user
In the previous section we created the users with the nologin pseudo-shell. It is therefore not possible
to login as that user. It might be tempting to use su or sudo to switch to the non-root
user. However, do not do this! Neither command creates the required 'login session'. If you try anyway, it may
appear to work, but you will get problems in the future. For more details see sudo rootless podman.
One way to create a shell with a login session is to use machinectl:
APPUSER=immich
machinectl -q shell ${APPUSER}@ /bin/bash
In this shell it is safe to run commands like podman ps.
How to prepare the docker-compose file
Prepare a directory in the user's home directly, and place the docker-compose.yaml file you downloaded
from the service's website in it. However, there are a few things that need to be changed to work with
podman-compose and rootless.
These are the changes we found:
Make image names fully specified. In particular you will need to prepend
docker.io/when the container registry is missing from the name. For example,image: wallabag:1.41becomesimage: docker.io/wallabag:1.41. Podman has a catalog of some short names. For example,image: ubuntuworks fine.Configure your application to bind to ports above 1024 (unprivileged ports). Even if the application thinks it is running as root inside the container, and there is a port mapping to a higher port, it is still not allowed to bind privileged ports.
You may also need to adjust the application configurations within the container. See below for some tips.
Another solution is to change the lowest privileged port on the host system. For example with:
sysctl -w net.ipv4.ip_unprivileged_port_start=80. This should be safe as long as you have a firewall in place (which you do, right?!).Replace
restart: unless-stoppedwithrestart: always. The systemd restart service only supportsalways.Disable health checks. We have yet to find a more better way to reduce the enormous amount of garbage logging that podmam produces.
Start the application as the non-root user
With the above done, we are ready to start the containers with podman-compose. For example, as root, run:
cd /home/wallabag/podman-wallabag
machinectl -q shell wallabag@ podman-compose pull
machinectl -q shell wallabag@ podman-compose up -d
To make things repeatable, you should create a script. Here is the script that we use to run Wallabag:
#!/bin/bash
set -euo pipefail
IFS=$'\n\t'
cd "$(dirname "$0")"
podman-compose pull
podman-compose down
podman-compose --podman-run-args=--log-driver=none up -d
sleep 2
podman-compose exec wallabag /var/www/wallabag/bin/console doctrine:migrations:migrate --env=prod --no-interaction
podman-compose exec db psql --user=postgres -c 'ALTER DATABASE wallabag REFRESH COLLATION VERSION'
podman image prune --force
Which root can run with:
machinectl -q shell wallabag@ /home/wallabag/podman-wallabag/pull-and-start.sh
User ID mapping and running Apache HTTPD
Apache HTTPD requires more attention than most services. It insists on running as root and then stepping down to
another user. While this is a good security measure, it is also annoying because the user which it steps down to in
the container (on Ubuntu, this defaults to www-data with user ID 33) gets mapped to a completely unique
user ID (something like 100032) on the host. This makes it more difficult to share a mounted directory.
Fortunately, podmap has an option to map some users (inside the container) to the user that started the container on
the host. For example, when podman-compose is run by user mainsite, the parameter --userns=keep-id:uid=0,uid=33
maps the in-container users 0 and 33 to the mainsite user on the host. (Note
that user 0 (root) is already mapped as such by default.)
Here is the list of changes we had to make while building the apache container:
- Add
USER rootsomewhere to the end of the Dockerfile (but before ENTRYPOINT or CMD). - The file
/etc/apache2/ports.confshould contain the textListen 8080 - Update the
VirtualHosttags in all/etc/apache2/sites-enabledfiles so they look like this:<VirtualHost *:8080>. - Run podman like this:
podman-compose --podman-run-args=--userns=keep-id:uid=0,uid=33 up -d
Why this is great!
Although Docker is very easy to use, we are happy that we no longer have to deal with these annoyances:
Solved Docker annoyance #1 — root access
This was the whole point of this exercise! Applications no longer have root access on the host system, which makes full system takeovers after a breach less likely.
Solved Docker annoyance #2 — network misery
Docker's networking setup collides very hard with the firewalls we have used (Shorewall and UFW). You have to jump through hoops to get everything working reliably. These issues are all gone with podman. The slirp4nets network mode simply opens a port without trying to change iptables.
(Hopefully) solved Docker annoyance #3 — poor image cleanup
Even running docker image prune often does not prevent an ever growing /var/lib/docker
directory. Hopefully, podman does not have this problem. At least, the images are cached per user and are therefore
easier to clean up.
Podman is not perfect
Here are some issues we encountered with podman:
Podman annoyance #1 — logging
For some reason, podman and podman-compose like to log every little detail. The result is that syslog becomes so cluttered with useless data that it becomes unusable. Moreover, this increases wear on our SSDs. We have yet to find a good way to deal with this. For now, we have disabled health checks, and added
[engine]
events_logger = "none"
to each user's .config/containers/containers.conf. You may have also noticed the --podman-run-args=--log-driver=none
argument in the start script above.
Perhaps the solution lies in logging directly to syslog (which supports filters) instead of via journald (which doesn't).
Podman-compose annoyance #2 — no incremental changes
Docker compose is smart, it detects and applies only the necessary changes. Podman-compose, however, is not so
advanced. Our workaround is to always run a podman-compose down before running a podman-compose
up -d.
Removing Docker
Once all services have been migrated, docker can be removed. This is not just a matter of running apt remove
docker.io docker-buildx docker-compose-v2 (in case you have Ubuntu's stock docker installed). You have to
actively search for remnants. For example with find / -iname '*docker*' 2>/dev/null. In particular
you should delete /var/lib/docker (for fun, first do du -h -s /var/lib/docker to see how
much disk space Docker needed).
Aside: podman-compose or docker compose on top of podman?
It is possible to run docker compose over podman. This should give you the best of both worlds: the sleek and complete support from docker compose, and the rootless safety of podman.
We did not go this route because:
- Even though podman-compose is a bit rough, it does what we need.
- Using tools from the same family feels more future-proof. Good luck when you have interoperability issues!
- For docker-compose to work, you need to set up a docker-socket. More moving parts mean less reliability.
Conclusions
- With minor changes you can migrate from docker compose to podman-compose.
- Podman is not as polished as Docker.
- Some docker annoyances disappear, but are replaced by podman annoyances.
- Using docker for internet-facing applications is irresponsible. Using rootless podman fixes that.
July 2007, I was so fed up with Maven's inability to exclude commons-logging that
