Wednesday, April 15, 2020

Traefik v2 enable HSTS, Docker and nextcloud

This took me days to figure out how to configure Traefik v2. Here it is for posterity.

This is a docker-compose.yaml fragment to append to a service section:

labels: - "traefik.enable=true" - "traefik.http.routers.service.rule=Host(``)" - "traefik.http.routers.service.entrypoints=websecure" - "traefik.http.routers.service.tls.certresolver=myresolver" - "traefik.http.middlewares.servicests.headers.stsincludesubdomains=false" - "traefik.http.middlewares.servicests.headers.stspreload=true" - "traefik.http.middlewares.servicests.headers.stsseconds=31536000" - "traefik.http.middlewares.servicests.headers.isdevelopment=false" - "traefik.http.routers.service.middlewares=servicests"

It will:

  • tell Traefik to direct traffic for to this container,
  • on the websecure entrypoint (this is configured statically),
  • using the myresolver (for Acme, resolver also configured statically),
  • configure middleware to add HSTS headers,
  • enable the middleware.


Here is a slightly more complex example for a nextcloud deployment which includes the recommended redirects.

labels: - "traefik.enable=true" - "traefik.http.routers.nextcloud.rule=Host(``)" - "traefik.http.routers.nextcloud.entrypoints=websecure" - "traefik.http.routers.nextcloud.tls.certresolver=myresolver" - "traefik.http.middlewares.nextcloudredir.redirectregex.permanent=true" - "traefik.http.middlewares.nextcloudredir.redirectregex.regex=https://(.*)/.well-known/(card|cal)dav" - "traefik.http.middlewares.nextcloudredir.redirectregex.replacement=https://$$1/remote.php/dav/" - "traefik.http.middlewares.nextcloudsts.headers.stsincludesubdomains=false" - "traefik.http.middlewares.nextcloudsts.headers.stspreload=true" - "traefik.http.middlewares.nextcloudsts.headers.stsseconds=31536000" - "traefik.http.middlewares.nextcloudsts.headers.isdevelopment=false" - "traefik.http.routers.nextcloud.middlewares=nextcloudredir,nextcloudsts"

Friday, April 10, 2020

Akka-http graceful shutdown


By default, when you restart a service, the old instance is simply killed. This means that all current requests are aborted; the caller will be left with a read timeout. We can do better!


A graceful shutdown looks as follows:

  1. The scheduler (Kubernetes, Nomad, etc.) sends a signal (usually SIGINT) to the service.
  2. The service gets the signal and closes all server-ports; it can no longer receive new request. This is very quickly picked up by the load-balancer. The load-balancer will no longer send new requests.
  3. All requests-in-progress complete one by one.
  4. When all requests are completed, or on a timeout, the service terminates.

Getting the signal to your service is unfortunately not always trivial. I have seen the following problems:

  • The Nomad scheduler by default does not send an SIGINT signal to the service. You will have to configure this.
  • When the service runs in a Docker container, by default the init process (with PID 1) will ignore the signal. Back when every Unix installation had control over the entire computer this made lots of sense. In a container though, not so much. This may be fixed in newer Docker version. Otherwise you will have to use a special init process such as tini.

Akka-http has excellent support for graceful shutdown. Unfortunately, the documentation is not very clear about it. Here follows an example which can be used as a template:

import akka.http.scaladsl.Http import akka.http.scaladsl.server._ import scala.concurrent.duration._ val logger = ??? val route: Route = ??? val interface: String = "" val port: Int = 80 val shutdownDeadline: FiniteDuration = 30.seconds Http() .bindAndHandle(route, interface, port) .map { binding => "HTTP service listening on: " + s"http://${binding.localAddress.getHostName}:${binding.localAddress.getPort}/" ) sys.addShutdownHook { binding .terminate(hardDeadline = shutdownDeadline) .onComplete { _ => system.terminate()"Termination completed") }"Received termination signal") } } .onComplete { case Failure(ex) => logger.error("server binding error:", ex) system.terminate() sys.exit(1) case _ => }