Wednesday, April 15, 2020

Traefik v2 enable HSTS, Docker and nextcloud

This took me days to figure out how to configure Traefik v2. Here it is for posterity.

This is a docker-compose.yaml fragment to append to a service section:

labels: - "traefik.enable=true" - "traefik.http.routers.service.rule=Host(`www.example.com`)" - "traefik.http.routers.service.entrypoints=websecure" - "traefik.http.routers.service.tls.certresolver=myresolver" - "traefik.http.middlewares.servicests.headers.stsincludesubdomains=false" - "traefik.http.middlewares.servicests.headers.stspreload=true" - "traefik.http.middlewares.servicests.headers.stsseconds=31536000" - "traefik.http.middlewares.servicests.headers.isdevelopment=false" - "traefik.http.routers.service.middlewares=servicests"

It will:

  • tell Traefik to direct traffic for www.example.com to this container,
  • on the websecure entrypoint (this is configured statically),
  • using the myresolver (for Acme, resolver also configured statically),
  • configure middleware to add HSTS headers,
  • enable the middleware.

Nextcloud

Here is a slightly more complex example for a nextcloud deployment which includes the recommended redirects.

labels: - "traefik.enable=true" - "traefik.http.routers.nextcloud.rule=Host(`nextcloud.example.com`)" - "traefik.http.routers.nextcloud.entrypoints=websecure" - "traefik.http.routers.nextcloud.tls.certresolver=myresolver" - "traefik.http.middlewares.nextcloudredir.redirectregex.permanent=true" - "traefik.http.middlewares.nextcloudredir.redirectregex.regex=https://(.*)/.well-known/(card|cal)dav" - "traefik.http.middlewares.nextcloudredir.redirectregex.replacement=https://$$1/remote.php/dav/" - "traefik.http.middlewares.nextcloudsts.headers.stsincludesubdomains=false" - "traefik.http.middlewares.nextcloudsts.headers.stspreload=true" - "traefik.http.middlewares.nextcloudsts.headers.stsseconds=31536000" - "traefik.http.middlewares.nextcloudsts.headers.isdevelopment=false" - "traefik.http.routers.nextcloud.middlewares=nextcloudredir,nextcloudsts"

Friday, April 10, 2020

Akka-http graceful shutdown

Why?

By default, when you restart a service, the old instance is simply killed. This means that all current requests are aborted; the caller will be left with a read timeout. We can do better!

What?

A graceful shutdown looks as follows:

  1. The scheduler (Kubernetes, Nomad, etc.) sends a signal (usually SIGINT) to the service.
  2. The service gets the signal and closes all server-ports; it can no longer receive new request. This is very quickly picked up by the load-balancer. The load-balancer will no longer send new requests.
  3. All requests-in-progress complete one by one.
  4. When all requests are completed, or on a timeout, the service terminates.
Caveats

Getting the signal to your service is unfortunately not always trivial. I have seen the following problems:

  • The Nomad scheduler by default does not send an SIGINT signal to the service. You will have to configure this.
  • When the service runs in a Docker container, by default the init process (with PID 1) will ignore the signal. Back when every Unix installation had control over the entire computer this made lots of sense. In a container though, not so much. This may be fixed in newer Docker version. Otherwise you will have to use a special init process such as tini.
Akka-HTTP

Akka-http has excellent support for graceful shutdown. Unfortunately, the documentation is not very clear about it. Here follows an example which can be used as a template:

import akka.http.scaladsl.Http import akka.http.scaladsl.server._ import scala.concurrent.duration._ val logger = ??? val route: Route = ??? val interface: String = "0.0.0.0" val port: Int = 80 val shutdownDeadline: FiniteDuration = 30.seconds Http() .bindAndHandle(route, interface, port) .map { binding => logger.info( "HTTP service listening on: " + s"http://${binding.localAddress.getHostName}:${binding.localAddress.getPort}/" ) sys.addShutdownHook { binding .terminate(hardDeadline = shutdownDeadline) .onComplete { _ => system.terminate() logger.info("Termination completed") } logger.info("Received termination signal") } } .onComplete { case Failure(ex) => logger.error("server binding error:", ex) system.terminate() sys.exit(1) case _ => }

Tuesday, March 3, 2020

Push Gauges

A colleague was complaining to me that Micrometer gauges didn't work the way he expected. This led to some interesting work.

What is a gauge?

In science a gauge is a device for making measurements. In computer systems a gauge is very similar: a 'metric' which tracks something in your system over time. For example, you could track the number of items in a job queue. Libraries like Micrometer and Dropwizard metrics make it easy to define gauges. Since the measurement in itself is not useful, those libraries also make it easy to send the measurements to a metric system such as Graphite or Prometheus. These systems are used for visualization and generating alerts.

Gauges are typically defined with a callback function that does the measurement. For example, using metrics-scala, the scala API for Dropwizard metrics, it looks like:

class JobQueue extends DefaultInstrumented { private val waitingJobsQueue = ??? // Defines a gauge metrics.gauge("queue.size") { // This code block is the callback which does a 'measurement'. waitingJobsQueue.size } }

Please note that the metric library determines when the callback function is invoked. For example, once every minute.

What is a push gauge?

My colleague had something else in mind. He didn't have access to the value all the time, but only when something was being processed. More like this:

class ExternalCacheUpdater extends DefaultInstrumented { def updateExternalCache(): Unit = { val items = fetchItemsFromDatabase() pushItemsToExternalCache(items) gauge.push(items.size) // Pushes a new measurement to the gauge. } }

In the example the application becomes responsible for pushing new measurements. The push gauge simply keeps track of the last value and reports that whenever the metrics library needs it. So under the covers the push gauge behaves like a normal gauge.

Push gauges like in this example are now made possible in this pull-request for metrics-scala. The only thing that was missing is the definition of the push gauge:

class ExternalCacheUpdater extends DefaultInstrumented { // Defines a push gauge private val gauge = metrics.pushGauge[Int]("cached.items", 0) def updateExternalCache(): Unit = // as above }
Push gauge with timeout

In some situations it may be misleading to report a very old measurement as the 'current' value. If the external cache in our example evicts items after 10 minutes, then the push gauge should not report measurements from more then 10 minutes ago. This is solved with a push gauge with timeout:

class ExternalCache extends DefaultInstrumented { // Defines a push gauge with timeout private val gauge = metrics.pushGaugeWithTimeout[Int]("cached.items", 0, 10.minutes) def updateExternalCache(): Unit = // as above }
Feedback wanted!

I have not seen this concept before in any metric library in the JVM ecosystem. Therefore I would like to collect as much feedback as possible before shipping this as a new feature of metrics-scala. If you have any ideas, comments or whatever, please leave a comment on the push-gauges pull-request or drop me an email!

Update 2020-03-05: The code example have been updated to reflect changes in the pull request.

Wednesday, July 12, 2017

Continuation parsers and encoders

I finally got around to writing about my hack project of last year. It was an exploration of what can be done with continuation parsers and encoders in order to implement a very fast single-copy asynchronous Thrift implementation.

Continuation parsers and encoders try to decode (read)/encode (write) their data directly from/to a network buffer. When the buffer has been fully read/written, it asks for more network buffers to continue.

For more information see the thrift-stream repository on GitHub.

Tuesday, April 18, 2017

Bash history backup

I like my bash history, and I proudly have this in my .bash_profile:

# Increase bash history size, append instead of overwrite history on exit export HISTSIZE=10000000 export HISTCONTROL=erasedups shopt -s histappend

However, after reading about Historian I realised I have no backup. Instead of installing Historian, I decided to take a simpler approach. Here it is:

#!/bin/bash set -euo pipefail IFS=$'\n\t' # # Makes a daily backup of your .bash_history, keeps the last 2 backups. # BACKUPDIR="${HOME}/.bash_history_backup" mkdir -p "${BACKUPDIR}" cp ${HOME}/.bash_history "${BACKUPDIR}/bash_history_$(date +%Y%m%d)" for h in $(ls -1r "${BACKUPDIR}"/bash_history_* | tail -n +3); do rm $h; done

Put the above contents in a file somewhere, e.g. in ~/bin/bash_history_backup.sh and activate it with:

ln -s ~/bin/bash_history_backup.sh /etc/cron.daily/
or add the following line with the more cumbersome route through crontab -e:
0 0 * * * $HOME/bin/bash_history_backup.sh

Update 2017-04-22: The script actually works now :)

Update 2017-04-23: The script actually actually works now :)

Tuesday, September 13, 2016

Exploring Rkt on Ubuntu

I have been using docker in my home server since 0.4 in 2013. For me the most attractive property of docker is that it provides a way to decrease the amount of stuff one has to install on a server. I have only one server, but it has many different tasks of which some need to be rock solid (my family email) and other are experimental. Containers provide a nice way to clean up experiments. Unfortunately, docker has never been stable. I have had many fights with docker during upgrades and I have never fully understood how docker interacts with the iptables setup from my firewall (Shorewall).

Today, I was trying to run docker-syncthing. Unfortunately, the container went into 'restarting' without any indication what was wrong. Inspired by "Can docker be ousted" and boring and stable containers I decide to give up and try something new: CoreOS' rkt.

Installation

The installation instructions are a bit hidden. This page refers to the script install-rkt.sh. Unfortunately, no link was given. A search on github finally gave the answer. However, in the end, I liked these instructions better: ask ubuntu: Is it possible to install rkt in Ubuntu?.

Because I wanted to run Syncthing and I only had a Docker recipe, I also needed the tool docker2aci. The instructions there are clear except that you need to install golang first (docs are fixed now):

$ sudo apt-get install golang $ git clone git://github.com/appc/docker2aci $ cd docker2aci $ ./build.sh

The docker2aci binary is located in the bin folder. Put it somewhere so you can find it back.

Building and converting a docker image

After building the docker image:

# docker build -t syncthing .

I had the following:

# docker images REPOSITORY TAG IMAGE ID CREATED SIZE syncthing latest 8ea0931f1196 29 hours ago 197.1 MB

Next step is to fetch the image into the rkt image repository. The rkt fetch command can fetch an image directly from a docker repository. However, I found no way to fetch directly from the local docker image repository. This is the work around:

# docker save -o syncthing-docker-image.tar syncthing # docker2aci syncthing-docker-image.tar # rkt --insecure-options=image fetch syncthing-latest.aci # rkt image list ID NAME SIZE IMPORT TIME LAST USED sha512-8161ad07a42e syncthing:latest 168MiB 7 hours ago 7 hours ago # rkt image cat-manifest syncthing:latest | less

Running the image

Now comes the time to run the image:

# rkt --insecure-options=image run --net=host \ --dns=$(awk '/nameserver/ {print $2}' < /etc/resolv.conf) \ --volume=volume-srv-config,kind=host,source=/media/nas/syncthing/config,readOnly=false \ --volume=volume-srv-data,kind=host,source=/media/nas/syncthing/data,readOnly=false \ syncthing

Creating that statement took actually longer then I expected. Here are the highlights:

  • Option --dns=... copies the nameserver from you local /etc/resolv.conf to the same file inside the container. This makes DNS work inside the container also. Depending on your DNS setup, you may need to pass more --dns* options.
  • The --net=host gives you the easiest access to the network. If you want a bit more security, you will have to dive deeper.
  • I also tried to add the options --user=nobody --group=nogroup. However, that resulted in my container not starting up at all with weird error messages.

You now have something like this:

# rkt list UUID APP IMAGE NAME STATE CREATED STARTED NETWORKS 17baa16c syncthing syncthing:latest running 1 minute ago 1 minute ago

Inspecting the container

Inspecting a running container is easy. With rkt enter you can directly open a shell in the container:

# rkt enter 17baa16c enter: no command specified, assuming "/bin/bash" root@rkt-5e9ad759-82b4-4b27-b03a-b6b5074b2ac2:/#

Cleanup

Every time you start a new container, the old one stays around. With rkt gc you cleanup containers that stopped some time ago (more then half an hour?). This command should be run from cron:

# echo -e '#!/bin/sh\nexec rkt gc' > /etc/cron.daily/rkt-gc # chmod +x /etc/cron.daily/rkt-gc

Networking

As a Shorewall user you need to add a rule to open the ports for the syncthing application in /etc/shorewall/rules and that's it. This might get a lot more hairy when you are not using host networking.

Automating startup

Modern Ubuntu's use systemd to start applications. Rkt's systemd manual seems well written but can be used as an introduction as best. Their suggestion to use systemd-run failed me:

# systemd-run --slice=machine /usr/bin/rkt --insecure-options=image run --net=host --dns=192.168.1.1 --volume=volume-srv-config,kind=host,source=/media/nas/syncthing/config,readOnly=false --volume=volume-srv-data,kind=host,source=/media/nas/syncthing/data,readOnly=false syncthing Failed to start transient service unit: Cannot set property ExecStart, or unknown property.

Also, it fails to mention where to put the systemd unit file. After lots of reading (in particular the systemd section of this manual) I created /etc/systemd/system/rkt-syncthing.service with the following content:

[Unit] Description=Rkt syncthing Requires=remote-fs.target After=remote-fs.target [Service] Slice=machine.slice ExecStart=/usr/bin/rkt --insecure-options=image run --net=host --dns=192.168.1.1 --volume=volume-srv-config,kind=host,source=/media/nas/syncthing/config,readOnly=false --volume=volume-srv-data,kind=host,source=/media/nas/syncthing/data,readOnly=false syncthing KillMode=mixed Restart=always [Install] WantedBy=multi-user.target

And finally:

# systemctl daemon-reload # systemctl enable syncthing.service # systemctl start syncthing.service

Ending thoughts

Although I got far in a few hours there are still a few open problems.

  1. None of the above software was available as a Ubuntu package. This results in me spending more time to keep everything up to date.
  2. Rkt's documentation is okay but not amazing. For example, hyperlinks are missing in crucial places (see above), and in the manual page for rkt-export does not tell you how to indicate which container you want to export.
  3. Rkt containers run from an image and store changes as a layer on top. As soon the container exists, you can not start it again with those changes. This means that everything that needs to be persisted must be external to the container. For other changes you can run rkt export to create a new image, or you rebuild your image from scratch.
  4. Having to use Docker to build an image for rkt is a bit weird. Next step is to create the image directly with acbuild.

Updates

2017-02-05 Fixed commands for enabling rkt garbage collection via cron.

Thursday, March 3, 2016

Don’t call your state ‘state’

In the OO world you are frowned upon if you call something object. Its time to extend this principle: don’t call your state state. This post is about why this is a bad idea and what you can do about it.

Recently we sat down to discuss the new data model for our messaging system at eBay’s Classifieds Group. One of the things we inherited from the past is the entity called conversation with a field called state. Possible values were Ok, On hold and Blocked.

So what was the problem?
A field called ‘state’ almost always has a very intuitive meaning. Unfortunately, the word is so vague that the meaning can easily warp, depending on the problem at hand. I noticed this in a couple of projects: the state field started to collect more and more possible values. With more values came increasingly difficult state transitions. This lead to code that was way more messy then necessary.

For example, in our conversation entity we could introduce the state Closed to indicate that a participant wants to stop the conversation. Then we continue by adding the state Archived to indicate that the conversation should be hidden until a new messages arrive.

What can we do?
The key observation is that each state value represents multiple behaviors. Think about it, what behavior is needed in each state? How do these behaviors change for each state? These questions will lead you to multiple fields that can represent the entire state of your entities.

Within a couple of minutes we found three behaviors we wanted to have for our conversations: a conversation is either visible or not (field visibility with values Displayed and Hidden), it will accept new messages or not (field acceptNew with values Accept and Reject) and we want to notify the recipient of a new message (or not) (field notifyOnNew with values Notify and Mute).Not only did our code become easier to extend and reason about, as a bonus we found a feature that would have been really hard with the old model: muting a conversation.

Conclusion
Don’t call your state ‘state’, instead, think about the behavior each state represents and model that instead.