Wednesday, December 8, 2021

Akka graceful shutdown - continued

Some time ago I wrote on how to gracefully shutdown Akka HTTP servers, crucial to prevent aborted requests during re-deployments or in elastic (cloud) environments where instances come and go. Look at the previous post for more details on how graceful shutdown works and some common caveats in setting it up.

This post refreshes how this works for newer Akka versions, and it gives some tips on how to speed up a shutdown.

Coordinated shutdown

Newer Akka versions have more extensive support for graceful shutdown in the form of coordinated shutdown. Here we show an example that uses coordinated shutdown to configure a graceful shutdown.

import akka.actor.{ActorSystem, CoordinatedShutdown} import akka.http.scaladsl.Http import scala.concurrent.duration._ import scala.util.{Failure, Success} implicit val system: ActorSystem = ??? val logger = ??? val routes: Route = ??? val interface: String = "0.0.0.0" val port: Int = 80 val shutdownDeadline: FiniteDuration = 5.seconds Http() .newServerAt(interface, port) .bind(routes) .map(_.addToCoordinatedShutdown(httpShutdownTimeout)) // ← that simple! .foreach { server => logger.info(s"HTTP service listening on: ${server.localAddress}") server.whenTerminationSignalIssued.onComplete { _ => logger.info("Shutdown of HTTP service initiated") } server.whenTerminated.onComplete { case Success(_) => logger.info("Shutdown of HTTP endpoint completed") case Failure(_) => logger.error("Shutdown of HTTP endpoint failed") } }

The important line is where we use addToCoordinatedShutdown. What follows is just logging so we know what's going on.

Shutting down more components

You probably have more parts that would benefit from a proper shutdown, e.g. a database connection pool. Here is an example on how to hook into the coordinated shutdown:

// Add this code _before_ construction of the HTTP server CoordinatedShutdown(system).addTask( CoordinatedShutdown.PhaseBeforeClusterShutdown, "database connection pool shutdown" ) { () => val dbPoolShutdown: Future[Done] = shutdownDatabase() dbPoolShutdown.onComplete { case Success(_) => logger.info("Shutdown of database connection pool completed") case Failure(_) => logger.error("Shutdown of database connection pool failed") } logger.info("Shutdown of database connection pool was initiated") dbPoolShutdown }

Coordinated shutdown consists of multiple phases. Which phases exist is configurable but the default phases are fine for this post.

Our code runs in the phase called before-cluster-shutdown. This phase runs after phase service-unbind in which the HTTP service shuts down.

Tips to speed up shutdown

The default phases need to complete in 10 seconds. If this is challenging for your system, here are 2 tips that might help.

First of all, you need to make sure that any blocking/synchronous code is wrapped in a blocking construct. This will signal to the Akka execution context that it needs to extend its threadpool. This is especially relevant if you have many shutdown tasks but it is good practice anyway. For example:

import akka.Done import scala.concurrent.blocking def shutdownDatabase: Future[Done] = { Future { blocking { database.close() // blocking code logger.info("Database connection closed") } Done } }

The second thing you could do is only shutdown on a best-effort basis. Closing the connection to a read-only system is probably not essential.
The trick is to use coordinated shutdown as initiator, but then immediately report completion. For example:

import akka.Done import scala.concurrent.blocking def bestEffortShutdownDatabase: Future[Done] = { // Starts db close in a Future, but ignore the result Future { blocking { database.close() } } Future.successful(Done) }

Now, when closing the databse takes too long, Akka won't complain about this in the logs.

For more details see Akka's coordinated shutdown documentation.

No comments:

Post a Comment