Wednesday, June 8, 2022

Zigzag bytes

I was playing around with a goofy idea for which I needed zigzag encoding for bytes. Zigzag encoding is often used in combination with variable length encoding in things like Avro, Thrift and Protobuf.

In zigzag encoded integers, the least significant bit is used for sign. To convert from regular encoding (2-complement) to zigzag (and back) you can use the following Scala code:

def i32ToZigZag(n: Int): Int = (n << 1) ^ (n >> 31) def zigZagToI32(n: Int): Int = (n >>> 1) ^ - (n & 1) def i64ToZigZag(n: Long): Long = (n << 1) ^ (n >> 63) def zigZagToI64(n: Long): Long = (n >>> 1) ^ - (n & 1)

Translate this to Java and the expressions after the = look exactly the same.

Using these bit shifting tricks for bytes is a whole lot more difficult. The problem is that Scala (like Java) does not support bit operations on Bytes. They always convert them to an Int first.

After a lot of fiddling, I settled on the following:

private def b(i: Int): Byte = (i & 0xff).toByte def i8ToZigZag(n: Byte): Byte = (b(n << 1) ^ (n >> 7)).toByte def zigZagToI8(n: Byte): Byte = b(((n & 0xff) >>> 1) ^ (256 - (n & 1)))

Translated to Java it should look like this (not tested!):

private byte b(int i) { return (byte)(i & 0xff); } public byte i8ToZigZag(byte n) { return (byte)(b(n << 1) ^ (n >> 7)); } public byte zigZagToI8(byte n) { return b(((n & 0xff) >>> 1) ^ (256 - (n & 1))); }

Is there a better way to do this?

Saturday, March 12, 2022

Upgrading Libreoffice with Homebrew

Reminder to self: this is the procedure to upgrade Libreoffice with Homebrew:

  1. brew update
  2. brew upgrade
  3. open -a /Applications/LibreOffice.app
  4. Quit the application
  5. brew reinstall libreoffice-language-pack, enter your password
  6. open "/usr/local/Caskroom/libreoffice-language-pack/$(cd /usr/local/Caskroom/libreoffice-language-pack; ls -1 | sort -rV | head)/LibreOffice Language Pack.app", click 'Ok', click 'Ok'

Please anyone, please make this simpler...

Wednesday, January 26, 2022

Having fun with Ordering in Scala

Challenge: sort a list of objects by name, but some names have priority. If these names appear, they should be ordered by the position they have in the priority list.

For example:

val priorityList = Seq("Willow", "James", "Ezra") val input = Seq("Olivier", "Charlotte", "Willow", "Declan", "Aurora", "Ezra") val ordered = ??? assert(ordered == Seq("Willow", "Ezra", "Aurora", "Charlotte", "Declan", "Olivier"))

Challenge accepted.

Luckily Scala has strong support for sorting in the standard library. All sequences have a sorted method which accepts an Ordering. The ordering is an implicit parameter which means that normally we don't need to provide it; it will be derived to the natural ordering of the items. However, we are going to provide this parameter explicitly. Let's build an Ordering!

The challenge explains we have 2 orderings:

  1. first order by the priority list
  2. failing that, order by alphabet

Let's focus on the first ordering. The idea is to assign an integer 'priority-value' to each possible string that is based on the position in the priority list. If the string is not in the list, we use some high integer. The first ordering will simply order by this priority-value. Lower numbers go before higher number, just like the natural ordering of integers.

// attempt 1 val priorityValue = priorityList.indexOf(name)

This works well for any name on the priority list. E.g. Willow gets 0 and Ezra gets 2. Unfortunately, all the other names get priority value -1 which orders them even before Willow. We need to convert the -1 to something higher.
Since I like to program without if statements whenever possible, I looked at math for a solution. Modulus can do the trick:

// attempt 2 val priorityValue = priorityList.indexOf(name) % priorityList.size

Oops, wrong modulus implementation: -1 % 3 == -1. Let's use floorMod:

// attempt 3 val priorityValue = Math.floorMod(priorityList.indexOf(name), priorityList.size)

Now -1 gets converted into priorityList.size which is definitely higher than the other priority-values. However, since we don't really care what the higher number is we can just use Int.MaxValue:

val priorityValue = Math.floorMod(priorityList.indexOf(name), Int.MaxValue)

Now we wrap that in an Ordering:

val priorityOrdering: Ordering[String] = Ordering.by(name => Math.floorMod(priorityList.indexOf(name), Int.MaxValue))

Unfortunately Scala can't derive the type. We either need to type the value directly, or add a type on the name parameter.

Now we need to add the second ordering. We can simply use Ordering.String from the library. We combine the orderings with orElse, available since Scala 2.13. The second ordering is used when priorityOrdering can't decide because the priority-value is the same.
Note that for the special case where we compare a priority value with itself, e.g. Willow with Willow, the second ordering is also applied. This is okay though, the outcome doesn't change because these values are the same for the alphabetic ordering also.

Here is the complete code:

val priorityList = Seq("Willow", "James", "Ezra") val priorityOrdering: Ordering[String] = Ordering.by(name => Math.floorMod(priorityList.indexOf(name), Int.MaxValue)) val combinedOrdering: Ordering[String] = priorityOrdering.orElse(Ordering.String) val input = Seq("Olivier", "Charlotte", "Willow", "Declan", "Aurora", "Ezra") val ordered = input.sorted(combinedOrdering) assert(ordered == Seq("Willow", "Ezra", "Aurora", "Charlotte", "Declan", "Olivier"))

We're almost there. The challenge was to work on any object. Lets wrap it up a bit and also make it work for any type of priority value:

def priorityOrdering[A, B : Ordering](priorityList: Seq[B], by: A => B): Ordering[A] = { def priorityValue(b: B): Int = Math.floorMod(priorityList.indexOf(b), Int.MaxValue) Ordering.by[A, Int](a => priorityValue(by(a))).orElseBy(by) }

We can use like this:

val ordered = input.sorted(priorityOrdering[String, String](priorityList, identity))

Or like this. Here we sort persons by birthdate, ordering today before the other days:

case class Person(name: String, birthdate: MonthDay) val persons: Seq[Person] = ??? val priorityDates = Seq(MonthDay.now()) persons.sorted(priorityOrdering(priorityDates, (_: Person).birthdate))

Some remarks

Note that in all these cases type derivation is quite awful. The compiler has problems finding the correct types, even though all the information is available.

You should also know that you can't use this approach if you are stuck on Scala 2.12 or earlier since Ordering.orElse is not available there.

Alternative

You can side-step all the type derivation problems by using the sortBy method. Give it a function that returns a tuple of Ints, Strings, or anything for which an Ordering is already defined. The sequence is then sorted on the first value of the tuple, then on the second value, etc.:

def priorityValue(name: String): Int = Math.floorMod(priorityList.indexOf(name), Int.MaxValue) val ordered = input.sortBy(name => (priorityValue(name), name))

Conclusion

Although I had fun learning all about Ordering, next time I'll avoid it and go directly for sortBy.

Monday, January 10, 2022

From Adoptopenjdk to Temurin on a Mac using Homebrew

Adoptopenjdk joined the Eclipse foundation and renamed their JDK to Temurin. Here are instructions on how to migrate on Macs with Homebrew.

The following instructions removes Adoptopenjdk JDKs you may still have:

brew remove adoptopenjdk/openjdk/adoptopenjdk8 brew remove adoptopenjdk/openjdk/adoptopenjdk11 brew untap AdoptOpenJDK/openjdk brew remove adoptopenjdk8 brew remove adoptopenjdk11 brew remove adoptopenjdk

Use /usr/libexec/java_home -V to get an overview of any other JDK you may still have. Just delete what you don't need any more.

Then install Temurin 8, 11 and 17. The first command (brew tap …) is only needed in case you need Temurin 8 or 11:

brew tap homebrew/cask-versions brew install --cask temurin8 brew install --cask temurin11 brew install --cask temurin

Bonus: execute the following to define aliases that let you easily switch between Java versions:

cat <<-EOF >> ~/.zshrc # Aliases for switching java version alias java17="export JAVA_HOME=\$(/usr/libexec/java_home -v 17)" alias java11="export JAVA_HOME=\$(/usr/libexec/java_home -v 11)" alias java8="export JAVA_HOME=\$(/usr/libexec/java_home -v 1.8)" java11 EOF

Are you looking for more power? For example you need to test for many more JDKs? Then maybe Sdkman is something for you.

Sunday, December 19, 2021

Customizing the Jitsi Meet UI in a Docker deployment

I manage a Jitsi instance for a small for-benefit organization. I wanted to make some changes to the UI to make it visually belong to the organization. Unfortunately, Jitsi doesn't make it easy to do this. Upon every upgrade your changes are gone. This post describes a workaround for Jitsi deployments that use Docker.

Although the details can be hairy, the idea is quite simple. We are going to put another layer over the provided Docker image called 'web'. The additional layer contains all the changes we need. When Jitsi publishes an update, we just apply the changes again as part of the deployment process.

Our starting point is the docker-compose.yaml provided by Jitsi. Make all the changes as instructed. However, before we make any changes to the UI, you should make sure your Jitsi instance is working.

Its working? Congratulations! Start with a small change to your docker-compose.yaml.
Replace:

web: image: jitsi/web:latest

with:

web: build: ./jitsi-web

This sets you up for building your own Docker image.

Create the jitsi-web directory, and put all the artwork you want to override in it. You should end up with a directory structure like this (more details follow):

Directory structure

The Dockerfile in the jitsi-web initially has just one line like this:

FROM jitsi/web

Build the image and deploy it with:

docker-compose build --pull docker-compose up -d

Make sure that Jitsi is still working.

Now it is your turn to get creative. With some RUN instructions you can change any file in the base image.

To get you started, I'll show what is in my Dockerfile. Details are discussed directly below:

FROM jitsi/web # Add wasm mime type # https://community.jitsi.org/t/loading-wasm-webassembly-file-on-jitsimeetjs/68071/3 RUN sed -i '/}/i \ application/wasm wasm;' /etc/nginx/mime.types # Replace/add some images COPY --chown=root:root overrides /usr/share/jitsi-meet/ RUN sed -i "s|\(// \)\?defaultLanguage:.*|defaultLanguage: 'nl',|" /defaults/config.js; \ sed -e 's/welcome-background.png/welcome-background.jpg/' \ -e 's|.deep-linking-mobile .header{width:100%;height:70px;background-color:#f1f2f5;text-align:center}|.deep-linking-mobile .header{width:100%;height:70px;background-color:#003867;text-align:center}|' \ -e 's|.deep-linking-mobile .header .logo{margin-top:15px;margin-left:auto;margin-right:auto;height:40px}|.deep-linking-mobile .header .logo{margin-top:10px;margin-left:auto;margin-right:auto;height:50px}|' \ -i /usr/share/jitsi-meet/css/all.css; \ sed -e 's|"headerTitle": "Jitsi Meet"|"headerTitle": "Mijn Organisatie"|' \ -e 's|"headerSubtitle": "Veilige vergaderingen van hoge kwaliteit"|"headerSubtitle": "Wij vergaderen online!"|' \ -i /usr/share/jitsi-meet/lang/main-nl.json; \ sed -e 's|C().createElement("h1",{className:"header-text-title"},t("welcomepage.headerTitle"))|C().createElement("h1",{className:"header-text-title"},C().createElement("img",{src:"images/logo-deep-linking.png",alt:"Mijn Organisatie",height:100}))|' \ -e 's|"headerTitle":"Jitsi Meet"|"headerTitle":"Mijn Organisatie"|' \ -e 's|"headerSubtitle":"Secure and high quality meetings"|"headerSubtitle":"Wij vergaderen online!"|' \ -i /usr/share/jitsi-meet/libs/app.bundle.min.js; \ sed -e "s|\bAPP_NAME: .*|APP_NAME: 'Mijn Organisatie Jitsi',|" \ -e "s|\bPROVIDER_NAME: .*|PROVIDER_NAME: 'Mijn Organisatie Cloud',|" \ -e "s|\bDEFAULT_REMOTE_DISPLAY_NAME: .*|DEFAULT_REMOTE_DISPLAY_NAME: 'Gespreksgenoot',|" \ -e "s|\bDEFAULT_LOCAL_DISPLAY_NAME: .*|DEFAULT_LOCAL_DISPLAY_NAME: 'Ik',|" \ -e "s|\bGENERATE_ROOMNAMES_ON_WELCOME_PAGE: .*|GENERATE_ROOMNAMES_ON_WELCOME_PAGE: false,|" \ -i /defaults/interface_config.js

The first RUN instruction adds a line to the Nginx configuration which enables clients to download Wasm files. Unfortunately this is not yet fixed in Jitsi docker itself (checked at December 2021, but YMMV).

The COPY instruction copies your images over the existing stuff. Feel free to add more as needed.

The second RUN instruction is where the magic happens. This changes existing files. Let's go through them one by one.

The first file that gets changes in /defaults/config.js where we set the default language to Dutch.

The next file that gets changes is /usr/share/jitsi-meet/css/all.css. Normally Jitsi uses a png background image on the welcome page but I needed to use a jpg image. The first line takes care of that. Note that there is no welcome-background.jpg image in the base image, but I added it in the overrides/images directory.
The next 2 changes for this file are some small color and layout changes to the welcome page for mobile browsers.

The next file that gets changes is /usr/share/jitsi-meet/lang/main-nl.json. There are many more files in this directory, one for each language.

The next one, file /usr/share/jitsi-meet/libs/app.bundle.min.js is tricky. This file contains a fully compiled React application in minified Javascript. The first change you see here replaces the header text with a header image. The next two lines replace the the default titles with the Dutch version. For some reason Jitsi initially renders the page in English and then re-renders it in the correct locale. On slow devices this can take quite some time. I found this quite disturbing, especially for the texts that make your first impression. By changing some default texts, most of my users (which are Dutch) will see less flapping texts.
This is the file that is most sensitive to changes in the base image. Make sure your tweaks still work after an upgrade.

Finally, in /defaults/interface_config.js some more settings are tweaked.

Some more tips

Don't worry if you break something. Just fix your changes and re-deploy. A re-deploy is very quick.

Finding out what to change can be pretty hard. Sometimes it helps to extract the file from the image to see what it contains. First find the file you want to change by opening a shell in the base image:

docker-compose exec web bash

Extract the file for more detailed inspection with something like this:

docker-compose exec web cat /usr/share/jitsi-meet/libs/app.bundle.min.js > app.bundle.min.js

Jitsi updates

When you see new images appear at Jitsi on docker hub you can deploy them as follows:

# Pulls the images that we're not changing (e.g. prosody, jicofo and jvb): docker-compose pull # Rebuild the 'web' image, checking for a new base image: docker-compose build --pull # Deploy changes: docker-compose up -d # Remove old images: docker image prune

Most of the things that were tweaked here were pretty stable over the last years. But I advice you check anyway.

That's it, go creative!

Wednesday, December 8, 2021

Akka graceful shutdown - continued

Some time ago I wrote on how to gracefully shutdown Akka HTTP servers, crucial to prevent aborted requests during re-deployments or in elastic (cloud) environments where instances come and go. Look at the previous post for more details on how graceful shutdown works and some common caveats in setting it up.

This post refreshes how this works for newer Akka versions, and it gives some tips on how to speed up a shutdown.

Coordinated shutdown

Newer Akka versions have more extensive support for graceful shutdown in the form of coordinated shutdown. Here we show an example that uses coordinated shutdown to configure a graceful shutdown.

import akka.actor.{ActorSystem, CoordinatedShutdown} import akka.http.scaladsl.Http import scala.concurrent.duration._ import scala.util.{Failure, Success} implicit val system: ActorSystem = ??? val logger = ??? val routes: Route = ??? val interface: String = "0.0.0.0" val port: Int = 80 val shutdownDeadline: FiniteDuration = 5.seconds Http() .newServerAt(interface, port) .bind(routes) .map(_.addToCoordinatedShutdown(httpShutdownTimeout)) // ← that simple! .foreach { server => logger.info(s"HTTP service listening on: ${server.localAddress}") server.whenTerminationSignalIssued.onComplete { _ => logger.info("Shutdown of HTTP service initiated") } server.whenTerminated.onComplete { case Success(_) => logger.info("Shutdown of HTTP endpoint completed") case Failure(_) => logger.error("Shutdown of HTTP endpoint failed") } }

The important line is where we use addToCoordinatedShutdown. What follows is just logging so we know what's going on.

Shutting down more components

You probably have more parts that would benefit from a proper shutdown, e.g. a database connection pool. Here is an example on how to hook into the coordinated shutdown:

// Add this code _before_ construction of the HTTP server CoordinatedShutdown(system).addTask( CoordinatedShutdown.PhaseBeforeClusterShutdown, "database connection pool shutdown" ) { () => val dbPoolShutdown: Future[Done] = shutdownDatabase() dbPoolShutdown.onComplete { case Success(_) => logger.info("Shutdown of database connection pool completed") case Failure(_) => logger.error("Shutdown of database connection pool failed") } logger.info("Shutdown of database connection pool was initiated") dbPoolShutdown }

Coordinated shutdown consists of multiple phases. Which phases exist is configurable but the default phases are fine for this post.

Our code runs in the phase called before-cluster-shutdown. This phase runs after phase service-unbind in which the HTTP service shuts down.

Tips to speed up shutdown

The default phases need to complete in 10 seconds. If this is challenging for your system, here are 2 tips that might help.

First of all, you need to make sure that any blocking/synchronous code is wrapped in a blocking construct. This will signal to the Akka execution context that it needs to extend its threadpool. This is especially relevant if you have many shutdown tasks but it is good practice anyway. For example:

import akka.Done import scala.concurrent.blocking def shutdownDatabase: Future[Done] = { Future { blocking { database.close() // blocking code logger.info("Database connection closed") } Done } }

The second thing you could do is only shutdown on a best-effort basis. Closing the connection to a read-only system is probably not essential.
The trick is to use coordinated shutdown as initiator, but then immediately report completion. For example:

import akka.Done import scala.concurrent.blocking def bestEffortShutdownDatabase: Future[Done] = { // Starts db close in a Future, but ignore the result Future { blocking { database.close() } } Future.successful(Done) }

Now, when closing the databse takes too long, Akka won't complain about this in the logs.

For more details see Akka's coordinated shutdown documentation.

Tuesday, July 7, 2020

Avoiding Scala's Option.fold

Scala's Option.fold is a bit nasty to use sometimes because type inference is not always great. Here is a stupid toy example:

val someValue: Option[String] = ??? val processed = someValue.fold[Either[Error, UserName]]( Left(Error("No user name given")) )( value => Right(UserName(value)) )

The [Either[Error, UserName]] on fold is necessary otherwise the scala compiler can not derive the type.

Here is a really small trick to avoid Option.fold when you need to convert an Option to an Either:

val someValue: Option[String] = ??? val processed = someValue .toRight(left = Error("No user name given")) .map(value => Right(UserName(value)))

Much nicer!