The incomplete guide for sending email notification from the Ubiquiti's Unifi Cloud Gateway (UCG)

Unfortunately, sending notification emails from a Unifi Cloud Gateway (UCG) with remote management disabled, is not all straight forward. Here are some tips, though it will end with a disappointment.

Check list:

1. Send a test email
2. Weird stuff for local email servers
3. Configure an email address for the admin user
4. Configure an alert
5. Wait for Ubiquiti to fix it

1. Send a test email

In 'Settings', 'General', setting 'Email Services', select 'Custom Server'. Fill in the details and press 'Send test email'. If it works, skip to section 3, otherwise, read on.

As 'SMTP Server' you need to fill in the fully qualified hostname of the email server. The hostname must match the DNS name in the TLS certificate of the email server. If there is a mismatch, UCG will reject the connection. Therefore, an IP address does not work!

The standardized SMTP submission port is 587, with SSL disabled. No worries, due to STARTTLS the traffic is still encrypted.

If port 587 does not work, your email server may support the legacy port 465, with SSL enabled.

If that also does not work, you may try port 25 with SSL disabled (again, STARTTLS should encrypt the traffic).

2. Weird things for local email servers

Something odd happens when the DNS name of the email server actually resolves to the UCG, and you have port forwarding for port 25 and 587 to the local device that contains the email server.

The problem is that 'hairpinning' does not fully work on the UCG. Ubiquiti describes hairpinning as follows (source):

When a device on the local network attempts to connect to the public IP address of the UniFi gateway, the traffic is redirected internally, ensuring that port forwarding rules apply as they would for external requests.

Hairpinning is super useful, but Ubiquiti's interpretation is not good enough! To reach the email server, the UCG itself should also be able to use hairpinning. Unfortunately, this is not supported.

Luckily, I learned a workaround from Ubiquiti's support staff. We can give the local email server a 'local DNS record', a kind of a DNS override. We set the local DNS record equal to the fully qualified hostname (e.g, mail.example.com) of the email server. After this change, any DNS client in the local network, including the UCG itself, resolves mail.example.com to the IP address of the local device and not to the public IP address of the UCG.

Here is how to set this up: in 'Client Devices', click the device that runs the email server. In the left panel click the cog-icon (settings). Check 'Local DNS Record' and enter the fully qualified hostname of the email server, and click 'Apply Changes'.

We check can that it works by using something like dig from any machine in the local network:

# Before % dig +short mail.example.com 1.1.1.1 # some public IP address # After % dig +short mail.example.com 192.168.1.24 # a local IP address

Try another test email (see section 1) before you continue.

3. Configure an email address for the admin user

Click 'Admin and users' (bottom left icon), click the relevant user. In the left panel click the cog-icon (settings). Enter the email address, and click 'Apply Changes'.

4. Configure an alert

Click 'Alarm Manager' (second icon from bottom left). Select all alarms you want to receive an email for. Then in the left panel make sure 'Email' is selected and click 'Save'. If you create new alarms, you may have to repeat the process.

5. Wait for Ubiquiti to fix it

If you have gotten this far (like I have), it was all for nothing. According to this discussion, you won't get email notifications, unless you enabled Remote Management, if only for 1 second.

I have reached out to Ubiquiti support and I will update this article when more information arrives.

Update 2025-09-28: The mentioned discussion thread shows a screenshot in which Ubiquiti states that 'it should work better' in UnifiOS 4.4.x. No release date is know at this moment.

Update 2025-12-03: Meanwhile our UCG upgraded to UnifiOS 4.4.9. Unfortunately, still no e-mail are being sent.

Reorganizing our server shelf

TL;DR: With some planning and tinkering, you can fit a lot of hardware in a small space.

In Dutch homes, the meter cupboard (called a "meterkast" in Dutch) is a small closet, usually placed directly behind the front door. It houses the electric meter, gas meter, water meter and the circuit breakers. In our case it is also the entry point for the internet with an (ADSL) telephone line, and more recently two glass fibers (yes we can choose 🤷).

The top shelf of our meter cupboard is the perfect spot for our family's little 'data center'. The only downside is that it is small; it measures 18cm/7" high, 75cm/30" wide and 30cm/12" deep. The challenge is housing all our electronics there: our home server, an entry-model Synology NAS, the internet modem, an ethernet switch and all the cables and power adapters, including those for our wifi access points.

Since we switched from ADSL to fiber, it was a good moment to reorganize. Our internet provider (Freedom, highly recommended) supports bring-your-own internet modems. The default provided AVM Fritz!Box is quite large. Since our switch and wifi acces points are from Ubiquiti anyway, I bought the Unifi Cloud Gateway Ultra. It is very small and at €94, it is cheaper than the Fritz!Box 5590 which costs €180 through Freedom or €225 through a retailer. The Fritz!Box does come with smart home and DECT phone support. We don't need a smart home, but we do want DECT. To fill the gap, we found a secondhand Grandstream DECT/VoIP server with two handsets for just €50 on Marktplaats (our old handsets needed to be replaced anyway).

Although the internet modem, ethernet switch, and DECT/VoIP server are small, they still need to be stacked to save space. For this purpose I designed a small rack. Snijmeesters, a cutting shop in my city, laser-cut it from 3mm birch wood.

Here you see the parts.

Here you see the rack being glued together. To keep the sides straight and make it easy to remove any spilled glue, I used two glass containers.

And here is the result. It is quite sturdy, sturdier than I had imagined.

Using your own internet modem had one unexpected consequence. To convert from fiber to ethernet, you need an ONT. Freedom provides a Huawei EG8242H ONT for this purpose. It turns out that this ONT is a very large box! Making room for it on the shelf would have been difficult. However, since the provided fiber was not long enough anyway, it now hangs lower in the meter cupboard. I tried to find a smaller alternative, but it is hard to find a compatible product at a decent price. In the end we left it like this.

Here is our updated server shelf. The new rack is on the left. We have had it place for a couple of weeks now without any problems.

Using the TransIP API from bash

TLDR: Signing API requests from Bash is tricky, but doable with a temporary file.

Every couple of months I rotate the DKIM keys of my email server, after which I publish them on my website. This article on publishing dkim keys gives a good overview of why this is a good idea.

Initially this was all done manually, but over time I automated more and more. The toughest part was finding a good DNS registrar (DKIM keys are published in DNS), that has a proper API, and then using that API from Bash. The DNS registrar I am using is TransIP.

Here is how I did it.

Before we can use any other endpoint of the API, we need to get a token. We get the token by sending an API request with your username. The request must be signed with the private key that you uploaded/obtained from the API part of the TransIP console.

Using Bash variables to hold the request makes signing very tricky, before you know it a newline is added or removed, invalidating the signature; the transmitted request must be byte-for-byte the same as what was signed. Instead, we side step all Bash idiosycrasies by storing the request in a temporary file. Here we go:

# Configure your TransIP username and location of the private key. TRANSIP_USERNAME=your-username TRANSIP_PRIVATE_KEY=/path/to/your-transip-private-key.pem # The temporary file that holds the request. TOKEN_REQUEST_BODY=$(mktemp) # Create the request from your username. # We're going to write DNS entries so 'read_only' must be 'false'. # The request also needs a random nonce. # The token is only needed for a short time, 30 seconds is enough # in a Bash script. # I vagely remember that the label must be unique, so some randomness # is added there as well. cat <<EOF > "$TOKEN_REQUEST_BODY" { "login": "$TRANSIP_USERNAME", "nonce": "$(openssl rand -base64 15)", "read_only": false, "expiration_time": "30 seconds", "label": "Add dkim dns entry $RANDOM", "global_key": true } EOF # Sign the request with openssl and encode the signature in base64. SIGNATURE=$( cat "$TOKEN_REQUEST_BODY" | openssl dgst -sha512 -sign $TRANSIP_PRIVATE_KEY | base64 --wrap=0 ) # Send the request with curl. # Note how we use '--data-binary' option to make sure curl transmit # the request byte-for-byte as it was generated. TOKEN_JSON=$( curl \ --silent \ --show-error \ -X POST \ -H "Content-Type: application/json" \ -H "SIGNATURE: $SIGNATURE" \ --data-binary "@$TOKEN_REQUEST_BODY" \ https://api.transip.nl/v6/auth ) rm -rf $TOKEN_REQUEST_BODY # Extract the TOKEN from the response using jq. TOKEN=$(echo "$TOKEN_JSON" | jq --raw-output .token) if [[ "$TOKEN" == "null" ]]; then echo "Failed to get token" echo "$TOKEN_JSON" exit 1 fi

Now we can collect the data to write a DNS entry:

DNS_DOMAIN="your-domain.com" DNS_NAME="unique-dkim-key-name._domainkey" DNS_VALUE="v=DKIM1; h=sha256; t=s; p=MIIBIjANBgkqhkiG9....DAQAB"

I am using amavisd for DKIM so the values can be fetched with some grep/awk trickery:

DNS_DOMAIN="my-domain.com" DNS_NAME=$(amavisd showkeys | grep -o '^[^.]*._domainkey') DNS_VALUE=$(amavisd showkeys | awk -F'"' '$2 != "" {VALUE=VALUE $2}; END {print VALUE}')

Now we can create the DNS entry for the DKIM key:

# Create the request. DNS_REQUEST_BODY=$( cat <<EOF {"dnsEntry":{"name":"${DNS_NAME}","expire":86400,"type":"TXT","content":"${DNS_VALUE}"}} EOF ) # Send the request with curl. REGISTER_RESULT="$( curl \ --silent \ --show-error \ -X POST \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $TOKEN" \ -d "$DNS_REQUEST_BODY" \ "https://api.transip.nl/v6/domains/${DNS_DOMAIN}/dns" )" if [[ "$REGISTER_RESULT" != "[]" ]]; then echo "Failed to register new DKIM DNS entry" echo "$REGISTER_RESULT" exit 1 fi

Note that this time this request is stored in a Bash variable.

Update 2024-01-50: Constructed variable DNS_REQUEST_BODY with cat instead of read because the latter exists with a non-zero exit code causing the script to exit.

Zio-kafka, faster than java-kafka

TlDR: Concurrency and pre-fetching gives zio-kafka a higher consumer throughput than the default java Kafka client for most workloads.

Zio-kafka is an asynchronous Kafka client based on the ZIO async runtime. It allows you to consume from Kafka with a ZStream per partition. ZStreams are streams on steroids; they have all the stream operators known from e.g. RX-Java and Monix, and then some. ZStreams are supported by ZIO's super efficient runtime.

So the claim is that zio-kafka consumes faster than the regular Java client. But zio-kafka wraps the default java Kafka client. So how can it be faster? The trick is that with zio-kafka, processing (meaning your code) runs in parallel with the code that pulls records from the Kafka broker.

Obviously, there is some overhead in distributing the received records to the streams that need to process it. So the question then is, when is the extra overhead of zio-kafka less than the gain by parallel processing? Can we estimate this? In turns out we can! For this estimate we use the benchmarks that zio-kafka runs from GitHub. In particular, we look at these 2 benchmarks with runs from 2024-11-30:

• the throughput benchmark, uses zio-kafka and takes around 592 ms
• the kafkaClients benchmark, uses Kafka java client directly and takes around 544 ms

Both benchmarks:

• run using JMH on a 4 core github runner with 16GB RAM
• consume 50k records of ~512 bytes from 6 partitions
• process the records by counting the number of incoming records, batch by batch
• are using the same consumer settings, in particular max.poll.records is set to 1000
• the broker runs in the same JVM as the consumer, so there is almost no networking overhead
• do not commit offsets (but note that committing does not change the outcome of this article)

First we calculate the overhead of java-kafka. For this we assume that counting the number of records takes no time at all. This is reasonable, a count operation is just a few CPU instructions, nothing compared to fetching something over the network, even if it is on the same machine. Therefore, in the figure, the 'processing' rectangle collapses to a thick vertical line.

Polling and processing as blocks of execution on a thread/fiber.

We also assume that every poll returns the same amount of records, and takes the same amount of time. This is not how it works in practise, but when the records are already available we're probably not that far off. As the program consumes 50k records, and each poll returns a batch of 1k records, there are 50 polls. Therefore, the overhead of java-kafka is 544/50 ≈ 11 ms per poll.

Now we can calculate the overhead of zio-kakfa. Again we assume that processing takes no time at all, so that every millisecond that the zio-kafka benchmark takes longer can be attributed to zio-kafka's overhead. The zio-kakfa benchmark runs longer for 592-544 = 48 ms. Therefore zio-kafka's overhead is 48/50 ≈ 1ms per poll.

Now lets look at a more realistic scenario where processing does take time.

Polling and processing as blocks of execution on a thread/fiber.

As you can see a java-kafka program alternates between polling and processing, while the zio-kafka program processes the records in parallel distributed over 6 streams (1 stream per partition, each stream runs independently on its own ZIO fiber). In the figure we assume that the work is evenly distributed, but unless the load is really totally skewed, it won't matter that much in practice due to zio-kafka's per-partition pre-fetching. As you can see the zio-kafka program in this scenario is much faster. But is this a realistic example? When do zio-kafka programs become faster than java-kafka programs?

Observe that how long polling takes is not important for this question because polling time is the same for both programs. So we will only look at the time between polls. We use (p) for the processing time per batch for the java-kafka program. For the zio-kafka program time between polls is equal to zio-kafka's overhead (o) plus processing time per batch divided by the number of partitions (n). So we want to know for which p:

$\mathrm{o}+\frac{\mathrm{p}}{\mathrm{n}}\le \mathrm{p}$

This solves to:

$\mathrm{p}\ge \frac{\mathrm{o\; n}}{\mathrm{n\; -\; 1}}$

For the benchmark we fill in the zio-kakfa overhead (o = 1 ms) and the number of partitions (n = 6) and we get: p ≥ 1.2 ms. (Remember, this is processing time per batch of 1000 records.)

In the previous paragraph we assumed that processing is IO intensive. When the processing is compute intensive, ZIO can not actually run more fibers in parallel than the number of cores available. In our example we have 4 cores, using n = 4 gives p ≥ 1.3 ms which is still a very low tipping point.

Conclusion

For IO loads, or with enough cores available, even quite low processing times per batch, makes zio-kafka consume faster than the plain java Kafka library. The plain java consumer is only faster for trivial processing tasks like counting.

If you'd like to know what this looks like in practice, you can take a look at this real-world example application: Kafka Big-Query Express. This is a zio-kafka application recently open sourced by Adevinta. Here is the Kafka consumer code. (Disclaimer: I designed and partly implemented this application.)

Traefik v2 enable HSTS, Docker and nextcloud

This took me days to figure out how to configure Traefik v2. Here it is for posterity.

This is a docker-compose.yaml fragment to append to a service section:

labels: - "traefik.enable=true" - "traefik.http.routers.service.rule=Host(`www.example.com`)" - "traefik.http.routers.service.entrypoints=websecure" - "traefik.http.routers.service.tls.certresolver=myresolver" - "traefik.http.middlewares.servicests.headers.stsincludesubdomains=false" - "traefik.http.middlewares.servicests.headers.stspreload=true" - "traefik.http.middlewares.servicests.headers.stsseconds=31536000" - "traefik.http.middlewares.servicests.headers.isdevelopment=false" - "traefik.http.routers.service.middlewares=servicests"

It will:

• tell Traefik to direct traffic for www.example.com to this container,
• on the websecure entrypoint (this is configured statically),
• using the myresolver (for Acme, resolver also configured statically),
• configure middleware to add HSTS headers,
• enable the middleware.

Nextcloud

Here is a slightly more complex example for a nextcloud deployment which includes the recommended redirects.

labels: - "traefik.enable=true" - "traefik.http.routers.nextcloud.rule=Host(`nextcloud.example.com`)" - "traefik.http.routers.nextcloud.entrypoints=websecure" - "traefik.http.routers.nextcloud.tls.certresolver=myresolver" - "traefik.http.middlewares.nextcloudredir.redirectregex.permanent=true" - "traefik.http.middlewares.nextcloudredir.redirectregex.regex=https://(.*)/.well-known/(card|cal)dav" - "traefik.http.middlewares.nextcloudredir.redirectregex.replacement=https://$$1/remote.php/dav/" - "traefik.http.middlewares.nextcloudsts.headers.stsincludesubdomains=false" - "traefik.http.middlewares.nextcloudsts.headers.stspreload=true" - "traefik.http.middlewares.nextcloudsts.headers.stsseconds=31536000" - "traefik.http.middlewares.nextcloudsts.headers.isdevelopment=false" - "traefik.http.routers.nextcloud.middlewares=nextcloudredir,nextcloudsts"

Continuation parsers and encoders

I finally got around to writing about my hack project of last year. It was an exploration of what can be done with continuation parsers and encoders in order to implement a very fast single-copy asynchronous Thrift implementation.

Continuation parsers and encoders try to decode (read)/encode (write) their data directly from/to a network buffer. When the buffer has been fully read/written, it asks for more network buffers to continue.

For more information see the thrift-stream repository on GitHub.

Generate a certificate signing request (CSR) as a 1 liner

As I run my own (secure) web and mail server I frequently have to get certificates. You could run with a selfsigned certificate, but that is not ideal; desktop browsers are very noisy about them, and mobile browsers are outright hostile. Installing certificates on a mobile phone are no fun (though, hat of for CAdroid).

Now that Letsencrypt is live, I wanted to try it out. However, Letsencrypt does not make it easy for you to keep using the same private key. This is necessary as the Android's http client pins your certificates (the one I needed anyway). Luckily Letsencrypt allows you to use a CSR that is generated outside of their tools.

Letsencrypt certificates are only valid for a short time (90 days) so automation is key. Unfortunately, openssl does not make it easy to fully automate creating CSRs, especially when you need 'Alternative Names', a requirement from Letsencrypt.

Luckily I found a solution from Andrew Leahy. Here it is:

openssl req -new -nodes -sha256 -key private-key.pem \ -subj "/C=US/ST=CA/O=Acme, Inc./OU=Acme Example/CN=example.com" \ -reqexts SAN \ -config <(cat /etc/ssl/openssl.cnf <(printf "[SAN]\nsubjectAltName=DNS:example.com,DNS:www.example.com")) \ -out domain.csr

This will generate a CSR in file domain.csr with set country (C), state (ST), organization (O), organization unit (OU) and most importantly the common name (CN) and in addition two alternative names example.com and www.example.com. Thanks Andrew!

Configuring Postfix/Dovecot for Microsoft Windows Live Mail

Personal mail gets no love from Microsoft. The last 10 year I have not seen their product change a lot. Notable I see name changes (always a bad sign) and some visual changes. The actual implementation is still the same: not respecting standards. I run a Postfix/Dovecot installation for my family mail. I have had many many different email clients connect to it without large problems. With Microsoft Windows Live Mail, Outlook Express or whatever it is called today, it just doesn't work. Anyway, here is what you can do:

(I am assuming you are using something like Ubuntu with Postfix for SMTP with TLS (actually STARTTLS) on port 25, and Dovecot with IMAPS on port 993.)

Open the file /etc/dovecot/dovecot.conf, and update the line with auth_mechanisms to the following. The trick is that login has to come first:

auth_mechanisms = login plain

Repeat this trick for Postfix in /etc/postfix/sasl/smtp.conf:

mech_list: login plain

Restart Postfix and Dovecot, and you're good to go!

Fast directory transfer on Unix machines

Here is a little trick to transfer a big folder from one unix machine to another in 2 variations.

In this variation netcat is in listen mode on the target (execute in the given order):

on target> nc -l 19001 | lzop -d | tar x on source> tar c [directory to copy] | lzop | nc [target] 19001

In the second variation netcat is in listen mode on the source system (again, execute in the given order):

on source> tar c [directory to copy] | lzop | nc -l 19001 on target> nc [source] 19001 | lzop -d | tar x

Make sure you have a decent network connection, 1 Gbit/s is fine.

Update 2015-11-18: I experimented with cpio and found that it is a lot faster then tar. I also added pipe viewer (pv) to get some sense of when a transfer is done.

This is using cpio with netcat in listen mode on the target and send mode on the source:

on target> nc -l 19001 | lzop -d | cpio -idm on source> cd [directory to copy]; find . -depth -print0 | cpio -o0 \ | pv -s $(du -ks . | awk '{print $1}')k \ | lzop | nc [target] 19001

This is what it looks like on the source side:

Reliably sending email with Spring

Update 2009-09-12: I no longer recommend this library. Please see the comments.

My colleague Allard just pointed me to an old but very useful library: HA-JavaMail.

The email sender that the JVM provides has some serious shortcomings. It does not automatically open a new connection when the connection was closed and you can forward your e-mail to 1 SMTP server only. Furthermore, it is not so fast. HA-JavaMail circumvents these problems by wrapping the JVM implementation.

In this tiny article I explain how you configure HA-JavaMail from Spring. First make sure you have the HA-JavaMail jar, the Spring jar, the JavaMail mail.jar and the JAF activation.jar library on your classpath. The latter two are available by default in full JEE containers.

Convert your mail sender bean declaration from this:

<bean id="mailSender" class="org.springframework.mail.javamail.JavaMailSenderImpl"> <property name="mail.host" value="localhost"/> </bean>

to something like this:

<bean id="mailSender" class="org.springframework.mail.javamail.JavaMailSenderImpl"> <property name="javaMailProperties"> <value> mail.transport.host=localhost,mail.example.com mail.transport.pool-size=1 mail.transport.connect-timeout=0 mail.transport.connect-retry-period=60 mail.transport.sender-strategy=net.sf.hajavamail.SimpleSenderStrategy </value> </property> </bean>

Note that the example adds a second SMTP server that will be used when the first is unreachable. Only a value for mail.transport.host is required. The other values shown here are also the defaults.

That’s it. You now have a much more reliable and faster email service.

Ubuntu server 7.10 to 8.04 upgrade experiences

Yesterday I upgraded my server at home (running 3 websites and a small e-mail server) from Ubuntu 7.10 to 8.04. My main system is a nice looking iMac, completed with an eeepc for traveling and kitchen table work. But I use Ubuntu in the closet and at work.
The upgrade went rather well. Of course, I had installed as little packages from outside the repositories as possible. I even did the upgrade over ssh while that is not recommended.

Unfortunately, I only just noticed that no e-mail was being received anymore. After some head scratching (I had also changed the firewall settings), I decided to compare my backup of /etc (yes, I had one!) to the current copy.

So how do you compare such a complex directory as /etc on a Mac? I tried Eclipse and IntelliJ. Despite their excellent compare tools, they have no such thing for complete directories. Searching the web for open source solutions did not give any conclusive evidence as well. Kdiff3 was mentioned most frequently, so I decided to try that one. Kdiff3 can be installed with MacPorts, but make sure you have plenty of time. It took more then more then 2 hours on this screaming iron!
Kdiff3 is not so intuitive as the commercial products Araxis and Beyond Compare (which I have used professionally), but once you get used to it, it is a nice and powerful merging tool.

And there it was: a small copy/paste error in my dovecot configuration. Postfix uses dovecot for authentication so every TLS connection failed. Copy, paste, restart, done!

Conclusions

• Make sufficient backups of all your configuration files.
• Use the packages from your distribution as much as possible. Do not install later versions of a package unless you are prepared to install it again and again and again.
• Kdiff3 is an acceptable diff tool on the Mac.
• Do not be afraid for an Ubuntu server upgrade!

Firewall gone crazy!

Today I saw the solution to one of my weirdest problems ever. We have been searching for about a whole week why our JEE application would suddenly stop. The problem was that it took at least an hour to reproduce it (if at all), and even then we could find next to nothing in the logs. We only started to see the light when we noticed the difference between environments where the error was reproducable, and where is was not reproducable: the firewall.

Turns out the firewall between the application server and the database server would stop all trafic on the JDBC connection after an hour if idling, without actually killing the connection. Weblogic, and all other applications we tried, totally freak out when this happens. What on earth did the creators of that firewall think? When you kill a connection, then kill it! Do not just stop the data flow!

Well anyway, the solution? We now let Weblogic 'test' the connection every 10 minutes. Apparently, the firewall is happier when there is some traffic now and then.