​I am a big fan of automation. But when it comes to my personal servers I am actually against it. We all know the XKDC comic about automation.

There is a lot of upfront effort to put into ansible or something like NixOS which will never be reused anyway. At least for my home server and VPS. These are my pets 🐈, my snowflakes ❄️ and that is okay. However, that does not mean I am a caveman, manually logging into my server and do maintenance. That would be silly.

So let's automate that with unattendedUpgrades:

• Automatic updates (security, kernel, updates, third-party repos)
• Automatic reboot
• Automatic clean-up of packages
• Get an email notification

Setup Emails

From my research, I think the easiest way is to go with a light SMTP client called msmtp.

So let's install that with sudo apt-get install msmtp ca-certificates and create a config here:

sudo bash -c 'cat <<EOF > /etc/msmtprc
# Set default values for all following accounts.
defaults
auth           on
tls            on
tls_trust_file /etc/ssl/certs/ca-certificates.crt
logfile        /var/log/msmtp.log
# Gmail
account        gmail
host           smtp.gmail.com
port           587
from           YOUR_FROM_EMAIL
user           YOUR_GMAIL_USER
password       APP_PASSWORD
# Set a default account
# This needs to be at the end!
account default: gmail
EOF'

The APP_PASSWORD cat be created here: https://myaccount.google.com/apppasswords. Add your your YOUR_FROM_EMAIL and YOUR_TO_EMAIL and we should be good.

After adding the password we should make this file only readable by root:

sudo chomd 600 /etc/msmtprc

Now we need to create a symlink for sendmail. I guess it is because unattendedUpgrades is using sendmail. I tried without the symlink and then mailing does not work. Maybe some of you know why?

ln -s /usr/bin/msmtp /usr/sbin/sendmail

Configure unattendedUpgrades

Now our server is able to send emails we need to configure unattendedUpgrades.

The default config can be found in /etc/apt/apt.conf.d/50unattended-upgrades with all options and some explanation regarding each option. It is recommended to create a new config since an OS update could overwrite everything. So here we go with my personal config:

sudo bash -c 'cat <<EOF > /etc/apt/apt.conf.d/51unattended-upgrades-custom
Unattended-Upgrade::Allowed-Origins {
        "${distro_id}:${distro_codename}";
        "${distro_id}:${distro_codename}-security";
        "${distro_id}ESMApps:${distro_codename}-apps-security";
        "${distro_id}ESM:${distro_codename}-infra-security";
        "${distro_id}:${distro_codename}-updates";
        "Docker:${distro_codename}";
        "Netdata:${distro_codename}";
};
Unattended-Upgrade::Mail "MY_MAIL";
Unattended-Upgrade::MailReport "on-change";
Unattended-Upgrade::Remove-Unused-Kernel-Packages "true";
Unattended-Upgrade::Remove-New-Unused-Dependencies "true";
Unattended-Upgrade::Remove-Unused-Dependencies "true";
Unattended-Upgrade::Automatic-Reboot "true";
Unattended-Upgrade::Automatic-Reboot-Time "08:00";
EOF'

This will:

• update basically all packages, as packages from Docker and Netdata
• set my email
• only send an email when something has changed
• will remove unused Kernel packages
• Removed new and not used dependencies
• will automatically reboot at 08:00

You can adjust that for your use case, but that is how I run my box for several years now and it is fine 🙂

How to configure third party repositories with unattendedUpgrades?

I was wondering this myself. When I was logging into my server I always saw something like this when running and sudo apt-get update && sudo apt-get dist-upgrade:

The following packages will be upgraded:
  containerd.io docker-ce docker-ce-cli

I was confused and did some digging.

With apt-cache policy you can view your repos and get a list like this:

 500 http://repo.netdata.cloud/repos/repoconfig/ubuntu noble/ Packages
     release v=24.04,o=Netdata,n=noble,l=Netdata,c=
     origin repo.netdata.cloud
 500 http://repo.netdata.cloud/repos/edge/ubuntu noble/ Packages
     release v=24.04,o=Netdata,n=noble,l=Netdata,c=
     origin repo.netdata.cloud
 500 https://download.docker.com/linux/ubuntu noble/stable amd64 Packages
     release o=Docker,a=noble,l=Docker CE,c=stable,b=amd64
     origin download.docker.com

The o is for origin and the a is for archive. Those are not configured by default for unattenededUpgrades.

The syntax is origin:archive. Therefore we would need to add them like Docker:noble and Netdata:noble. But we can also use the variable ${distro_codename}" to make it more generic, if we ever upgrade the system.

Unattended-Upgrade::Allowed-Origins {
        "Docker:${distro_codename}";
        "Netdata:${distro_codename}";
};

Testing

Now we have everything configured and should test if everything is working, including mailing.

First, we should set email to always Unattended-Upgrade::MailReport "on-change";.

Install old docker version

Let's see what docker has to offer in regard of older versions, that way we also test if the third-party repos will be upgraded as well.

apt-cache madison docker-ce
 docker-ce | 5:26.1.4-1~ubuntu.24.04~noble | https://download.docker.com/linux/ubuntu noble/stable amd64 Packages
 docker-ce | 5:26.1.3-1~ubuntu.24.04~noble | https://download.docker.com/linux/ubuntu noble/stable amd64 Packages
 docker-ce | 5:26.1.2-1~ubuntu.24.04~noble | https://download.docker.com/linux/ubuntu noble/stable amd64 Packages
 docker-ce | 5:26.1.1-1~ubuntu.24.04~noble | https://download.docker.com/linux/ubuntu noble/stable amd64 Packages
 docker-ce | 5:26.1.0-1~ubuntu.24.04~noble | https://download.docker.com/linux/ubuntu noble/stable amd64 Packages
 docker-ce | 5:26.0.2-1~ubuntu.24.04~noble | https://download.docker.com/linux/ubuntu noble/stable amd64 Packages
 docker-ce | 5:26.0.1-1~ubuntu.24.04~noble | https://download.docker.com/linux/ubuntu noble/stable amd64 Packages
 docker-ce | 5:26.0.0-1~ubuntu.24.04~noble | https://download.docker.com/linux/ubuntu noble/stable amd64 Packages

Assign a variable for easy typing:

VERSION_STRING=5:26.1.0-1~ubuntu.24.04~noble

Then let's install an older version of docker:

sudo apt-get install docker-ce=$VERSION_STRING docker-ce-cli=$VERSION_STRING containerd.io docker-buildx-plugin docker-compose-plugin

Check if we have a package to update:

apt list --upgradable
Listing... Done
docker-ce-cli/noble 5:26.1.4-1~ubuntu.24.04~noble amd64 [upgradable from: 5:26.1.0-1~ubuntu.24.04~noble]
docker-ce/noble 5:26.1.4-1~ubuntu.24.04~noble amd64 [upgradable from: 5:26.1.0-1~ubuntu.24.04~noble]

Install old kernel

Since we also want to know if we get an email on reboot let's install an older kernel as well.

apt-cache madison linux-image-generic
linux-image-generic | 6.8.0-35.35 | https://mirror.hetzner.com/ubuntu/packages noble-updates/main amd64 Packages
linux-image-generic | 6.8.0-35.35 | https://mirror.hetzner.com/ubuntu/security noble-security/main amd64 Packages
linux-image-generic | 6.8.0-31.31 | https://mirror.hetzner.com/ubuntu/packages noble/main amd64 Packages

sudo apt-get install linux-image-generic=6.8.0-31.31

On my Hetzner Ubuntu 22.04 VPS, I had to edit the timeout for grub in order to choose a kernel at startup time:

sudo vim /etc/default/grub

GRUB_TIMEOUT=10

Reboot the system with the old kernel.

Uninstall the newest kernel:

sudo apt-get remove linux-image-6.8.0-35-generic

Test email setup

Now the test with sudo unattended-upgrade -v :

Starting unattended upgrades script
Allowed origins are: o=Ubuntu,a=noble, o=Ubuntu,a=noble-security, o=UbuntuESMApps,a=noble-apps-security, o=UbuntuESM,a=noble-infra-security, o=Ubuntu,a=noble-updates, o=Docker,a=noble, o=Netdata,a=noble, o=,a=-updates, o=Docker,a=, o=Netdata,a=
Initial blacklist:
Initial whitelist (not strict):

...

I did not paste the whole output, but you can see in the second line the allowed origins and that the Docker and Netdata repository are part of this list.

After running for quite some time you should get a message like this in the end:

All upgrades installed
Found /var/run/reboot-required, rebooting
Shutdown msg: b"Reboot scheduled for Mon 2024-06-17 08:00:00 UTC, use 'shutdown -c' to cancel."

Since a Kernel update was done and this requires a reboot we also get an email:

Conclusion

We now have a server running that updates itself and will also do a reboot. For me that is okay, others get scared. You can adjust the configuration to your use case and your preference. I like the email notification to know that something is updating itself, but I do not want to rely on my lazy self to ssh into that machine and reboot. I know myself 😅

One downside of email notification: If it does not work for whatever reason, you do not have a reminder to reboot. Therefore I just do the reboot.

Last Words

How am I doing?

I love hearing from readers, and I am always looking for feedback. Is there anything you would like to read more or less of? You can just click on the 👍 or 👎 down below. What do you enjoy most about the newsletter, just let me know in the comments.

Have a great day! 👋