An Ansible playbook to implement automatic updates (and upgrades) on our hosts.
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Max Mehl 77f51841b9
fix group name
1 week ago
group_vars enlarge reboot window from 30m to 60m 1 week ago
inventory@d3f07cc707 fix group name 1 week ago
roles shifted file deletion to configure-timers 2 weeks ago
.gitmodules added README.md and submodules 2 weeks ago
README.md fix group name 1 week ago
auto_update.yml shifted position of role-call and put file deletion into configure-timers 2 weeks ago
auto_upgrade.yml shifted position of role-call and put file deletion into configure-timers 2 weeks ago
unattended-upgrades.drawio updated timeline 2 weeks ago
unattended-upgrades.png updated timeline 2 weeks ago

README.md

Unattended updates and upgrades

Aim

Implement an automatic update and upgrade mechanism both on existing and new hosts using the package unattended-upgrades and - in the case of updating only the package lists - configuration files in etc/apt/apt.conf.d.

Usage

First, clone the repos (including its submodules fsfe-system-hackers/inventory and jnv/ansible-role-unattended-upgrades)

git clone --recurse-submodules git@git.fsfe.org:fsfe-system-hackers/unattended-upgrades.git

Please note: It is advisable to regularly Update the inventory submodule in order to reflect the newest changes to the list of our hosts and the groups that they are in:

git submodule update --remote inventory

Apply upgrades automatically and reboot if necessary

TLDR

to reboot at 3am if required

ansible-playbook -i inventory/hosts -l reboot_3am auto_upgrade.yml

to reboot at 5am if required:

ansible-playbook -i inventory/hosts -l reboot_5am auto_upgrade.yml

The whole story

Most machines in our hosts should not require manual intervention for installing updates and should be automatically rebooted if the update requires it. Some machines, however should never reboot at the same time. Hence, there are two groups, namely reboot_3am and reboot_5am in our inventory of hosts. The first step for every update is an update of the package lists (apt-get update). As specified in the role configure timers, this happens at 1:00AM UTC (with a randomized delay of 30min) every night. This time can be specified using the variable apt_update_time.

The next step is then to run apt-get upgrade. This happens via the configuration of the package unattended-upgrades whose configuration will reside in /etc/apt.conf.d/50unattended-upgrades. The setup and configuration of unattended-upgrades is handled through the eponymous Ansible role by jnv. For details see on how this rule is used here, see the role's README and the corresponding playbook file. If not otherwise specified via the variable apt-upgrade-time, upgrades are run at 2:00 UTC, again with a randomized delay of 30min. Subsequently, and only if required, the hosts are rebooted at either 3:00AM or 5:00AM depending on which group they are in.

a timeline

Only download new packages (no upgrades, no reboot)

TLDR

ansible-playbook -i inventory/hosts -l update_only auto_update.yml

The whole story

Some hosts should not apply upgrades automatically. It's fine for them to download the packages that should be upgraded, but the run of apt-get upgrade itself should always be triggered by a human.

This is realised by configuring /etc/apt/apt.conf.d/20auto-updates-only as follows:

APT::Periodic::Unattended-Upgrade "0";
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "1";
APT::Periodic::AutocleanInterval "21";

To be sure, the apt-daily-upgrade.timer is disabled, while apt-daily.timer is left intact as specified above.

Resources

Troubleshooting

FATAL -> Failed to fork

This error seems to happen if too little RAM is available. However, there might also be other reasons, as this thread suggests.