An Ansible playbook to implement automatic updates (and upgrades) on our hosts.
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
This repo is archived. You can view files and clone it, but cannot push or open issues/pull-requests.
Max Mehl 069a71e101
bump inventory
1 year ago
group_vars enlarge reboot window from 30m to 60m 1 year ago
inventory@71c14eb07c bump inventory 1 year ago
roles bumped unattended-upgrades 1 year ago
.gitmodules added README.md and submodules 1 year ago
README.md added resource 1 year ago
ansible.cfg define default inventory 1 year ago
auto_update.yml enhanced comments 1 year ago
auto_upgrade.yml enhanced comments 1 year ago
unattended-upgrades.drawio fixed diagram 1 year ago
unattended-upgrades.png fixed diagram 1 year ago

README.md

Unattended updates and upgrades

Aim

Implement an automatic update and upgrade mechanism both on existing and new hosts using the package unattended-upgrades and - in the case of updating only the package lists - configuration files in etc/apt/apt.conf.d.

Usage

First, clone the repos (including its submodules fsfe-system-hackers/inventory and jnv/ansible-role-unattended-upgrades)

git clone --recurse-submodules git@git.fsfe.org:fsfe-system-hackers/unattended-upgrades.git

Please note: It is advisable to regularly Update the inventory submodule in order to reflect the newest changes to the list of our hosts and the groups that they are in:

git submodule update --remote inventory

Apply upgrades automatically and reboot if necessary

TLDR

to reboot at 3am if required

ansible-playbook -i inventory/hosts -l reboot_3am auto_upgrade.yml

to reboot at 5am if required:

ansible-playbook -i inventory/hosts -l reboot_5am auto_upgrade.yml

The whole story

Most machines in our hosts should not require manual intervention for installing updates and should be automatically rebooted if the update requires it. Some machines, however should never reboot at the same time. Hence, there are two groups, namely reboot_3am and reboot_5am in our inventory of hosts. The first step for every update is an update of the package lists (apt-get update). As specified in the role configure timers, this happens at 1:00AM UTC (with a randomized delay of 30min) every night. This time can be specified using the variable apt_update_time.

The next step is then to run apt-get upgrade. This happens via the configuration of the package unattended-upgrades whose configuration will reside in /etc/apt.conf.d/50unattended-upgrades. The setup and configuration of unattended-upgrades is handled through the eponymous Ansible role by jnv. For details see on how this rule is used here, see the corresponding playbook file. If not otherwise specified via the variable apt-upgrade-time, upgrades are run at 2:00 UTC, again with a randomized delay of 30min. Subsequently, and only if required, the hosts are rebooted at either 3:00AM or 5:00AM depending on which group they are in.

a timeline

Only download new packages (no upgrades, no reboot)

TLDR

ansible-playbook -i inventory/hosts -l update_only auto_update.yml

The whole story

Some hosts should not apply upgrades automatically. It's fine for them to download the packages that should be upgraded, but the run of apt-get upgrade itself should always be triggered by a human.

This is realised by configuring /etc/apt/apt.conf.d/20auto-updates-only as follows:

APT::Periodic::Unattended-Upgrade "0";
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "1";
APT::Periodic::AutocleanInterval "21";

To be sure, the apt-daily-upgrade.timer is disabled, while apt-daily.timer is left intact as specified above.

Resources

Troubleshooting

FATAL -> Failed to fork

This error seems to happen if too little RAM is available. However, there might also be other reasons, as this thread suggests.