Commit 5a65b590 authored by Evangelos Foutras's avatar Evangelos Foutras 🐱
Browse files

Merge branch 'upgrade-servers-task' into 'master'

Add simple playbook task for upgrading servers

We want to treat our servers as cattle; hopefully when this is fleshed
out a bit more, it can accomplish the job without too many casualties.

See merge request !475
parents 9227825b 006cdc04
Pipeline #10574 passed with stage
in 33 seconds
......@@ -128,6 +128,23 @@ The following steps should be used to update our managed servers:
* checkservices
* reboot
##### Semi-automated server updates (experimental)
For updating a lot of servers in a more unattended manner, the following
playbook can be used:
ansible-playbook playbooks/tasks/upgrade-servers.yml [-l SUBSET]
It runs `pacman -Syu` on the targeted hosts in batches and then reboots them.
If any server fails to reboot successfully, the rolling update stops and
further batches are cancelled. To display the packages updated on each host,
you can pass the `--diff` option to ansible-playbook.
Using this update method, `.pacnew` files are left unmerged which is OK for
most configuration files that are managed by Ansible. However, care must be
taken with updates that require manual intervention (e.g. major PostgreSQL
releases).
## Servers
This section has been moved to [docs/servers.md](docs/servers.md).
......
---
- name: ensure latest keyring
pacman:
name: archlinux-keyring
state: latest
update_cache: yes
- name: upgrade all packages
pacman:
update_cache: yes
upgrade: yes
register: pacman_upgrade
- name: check for running builds
block:
- name: list build-related processes
command: pgrep -x 'mkarchroot|makechrootpkg|systemd-nspawn'
register: pgrep
ignore_errors: true
- name: abort reboot with running builds
meta: end_host
when: pgrep is succeeded
when: "'buildservers' in group_names"
- name: gemini pre-reboot checks
block:
- name: wait for svntogit to finish
wait_for:
path: /srv/svntogit/update-repos.sh.lock
state: absent
- name: list logged on users
command: who
register: who
- name: abort reboot with logged on users
meta: end_host
when:
- who is changed
- who.stdout_lines|length > 1
when: inventory_hostname == "gemini.archlinux.org"
- name: reboot
reboot:
when: pacman_upgrade is changed
---
- name: upgrade and reboot all hetzner servers
hosts: all,!kape_servers,!packet_net,!rsync_net,!hetzner_storageboxes
max_fail_percentage: 0
serial: 20%
gather_facts: false
tasks:
- name: upgrade each host in this batch
include_tasks: include/upgrade-server.yml
- name: upgrade and reboot all kape and packet.net servers
hosts: kape_servers,packet_net
max_fail_percentage: 0
serial: 1
gather_facts: false
tasks:
- name: upgrade each host in this batch
include_tasks: include/upgrade-server.yml
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment