README.md 3.56 KB
Newer Older
Sven-Hendrik Haase's avatar
Sven-Hendrik Haase committed
1
2
3
4
# Arch Infrastructure

This repository contains the complete collection of ansible playbooks and roles for the Arch Linux infrastructure.

5
6
7
It also contains git submodules so you have to run `git submodule update --init
--recursive` after cloning or some tasks will fail to run.

Sven-Hendrik Haase's avatar
Sven-Hendrik Haase committed
8
9
#### Instructions
All systems are set up the same way. For the first time setup in the Hetzner rescue system,
10
run the provisioning script: `ansible-playbook playbooks/tasks/install-arch.yml -l $host`.
Sven-Hendrik Haase's avatar
Sven-Hendrik Haase committed
11
12
13
14
The provisioning script configures a sane basic systemd with sshd. By design, it is NOT idempotent.
After the provisioning script has run, it is safe to reboot.

Once in the new system, run the regular playbook: `ansible-playbook playbooks/$hostname.yml`. This
15
playbook is the one regularity used for administrating the server and is entirely idempotent.
Sven-Hendrik Haase's avatar
Sven-Hendrik Haase committed
16

17
18
19
20
21
22
##### Note about first time certificates

The first time a certificate is issued, you'll have to do this manually by yourself. First, configure the DNS to
point to the new server and then run a playbook onto the server which includes the nginx role. Then on the server,
it is necessary to run the following once:

Sven-Hendrik Haase's avatar
Sven-Hendrik Haase committed
23
    certbot certonly --email webmaster@archlinux.org --agree-tos --rsa-key-size 4096 --renew-by-default --webroot -w /var/lib/letsencrypt/ -d <domain-name>
24

25
26
Note that some roles already run this automatically.

Sven-Hendrik Haase's avatar
Sven-Hendrik Haase committed
27
28
29
30
31
32
33
34
35
##### Note about packer

We use packer to build snapshots on hcloud to use as server base images.
In order to use this, you need to install packer and then run

	packer build -var $(./packer/get_hetzner_cloud_api_key.sh) packer/archlinux.json

This will take some time after which a new snapshot will have been created on the primary hcloud archlinux project.

36
37
38
39
40
41
42
43
44
45
46
47
48
49
##### Note about terraform

We use terraform to provision a part of the infrastructure on hcloud.
In order to use this, you need to install terraform and then run

    terraform plan -var $(./packer/get_hetzner_cloud_api_key.sh) terraform

This will show you planned changes between the current infrastructure and the desired infrastructure.
You can then run

    terraform apply -var $(./packer/get_hetzner_cloud_api_key.sh) terraform

to actually apply your changes.

50
51
52
53
54
55
##### Note about opendkim

The opendkim DNS data has to be added to DNS manually. The roles verifies that the DNS is correct before starting opendkim.

The file that has to be added to the zone is `/etc/opendkim/private/$selector.txt`.

56
57
58
59
60
#### Updating servers

The following steps should be used to update our managed servers:

* pacman -Syu
61
* manually update the kernel, since it is in IgnorePkg by default
62
63
64
65
* sync
* checkservices
* reboot

Sven-Hendrik Haase's avatar
Sven-Hendrik Haase committed
66
67
68
69
70
71
72
73
74
75
76
## Servers

### vostok

#### Services
- backups

### orion

#### Services
- repos/sync (repos.archlinux.org)
77
- sources (sources.archlinux.org)
Sven-Hendrik Haase's avatar
Sven-Hendrik Haase committed
78
- archive (archive.archlinux.org)
79
- torrent tracker hefurd (tracker.archlinux.org)
Sven-Hendrik Haase's avatar
Sven-Hendrik Haase committed
80
81
82
83
84
85
86

### apollo

#### Services
- bbs (bbs.archlinux.org)
- wiki (wiki.archlinux.org)
- aur (aur.archlinux.org)
Jelle van der Waa's avatar
Jelle van der Waa committed
87
- flyspray (bugs.archlinux.org)
Sven-Hendrik Haase's avatar
Sven-Hendrik Haase committed
88
- mailman
Sven-Hendrik Haase's avatar
Sven-Hendrik Haase committed
89
- planet (planet.archlinux.org)
Sven-Hendrik Haase's avatar
Sven-Hendrik Haase committed
90
91
92
- bugs (bugs.archlinux.org)
- archweb
- patchwork
Sven-Hendrik Haase's avatar
Sven-Hendrik Haase committed
93
- projects (projects.archlinux.org)
Sven-Hendrik Haase's avatar
Sven-Hendrik Haase committed
94
95
96
97
98
99

### soyuz

#### Services
- build server (pkgbuild.com)
- releng
100
101
102
103
- sogrep
- /~user/ webhost
- irc bot (phrik)
- quassel core
Jelle van der Waa's avatar
Jelle van der Waa committed
104
105
106
- matrix
- docker images
- arch boxes (packer)
107
108
109
110
111
112
113
114
115
116
117


## Ansible repo workflows

### Replace vault password and change vaulted passwords

 - Generate a new key and save it as ./new-vault-pw: `pwgen -s 64 1 > new-vault-pw`
 - `for i in $(ag ANSIBLE_VAULT -l); do ansible-vault rekey --new-vault-password-file new-vault-pw $i; done`
 - Change the key in misc/vault-password.gpg
 - `rm new-vault-pw`