README.md 4.84 KB
Newer Older
Sven-Hendrik Haase's avatar
Sven-Hendrik Haase committed
1
2
3
4
# Arch Infrastructure

This repository contains the complete collection of ansible playbooks and roles for the Arch Linux infrastructure.

5
6
7
It also contains git submodules so you have to run `git submodule update --init
--recursive` after cloning or some tasks will fail to run.

8
9
## Requirements

10
Install these packages:
Sven-Hendrik Haase's avatar
Sven-Hendrik Haase committed
11
- hcloud-python
Sven-Hendrik Haase's avatar
Sven-Hendrik Haase committed
12
13
- terraform
- terraform-provider-hcloud
14

15
16
### Instructions

Sven-Hendrik Haase's avatar
Sven-Hendrik Haase committed
17
All systems are set up the same way. For the first time setup in the Hetzner rescue system,
18
run the provisioning script: `ansible-playbook playbooks/tasks/install-arch.yml -l $host`.
Sven-Hendrik Haase's avatar
Sven-Hendrik Haase committed
19
20
21
The provisioning script configures a sane basic systemd with sshd. By design, it is NOT idempotent.
After the provisioning script has run, it is safe to reboot.

22
23
Once in the new system, run the regular playbook: `HCLOUD_TOKEN=$(misc/get_hcloud_api_key_ansible.sh) ansible-playbook playbooks/$hostname.yml`.
This playbook is the one regularity used for administrating the server and is entirely idempotent.
Sven-Hendrik Haase's avatar
Sven-Hendrik Haase committed
24

25
26
27
28
29
30
31
32
#### Note about Ansible dynamic inventories

We use a dynamic inventory script in order to automatically get information for
all servers directly from hcloud. You don't really have to do anything to make
this work but you should keep in mind to NOT add hcloud servers to `hosts`!
They'll be available automatically.

#### Note about first time certificates
33
34
35
36
37

The first time a certificate is issued, you'll have to do this manually by yourself. First, configure the DNS to
point to the new server and then run a playbook onto the server which includes the nginx role. Then on the server,
it is necessary to run the following once:

Sven-Hendrik Haase's avatar
Sven-Hendrik Haase committed
38
    certbot certonly --email webmaster@archlinux.org --agree-tos --rsa-key-size 4096 --renew-by-default --webroot -w /var/lib/letsencrypt/ -d <domain-name>
39

40
41
Note that some roles already run this automatically.

42
#### Note about packer
Sven-Hendrik Haase's avatar
Sven-Hendrik Haase committed
43
44
45
46

We use packer to build snapshots on hcloud to use as server base images.
In order to use this, you need to install packer and then run

47
	packer build -var $(./misc/get_hetzner_cloud_api_key_packer.sh) packer/archlinux.json
Sven-Hendrik Haase's avatar
Sven-Hendrik Haase committed
48
49
50

This will take some time after which a new snapshot will have been created on the primary hcloud archlinux project.

51
#### Note about terraform
52
53

We use terraform to provision a part of the infrastructure on hcloud.
54
55
56
The very first time you run terraform on your system, you'll have to init it:

    terraform init -backend-config="conn_str=postgres://terraform:$(ansible-vault view group_vars/all/vault_terraform.yml | grep vault_terraform_db_password | cut -f2 -d'"')@state.cloud.archlinux.org"
57

58
59
60
After making changes to the infrastructure in `archlinux.fg`, run

    terraform plan
61
62
63
64

This will show you planned changes between the current infrastructure and the desired infrastructure.
You can then run

65
    terraform apply
66
67
68

to actually apply your changes.

69
70
71
72
73
74
75
We store terraform state on a special server that is the only hcloud server NOT
managed by terraform so that we do not run into a chicken-egg problem. The
state server is assumed to just exist so in an unlikely case where we have to
entirely redo this infrastructure, the state server would have to be manually
set up.

#### Note about opendkim
76
77
78
79
80

The opendkim DNS data has to be added to DNS manually. The roles verifies that the DNS is correct before starting opendkim.

The file that has to be added to the zone is `/etc/opendkim/private/$selector.txt`.

81
82
83
84
85
86
87

### Finding servers requiring security updates

Arch-audit can be used to find servers in need of updates for security issues.

    ansible all -a "arch-audit -u"

88
89
90
91
92
#### Updating servers

The following steps should be used to update our managed servers:

* pacman -Syu
93
* manually update the kernel, since it is in IgnorePkg by default
94
95
96
97
* sync
* checkservices
* reboot

Sven-Hendrik Haase's avatar
Sven-Hendrik Haase committed
98
99
100
101
102
103
104
105
106
107
108
## Servers

### vostok

#### Services
- backups

### orion

#### Services
- repos/sync (repos.archlinux.org)
109
- sources (sources.archlinux.org)
Sven-Hendrik Haase's avatar
Sven-Hendrik Haase committed
110
- archive (archive.archlinux.org)
111
- torrent tracker hefurd (tracker.archlinux.org)
Sven-Hendrik Haase's avatar
Sven-Hendrik Haase committed
112
113
114
115
116
117
118

### apollo

#### Services
- bbs (bbs.archlinux.org)
- wiki (wiki.archlinux.org)
- aur (aur.archlinux.org)
Jelle van der Waa's avatar
Jelle van der Waa committed
119
- flyspray (bugs.archlinux.org)
Sven-Hendrik Haase's avatar
Sven-Hendrik Haase committed
120
- mailman
Sven-Hendrik Haase's avatar
Sven-Hendrik Haase committed
121
- planet (planet.archlinux.org)
Sven-Hendrik Haase's avatar
Sven-Hendrik Haase committed
122
123
124
- bugs (bugs.archlinux.org)
- archweb
- patchwork
Sven-Hendrik Haase's avatar
Sven-Hendrik Haase committed
125
- projects (projects.archlinux.org)
Sven-Hendrik Haase's avatar
Sven-Hendrik Haase committed
126
127
128
129
130
131

### soyuz

#### Services
- build server (pkgbuild.com)
- releng
132
133
134
- sogrep
- /~user/ webhost
- irc bot (phrik)
Jelle van der Waa's avatar
Jelle van der Waa committed
135
136
137
- matrix
- docker images
- arch boxes (packer)
138

139
140
141
142
143
144
145
146
147
148
149
### dragon

#### Services
- build server (pkgbuild.com)
- sogrep

### state.cloud.archlinux.org

#### Services:
- postgres server for terraform state

150
151
152
153
### quassel.archlinux.org

#### Services:
- quassel core
154
155
156
157
158
159
160
161
162
163

## Ansible repo workflows

### Replace vault password and change vaulted passwords

 - Generate a new key and save it as ./new-vault-pw: `pwgen -s 64 1 > new-vault-pw`
 - `for i in $(ag ANSIBLE_VAULT -l); do ansible-vault rekey --new-vault-password-file new-vault-pw $i; done`
 - Change the key in misc/vault-password.gpg
 - `rm new-vault-pw`