Verified Commit f42fd92b authored by Frederik Schwan's avatar Frederik Schwan
Browse files

Merge wip-keyclaok into master

parent a9a6e58c
......@@ -9,6 +9,7 @@ It also contains git submodules so you have to run `git submodule update --init
Install these packages:
- terraform
- terraform-provider-keycloak
### Instructions
......@@ -58,13 +59,24 @@ This will take some time after which a new snapshot will have been created on th
#### Note about terraform
We use terraform to provision a part of the infrastructure on hcloud.
We use terraform in two ways:
1) To provision a part of the infrastructure on hcloud (and possibly other service providers in the future)
2) To declaratively configure applications
For both of these, we have set up a separate terraform script. The reason for that is that sadly terraform can't have
providers depend on other providers so we can't declaratively state that we want to configure software on a server which
itself needs to be provisioned first. Therefore, we use a two-stage process. Generally speaking, scenario 1) is configured in
`tf-stage1` and 2) is in `tf-stage2`. Maybe in the future, we can just have a single terraform script for everything
but for the time being, this is what we're stuck with.
The very first time you run terraform on your system, you'll have to init it:
terraform init -backend-config="conn_str=postgres://terraform:$(misc/get_key.py group_vars/all/vault_terraform.yml vault_terraform_db_password)@state.archlinux.org"
terraform init -backend-config="conn_str=postgres://terraform:$(../misc/get_key.py group_vars/all/vault_terraform.yml vault_terraform_db_password)@state.archlinux.org"
After making changes to the infrastructure in `archlinux.fg`, run
After making changes to the infrastructure in `tf-stage1/archlinux.fg`, run
cd tf-stage1
terraform plan
This will show you planned changes between the current infrastructure and the desired infrastructure.
......@@ -74,6 +86,9 @@ You can then run
to actually apply your changes.
The same applies to changed application configuration in which case you'd run
it inside of `tf-stage2` instead of `tf-stage1`.
We store terraform state on a special server that is the only hcloud server NOT
managed by terraform so that we do not run into a chicken-egg problem. The
state server is assumed to just exist so in an unlikely case where we have to
......@@ -193,10 +208,24 @@ The following steps should be used to update our managed servers:
#### Services:
- quassel core
## homedir.archlinux.org
### homedir.archlinux.org
#### Services:
- ~/user/ webhost
### accounts.archlinux.org
This server is /special/. It runs keycloak and is central to our unified Arch Linux account management world.
It has an Ansible playbook for the keycloak service but that only installs the package and starts it but it's configured via a secondary Terraform file only for keycloak `keycloak.tf`.
The reason for doing it this way is that Terraform support for Keycloak is much superior and it's declarative too which is great for making sure that no old config remains in the case of config changes.
So to set up this server from scratch, run:
- `terraform apply tf-first-stage`
- `terraform apply tf-second-stage`
#### Services:
- keycloak
## mirror.pkgbuild.com
......@@ -252,3 +281,8 @@ Example
Example
borg list borg@vostok.archlinux.org:/backup/homedir.archlinux.org::20191127-084357
## One-shots
A bunch of once-only admin task scripts can be found in `one-shots/`.
We try to minimize the amount of manual one-shot admin work we have to do but sometimes for some migrations it might be necessary to have such scripts.
......@@ -13,3 +13,4 @@ callback_whitelist = profile_tasks
[ssh_connection]
pipelining = True
scp_if_ssh = True
retries = 5
......@@ -14,6 +14,9 @@ arch_users:
ssh_key: aaron.pub
groups:
- dev
accounts:
name: ""
groups: []
aginiewicz:
name: "Andrzej Giniewicz"
ssh_key: aginiewicz.pub
......@@ -227,6 +230,7 @@ arch_users:
- name: foutrelis_buildhost.pub
hosts:
- dragon.archlinux.org
- sgp.mirror.pkgbuild.com
groups:
- dev
- tu
......
$ANSIBLE_VAULT;1.1;AES256
38386666643233373639363835396530396162636562393531373566623531346131613739386637
3238633664333561343139663665663537336633303036610a386436626330646262333130626539
35323033316530616437326630393632646630363664303765636362353063653232373233353862
6135346434373562350a376133626564643138386631366331333261376239636236343630303762
64633431326164386332396238363332303965363666663636373465626563373535343534633232
64313366623238656663383066613030633861333239623964633830323535363666303637663864
35366131663337663534393863313634376433303935363733366234326639613034363465366538
37343866306439336165666266323034666331616365333839343436306632643339386532623566
34373165323664663365663237323361643137616165666130333537653862633730646637656635
30656434366431353863333961353232653538616663313331343932363163353833633332383735
35313531333839366132343038326230643235663133373334393562393435333136363534383134
37643431666631666564383533366235313563636438666464343738376431643463373134346530
61613461326137333162346330323232333562306638353332386538386465396238
---
filesystem: btrfs
zabbix_agent_templates:
- Template OS Linux
- Template App Borg Backup
- Template App HTTP Service
- Template App HTTPS Service
- Template App Nginx
- Template App SSH Service
- Template App PostgreSQL
......@@ -38,6 +38,8 @@ bbs.archlinux.org
homedir.archlinux.org
bugs.archlinux.org
aur-dev.archlinux.org
gitlab.archlinux.org
accounts.archlinux.org
[borg_hosts]
vostok.archlinux.org
......@@ -66,6 +68,7 @@ bugs.archlinux.org
[buildservers]
dragon.archlinux.org
sgp.mirror.pkgbuild.com
[gitlab_runners]
runner1.archlinux.org
......
This directory contains a bunch of one-off scripts which might be modified ad-hoc in some ways.
We keep them around for documentation reasons.
---
arch_groups:
- dev
- tu
- devops
- fellows
- multilib
- archboxes-sudo
- docker-image-sudo
arch_users:
# aaron:
# name: "Aaron Griffin"
# ssh_key: aaron.pub
# groups:
# - dev
# aginiewicz:
# name: "Andrzej Giniewicz"
# ssh_key: aginiewicz.pub
# groups:
# - tu
# ainola:
# name: "Brett Cornwall"
# ssh_key: ainola.pub
# groups:
# - tu
# alad:
# name: "Alad Wenter"
# ssh_key: alad.pub
# groups:
# - tu
# allan:
# name: "Allan McRae"
# ssh_key: allan.pub
# groups:
# - dev
# - multilib
# - tu
# alucryd:
# name: "Maxime Gauduin"
# ssh_key: alucryd.pub
# groups:
# - dev
# - tu
# - multilib
# anatolik:
# name: "Anatol Pomozov"
# ssh_key: anatolik.pub
# groups:
# - dev
# - tu
# - multilib
# andrea:
# name: "Andrea Scarpino"
# ssh_key: andrea.pub
# groups: []
# andrew:
# name: "Andrew Gregory"
# ssh_key: andrew.pub
# groups:
# - dev
# andrewsc:
# name: "Andrew Crerar"
# ssh_key: andrewsc.pub
# groups:
# - tu
# anthraxx:
# name: "Levente Polyak"
# ssh_key: anthraxx.pub
# shell: /bin/zsh
# groups:
# - dev
# - devops
# - tu
# - multilib
# andyrtr:
# name: "Andreas Radke"
# ssh_key: andyrtr.pub
# groups:
# - dev
# - tu
# arcanis:
# name: "Evgeniy Alekseev"
# ssh_key: arcanis.pub
# groups:
# - tu
# archange:
# name: "Bruno Pagani"
# ssh_key: archange.pub
# shell: /bin/zsh
# groups:
# - tu
# - multilib
# arodseth:
# name: "Alexander Rødseth"
# ssh_key: arodseth.pub
# groups:
# - tu
# - multilib
# arojas:
# name: "Antonio Rojas"
# ssh_key: arojas.pub
# groups:
# - dev
# - tu
# - multilib
# aur-notify:
# name: ""
# groups: []
# bgyorgy:
# name: "Balló György"
# ssh_key: bgyorgy.pub
# groups:
# - tu
# bisson:
# name: "Gaëtan Bisson"
# ssh_key: bisson.pub
# groups:
# - dev
# - tu
# bluewind:
# name: "Florian Pritz"
# ssh_key: bluewind.pub
# shell: /bin/zsh
# groups:
# - dev
# - devops
# - tu
# - multilib
# bpiotrowski:
# name: "Bartłomiej Piotrowski"
# ssh_key: bpiotrowski.pub
# groups:
# - dev
# - devops
# - tu
# - multilib
# cbehan:
# name: "Connor Behan"
# ssh_key: cbehan.pub
# groups:
# - tu
# cesura:
# name: "Brad Fanella"
# ssh_key: cesura.pub
# groups:
# - tu
# coderobe:
# name: "Robin Broda"
# ssh_key: coderobe.pub
# groups:
# - tu
# daurnimator:
# name: "Daurnimator"
# ssh_key: daurnimator.pub
# groups:
# - tu
# dbermond:
# name: "Daniel Bermond"
# ssh_key: dbermond.pub
# groups:
# - tu
# demize:
# name: "Johannes Löthberg"
# ssh_key: demize.pub
# shell: /bin/zsh
# groups:
# - dev
# - tu
# - multilib
# diabonas:
# name: "Jonas Witschel"
# ssh_key: diabonas.pub
# groups:
# - tu
# donate:
# name: ""
# groups: []
# dreisner:
# name: "Dave Reisner"
# ssh_key: dreisner.pub
# groups:
# - dev
# - multilib
# - tu
# dvzrv:
# name: "David Runge"
# ssh_key: dvzrv.pub
# groups:
# - dev
# - multilib
# - tu
# eschwartz:
# name: "Eli Schwartz"
# ssh_key: eschwartz.pub
# groups:
# - tu
# - multilib
# escondida:
# name: "Ivy Foster"
# ssh_key: escondida.pub
# groups:
# - tu
# eworm:
# name: "Christian Hesse"
# ssh_key: eworm.pub
# shell: /bin/zsh
# groups:
# - dev
# - tu
# - multilib
# farseerfc:
# name: "Jiachen Yang"
# ssh_key: farseerfc.pub
# groups:
# - tu
# felixonmars:
# name: "Felix Yan"
# ssh_key: felixonmars.pub
# groups:
# - dev
# - tu
# - multilib
# ffy00:
# name: "Filipe Laíns"
# ssh_key: ffy00.pub
# shell: /bin/bash
# groups:
# - tu
# foutrelis:
# name: "Evangelos Foutras"
# ssh_key: foutrelis.pub
# additional_ssh_keys:
# - name: foutrelis_buildhost.pub
# hosts:
# - dragon.archlinux.org
# - sgp.mirror.pkgbuild.com
# groups:
# - dev
# - devops
# - tu
# - multilib
# foxboron:
# name: "Morten Linderud"
# ssh_key: foxboron.pub
# groups:
# - tu
# foxxx0:
# name: "Thore Bödecker"
# ssh_key: foxxx0.pub
# shell: /bin/zsh
# groups:
# - tu
# fukawi2:
# name: "Phillip Smith"
# ssh_key: fukawi2.pub
# groups:
# - devops
# giovanni:
# name: ""
# ssh_key: giovanni.pub
# groups:
# - dev
# - multilib
# gitlab:
# name: ""
# groups: []
# grazzolini:
# name: "Giancarlo Razzolini"
# ssh_key: grazzolini.pub
# groups:
# - dev
# - devops
# - multilib
# - tu
# heftig:
# name: "Jan Steffens"
# ssh_key: heftig.pub
# additional_ssh_keys:
# - name: heftig_work.pub
# hosts:
# - dragon.archlinux.org
# - name: heftig_dragon.pub
# hosts:
# - homedir.archlinux.org
# groups:
# - dev
# - devops
# - tu
# - multilib
# idevolder:
# name: "Ike Devolder"
# ssh_key: idevolder.pub
# groups:
# - tu
jelle:
name: "Jelle van der Waa"
ssh_key: jelle.pub
groups:
- dev
- devops
- tu
- multilib
# jgc:
# name: "Jan de Groot"
# ssh_key: jgc.pub
# groups:
# - dev
# - multilib
# - tu
# jleclanche:
# name: "Jerome Leclanche"
# ssh_key: jleclanche.pub
# shell: /bin/zsh
# groups:
# - tu
# jlichtblau:
# name: "Jaroslav Lichtblau"
# ssh_key: jlichtblau.pub
# groups:
# - tu
# jouke:
# name: "Jouke Witteveen"
# ssh_key: jouke.pub
# groups:
# - ""
# jsteel:
# name: "Jonathan Steel"
# ssh_key: jsteel.pub
# groups:
# - tu
# juergen:
# name: "Jürgen Hötzel"
# ssh_key: juergen.pub
# groups:
# - dev
# - multilib
# - tu
# kgizdov:
# name: "Konstantin Gizdov"
# ssh_key: kgizdov.pub
# groups:
# - tu
# kkeen:
# name: "Kyle Keen"
# ssh_key: kkeen.pub
# groups:
# - tu
# - multilib
# lcarlier:
# name: "Laurent Carlier"
# ssh_key: lcarlier.pub
# groups:
# - dev
# - tu
# - multilib
# lfleischer:
# name: "Lukas Fleischer"
# ssh_key: lfleischer.pub
# shell: /bin/zsh
# groups:
# - dev
# - tu
# - multilib
# maximbaz:
# name: "Maxim Baz"
# ssh_key: maximbaz.pub
# groups:
# - tu
# mtorromeo:
# name: "Massimiliano Torromeo"
# ssh_key: mtorromeo.pub
# groups:
# - tu
# muflone:
# name: "Fabio Castelli"
# ssh_key: muflone.pub
# groups:
# - tu
# nicohood:
# name: "NicoHood"
# ssh_key: nicohood.pub
# groups:
# - tu
# pierre:
# name: "Pierre Schmitz"
# ssh_key: pierre.pub
# groups:
# - dev
# - multilib
# - tu
# polyzen:
# name: "Daniel M. Capella"
# ssh_key: polyzen.pub
# groups:
# - tu
# remy:
# name: "Rémy Oudompheng"
# ssh_key: remy.pub
# groups:
# - dev
# - tu
# ronald:
# name: "Ronald van Haren"
# ssh_key: ronald.pub
# groups:
# - dev
# - tu
# sangy:
# name: "Santiago Torres-Arias"
# ssh_key: sangy.pub
# groups:
# - tu
# - docker-image-sudo
# schiv:
# name: "Ray Rashif"
# ssh_key: schiv.pub
# groups:
# - dev
# - tu
# - multilib
# schuay:
# name: "Jakob Gruber"
# ssh_key: schuay.pub
# groups:
# - tu
# - multilib
# scimmia:
# name: "Doug Newgard"
# ssh_key: scimmia.pub
# groups: []
# morganamilo:
# name: "Morgan Adamiec"
# ssh_key: morganamilo.pub
# groups: []
# seblu:
# name: "Sébastien Luttringer"
# ssh_key: seblu.pub
# shell: /bin/zsh
# groups:
# - dev
# - tu
# - multilib
# shibumi:
# name: "Christian Rebischke"
# ssh_key: shibumi.pub
# shell: /bin/zsh
# groups:
# - tu
# - archboxes-sudo
# kpcyrd:
# name: "Kpcyrd"
# ssh_key: kpcyrd.pub
# groups:
# - tu
# spupykin:
# name: "Sergej Pupykin"
# ssh_key: spupykin.pub
# groups:
# - tu
# - multilib
# svenstaro:
# name: "Sven-Hendrik Haase"
# ssh_key: svenstaro.pub
# groups:
# - dev
# - devops
# - tu
# - multilib
# tensor5:
# name: "Nicola Squartini"
# ssh_key: tensor5.pub
# groups:
# - tu
# thomas:
# name: "Thomas Bächler"
# ssh_key: thomas.pub
# groups:
# - dev
# - multilib
# tpowa:
# name: "Tobias Powalowski"
# ssh_key: tpowa.pub