Look into Kata Containers for GitLab runners

We can't easily build arch-boxes in a safe way. We need access to either a loop device (which isn't safe) or use libguestfs (require FUSE + unprivileged user namespaces + qemu).

Either solutions is messy. It would be easier if could run docker with --privileged and be done with it. Kata Containers could probably allow us to do that.

mentioned in commit klausenbusk/arch-boxes@c251d45a

mentioned in commit klausenbusk/arch-boxes@9c98aaa0

mentioned in commit klausenbusk/arch-boxes@26cd9cc5

added hinthelp-wanted scopeenhancement labels

From IRC:

22:44 <zer0def> the tl;dr: of Kata being unfeasible is that due to targeting OCI behavior, it doesn't run an inited container and there's no udev to properly manage a separate /dev, which increases complexity for existing build scripts in arch-boxes (and this logic probably extends to archiso, as well)
22:46 <zer0def> you can try it out from https://github.com/zer0def/arch-boxes/commit/b1eba8251f5728c1ab9a22e67a267d68d7437534 , but I'm pretty sure I'm offsetting something wrong, because it results in an unmountable rootfs :)
22:46 <wCPO> I added the tldr to the issue, thanks
22:48 <wCPO> zer0def: I had a quick look at the diff and I assume https://github.com/zer0def/arch-boxes/commit/b1eba8251f5728c1ab9a22e67a267d68d7437534#diff-1b6f6911aeedb9724e38374e733001202c0acc33b7abd59e702fcf9320b83a76R46 and https://github.com/zer0def/arch-boxes/commit/b1eba8251f5728c1ab9a22e67a267d68d7437534#diff-1b6f6911aeedb9724e38374e733001202c0acc33b7abd59e702fcf9320b83a76R53-R60 is due to the lack of udev?
22:50 <zer0def> as far as my current understanding stands, yes, though I have tried putting in `/usr/lib/systemd/systemd-udevd -d` in `before_script` to little avail
22:52 <zer0def> that's also why I've switched to ext4, since there apparently was a missing module for btrfs within the kernel running inside the VM runtime, which would require building a dedicated kernel for this, possibly with a dedicated rootfs image to boot these runtimes off of
22:53 <zer0def> but frankly, even the amalgamation of these changes is fugly

The gist of the situation is that using OCI/Docker containers with Kata still leaves us with a largely uninited system that doesn't populate internal /dev properly, forcing to do a lot more error-prone legwork to just get a build started.

The next best thing would be to try to work LXD VM-based custom runners into the process as hypervised system containers to provide a booted system for builds requiring kernel separation. The next best option after that would be libvirt runners, as Kristian eluded to previously in IRC.

That way, invasive tweaks to existing scripts can be minimized, while custom executors shoulder the responsibility of expected isolation.

Some quick notes from testing your custom-exec branch locally.

$ sudo pacman -S lxd gitlab-runner
$ sudo usermod -a -G lxd $USER
$ sudo systemctl start lxd
$ sudo lxd init
$ firewall-cmd --zone=trusted --change-interface=lxdbr0
$ firewall-cmd --direct --add-rule ipv4 filter FORWARD 0 -i lxdbr0 -s 10.204.79.1/24 -m comment --comment "generated by firewalld for LXD" -j ACCEPT
$ firewall-cmd --direct --add-rule ipv4 filter FORWARD 0 -o lxdbr0 -d 10.204.79.1/24 -m comment --comment "generated by firewalld for LXD" -j ACCEPT
$ firewall-cmd --direct --add-rule ipv4 nat POSTROUTING 0 -s 10.204.79.1/24 ! -d 10.204.79.1/24 -m comment --comment "generated by firewalld for LXD" -j MASQUERADE
$ newgrp lxd
$ gitlab-runner exec custom --builds-dir="$PWD/build" --cache-dir="$PWD/cache" --custom-prepare-exec "$PWD/gitlab-runner/lxd/prepare.sh" --custom-run-exec "$PWD/gitlab-runner/lxd/run.sh" --custom-cleanup-exec "$PWD/gitlab-runner/lxd/cleanup.sh" .build

It fails with though (kinda expected):

Getting source from Git repository
Fetching changes...
Initialized empty Git repository in /tmp/lxd/arch-boxes/build/project-0/.git/
Created fresh repository.
fatal: '/tmp/lxd/arch-boxes' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
Cleaning up file based variables
ERROR: Job failed: exit status 1
FATAL: exit status 1

It works: https://gitlab.archlinux.org/klausenbusk/arch-boxes/-/jobs/8937 and it is much faster (I ran it on my laptop so a lot of the time was spent on network download/upload).

MR opened: !139 (closed)

Managed to eventually get Kata working and the solution seems to overlap with a loop device detection race condition in Docker-in-LXD, stemming from bind-mounting /dev into the OCI container, instead of leveraging elevated priviledges to mount the same devtmpfs into it.

mentioned in merge request !139 (closed)

Eventually settled on something along these lines:

• ext4 image build
• btrfs image build requiring an additional patch to the kernel, as upstream doesn't ship with btrfs built-in

Just to shed some light on the context of proposed changes to arch-boxes:

• added a devtmpfs mount inside the Kata OCI container to help out with loopdev partition detection and settling through blockdev, two caveats here
  • with Docker-in-LXD-VM, bind-mounting /dev into the runc container from it's LXD VM host manifested as a race condition, causing "Operation not permitted" or "Permission denied" errors from kernel's loopdev code, as changes detected in the VM seemed to be late to propagate into the runc container
  • with Kata without devtmpfs, instead creating separate loopdevs related to partitions using offsetting on the image, the same kinds of errors occurred, but since there was no udev running to settle block devices from Kata VM's kernel, they were persistent; it also bears mention that bind-mounting /dev into a Kata container mounts it from the hypervisor running the VM, not the VM itself
• originally udev was launched as a daemon in before_script to help with populating /dev/disk into the container (from host kernel on runc, from Kata kernel on Kata), which in turn is used by grub-probe to properly populate root= with a UUID=<uuid4> entry, instead of root=/dev/loop0p2 (check /etc/grub.d/10_linux); on Kristian's feedback to omit launching udev, I've settled for injecting GRUB_DEVICE="/dev/disk/by-uuid/$(blkid -o value -s UUID ${LOOPDEV}p2)" into /etc/default/grub, as it achieves the same result
• due to upstream Kata kernel being built without btrfs support, building an image with btrfs as rootfs requires either a built-in kernel module or loadable one in the accompanying Kata boot image, which may later require exposing modules to the cgroup inside the Kata VM - that's the primary reason for proposing an ext4-based rootfs instead, as it doesn't additional require tinkering with upstream artifacts to achieve the result

Thanks for all your time, research and inputs on this matter @zer0def! Unfortunately it didn't work out in the end.

The whole "custom executor" idea is currently on hold due to maintenance concerns (discussed on the 2021-03-11 devops meeting) and I think using libvirt is more suitable (#283 (closed)), as we can use our own images and libvirt is for VMs and LXD is containers + VM magic.

closed