'systemctl reload' hangs

Hello folks,

I just downloaded https://gitlab.archlinux.org/archlinux/arch-boxes/-/jobs/109086/artifacts/raw/output/Arch-Linux-x86_64-cloudimg-20221208.109086.qcow2 and imported the image file on my promox cluster. After starting the vm, every run of systemctl daemon-reload (also invoked by i.e. pacman -S <whatever-pkg-including-a-service-file>) hangs for about 1 or 2 minutes (see screenshot).

Has anyone else seen this behaviour? Am i missing something important?

I also ran strace systemctl daemon-reload. The hang is happening in line 914/915 of the log output (see systemctl-daemon-reload.txt)

Any help would be appreciated, thank you in advance ;)

Can you check the status of: systemd-time-wait-sync.service, time-sync.target and timers.target?

It looks like these services/targets are enabled and active.

One thing that caught my eye is a failed service (systemd-firstboot), but there don't seem to be any journal/log entry about it…

@klausenbusk
Hm, this keeps being strange (and kinda random). If for example i run systemctl status time-sync.target directly after booting the vm, then different hangs/timeouts happen.

Well, looks like i found the culprit.

• https://bbs.archlinux.org/viewtopic.php?pid=2071688#p2071688

After removing the following packages the 60s(+) waits are gone:

• netplan
• cloud-init

Is there a way to use cloud-init here without netplan on arch linux? I know that cloud-init can be run with plain systemd-networkd (we're using it this way w/ our customized debian vms). Why would anybody include yet another abstraction layer, but puh, hey, that's not my war…

Annoying bug :/, could you report it upstream?

arch-boxes predates the systemd-networkd render. Could you try removing netplan from a not-booted cloud image with pacman -Rdd netplan and see if the systemd-networkd render works? Perhaps I can then convince the cloud-init maintainer to make it a optional dependency.

arch-boxes predates the systemd-networkd render. Could you try removing netplan from a not-booted cloud image with pacman -Rdd netplan and see if the systemd-networkd render works?

As stated in #156 (comment 83818), this seems to work fine here. Now a file per interface is generated in /etc/systemd/network, so works as expected.

Perhaps I can then convince the cloud-init maintainer to make it a optional dependency.

Yes please. In my optinion netplan is a useless dependency, so making it optional would be great ;)

Upstream issue here: https://bugs.launchpad.net/netplan/+bug/1999178, see also the linked systemd issue for more details.

@klausenbusk i am working with @boris64 and was able to create an image without netplan, as you suggested.

as i am using proxmox (pve 7.3-3 for that matter), i was only able to check a subset of available cloud-init networking settings. see these screenshots for clarification:

i can confirm that:

• search domain in /etc/resolv.conf ist correctly set to "thisis.mydomin" (yep, it's a typo).
• the requested ipv4 address, mask and gateway route have been set.
• the requested ipv6 address, mask and gateway route have been set.
• resolvectl also lists the fd00:1023::5 (and ::1) dns servers.

cloud-init.service, cloud-init-local.service, systemd-network-generator.service are all enabled and ran without throwing an error.

the dreaded 60s wait for the timeout is gone, too. systemctl daemon-reload now completes in less than 250ms.

netplan is now a optional dependency: https://github.com/archlinux/svntogit-community/commit/a8d7778e56b6c43eb833edbcad45543b03a59373. The next release on the mirrors will have this (1th January) or the newest nightly on GitLab.

Thanks for testing cloud-init without netplan :)

closed

For older systems which are using netplan after upgrading to cloud-init 22.4.2-2 which netplan is an optional dependency, this command can be used to switch to systemd-networkd.

sudo pacman -Rnsc netplan --noconfirm && sudo rm -rf /var/lib/cloud/ && sudo reboot

Removing /var/lib/cloud leads to cloud-init using systemd-networkd to configure networking again.

@klausenbusk after remove netplan and install the latest cloud-init 22.4.2-2, re-generating the new profile and the new network config file for systemd-networkd is something like

root@arch /etc/systemd/network # cat 10-cloud-init-eth0.network 
[Address]
Address=192.0.2.2/24

[Address]
Address=2001:db8::2/48

[Match]
MACAddress=ea:3a:ec:46:79:8e
Name=eth0

[Network]
DHCP=no
DNS=8.8.8.8 8.8.4.4

[Route]
Gateway=192.0.2.1
Gateway=2001:db8::1

Only IPv6 is working, IPv4 is not working, there's no default route for IPv4, we need to change to

[Route]
Gateway=192.0.2.1

[Route]
Gateway=2001:db8::1

Don't know if this is a bug from cloud-init or systemd-networkd:

https://bugs.launchpad.net/cloud-init/+bug/1973724?comments=all https://github.com/canonical/cloud-init/pull/1685