These are managed services and Ansible doesn't run on them. It got boring writing 'all,!rsync_net,!hetzner_storageboxes' in playbooks and ad-hoc commands, so remove these borg hosts from our inventory.
In the process to accomplish the above, I refactored the roles that set up our borg hosts to run on localhost; seems to work well and should prove more flexible if we were to get another storage box.