Skip to content
Snippets Groups Projects

Mirror hardening - complementing RFC 29

Closed pita strudl requested to merge torxed/rfcs:mirror-hardening into master
36 unresolved threads
+ 147
0
---
title: "Mirror Definition"
draft: true
---
=================
Mirror Definition
=================
- Date proposed: 2024-03-28
- RFC MR: https://gitlab.archlinux.org/archlinux/rfcs/-/merge_requests/35/
Summary
-------
Before this proposal, there was a lack of security definitions and measures on mirrors within the Arch Linux community. This RFC therefore outlines the requirements, definitions and guidelines surrounding not only hardening but also maintaining proper security of a mirror.
Motivation
----------
At the current moment we have no means of security within the mirror infrastructure. We can neither prove ownership, track ownership changes or flag breaches. Current security model relies solely on package validation on pacman. Other security aspects like data privacy or availability of the supply chain are not taken care of.
    • Let's set the record straight, what are the real security issues of using a mirror:

      • All packages are signed, so unless you break GPG or a signature you can't temper with this.
      • The repository databases are not signed, so an attacker can potentially "downgrade" a repo database and force users to install an outdated and vulnerable package
      • The repository database are not signed, so an attacker can potentially add a new package glib-foo.pkg.tar.zst which replaces glibc but then again they can't sign it with their own key as that is not in the keyring. So they have to tamper with archlinux-keyring.

      In conclusion, we should sign our repository databases for extra security. This RFC does not address that issue.

      Edited by Jelle van der Waa
      • +1, I'd also rather not move forward with this RFC.

        All this crypto-identity-protocol requires an agent on every mirror and could be replaced with a "does the mirror currently serve the .db files we expect them to" monitoring script that automatically removes defunct mirrors from the official mirror list (which would be a more effective security control than what this RFC proposes).

        The protocol described in the RFC is trying to "solve domain-hijacking", but the idea of domain-hijacking doesn't really apply in a system that is open-submission.

      • Please register or sign in to reply
Please register or sign in to reply
This RFC aims to introduce security awareness where there is none yet. We are fully aware that some attack vectors still remain available. The security measures would mostly if not all be supported by proper tooling like `mirrorctl`_.
The RFC introduces the concept of mirror ownerships and a process of validation by means of cryptographic signature. We also discuss the notion of having sane security defaults.
This excludes ISOs which are hosted on the mirrors.
Please register or sign in to reply
This RFC complements `RFC 0029`_.
Some points following in this discussion:
- Defining hardened mirror minimum requirements
- Defining a validation of ownership of the mirror domain
- Defining Arch Linux commitment towards community in improving security goals
- Proposal of tooling to deal with above changes
What will happen with existing mirrors
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Mirrors will be allowed a certain grace period until they might be considered for the decommission process. These criteria changes depending on the tier levels the mirror might be.
Depending on your mirror tier and mirror configuration, your mirror might be lowered or upgraded on the new tiering scheme. The transition period will be adequate.
    • I understand "lowering" in the tier scheme due to insufficient security configuration, but how would "upgrading" a mirror based on stellar security work? There are other requirements for upgrading and one of them is the consent of the owner.

Please register or sign in to reply
Timeline
--------
| 3 months notice period of change | 3 months migration/grace period | 2 months clean-up phase |
Suggested start date for these changes is 2024-05-01.
Please register or sign in to reply
This would mean that the clean up phase would start at somewhere around December ideally.
Specification
-------------
Decommission procedure
^^^^^^^^^^^^^^^^^^^^^^
The decommission of a mirror should be announced with proper tooling. In addition to the described process in `RFC 0029`_, the decommission process would also start if the signature checks would fail repeatedly.
- Tier 1 and 2 mirrors will have 6 months to secure their servers, which can be extended if the procedure might take a longer time.
- Tier 3 mirrors will have 12 months to transition to the new specification.
In case of missing the grace period, additional warnings will be sent out which are then followed by removal from the official mirror list. Tier1 and Tier2 mirrors that don't follow the spec within the grace period will be downgraded to one level below after two attempts of contact without response over a period of two months. The new requirements will first be enacted only on **new** mirrors.
    • The new requirements will first be enacted only on new mirrors.

      What does this mean? How would decommissioning be done on mirrors that are not new/ do not exist? Is this relevant information? If so, maybe it should be at the beginning of the section.

      • The idea was to not force requirements on existing mirrors, as they had no chance to agree on the requirements. But we'll attempt to convert existing mirrors to the new requirements.

        But we also want to give leeway wherever needed, at the mirror admins discretion. Considering some mirror regions have different pre-existing conditions such as average speed in that region etc.

        Perhaps we can emphasise this better, as it becomes more and more clear that the message we want to get across can still be interpreted in a variety of personal ways.

        @wahrwolf & @pitastrudl - note for next meeting.

        Edited by Anton Hvornum
      • Please register or sign in to reply
Please register or sign in to reply
Recommendation and requirements for different mirror types
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
For mirror definitions please refer to `RFC 0029`_.
Tier 1 Requirements
- E-mail communication should be OpenPGP signed, so that the authenticity of messages can be cryptographically verified
    • Where is the web of trust? You have to trust an admin at some point? But maybe a better question is, why do we require this, what benefit does this give to Arch?

Please register or sign in to reply
- Active monitoring of tagged GitLab issues (initial response within 1-2 days)
Please register or sign in to reply
- HTTPS TLS1.2+
- Signed domain+lastupdate
Please register or sign in to reply
Tier 2 Requirements
- HTTPS TLS1.2+
Tier 2 Recommendations
- Signed ``domain``+``lastupdate``
- “Service JSON manifest”
Please register or sign in to reply
Tier 3 Requirements
- HTTPS TLS1.2+
Validation of Ownership
^^^^^^^^^^^^^^^^^^^^^^^
A new way of verifying *(continued)* ownership of a mirror domain is hereby proposed. This is primarily to combat `domain hijacking`_.
    • I'm wondering whether what you are referring to as "domain hijacking" is actually guarded against at all, following the linked to Wikipedia definition:

      Domain hijacking or domain theft is the act of changing the registration of a domain name without the permission of its original registrant, or by abuse of privileges on domain hosting and registrar software systems.

      IIUC, what is proposed in this section is tooling, that is supposed to (automatically) sign a token file, which is used to evaluate when a given host has last synced. This means, that mirrorctl has to be used in an automated fashion on that host (as a post action to syncing). In consequence an attacker has access to these key material used to sign the token file and can sign anything they want.

      When it comes to actual "domain hijacking" I think what you would want to have is a proof in the domain records that you can validate or something like that.

      As is, this section appears to conflate two topics.

Please register or sign in to reply
The mirror operator(s) must first submit a valid identifier or public component their `GitLab`_ account or in mirror TOML definition. After the key is submitted the proposed workflow looks like this:
1. With the private key sign the contents of ``updatetime`` appended with a semicolon ``;`` and the `FQDN`_ of the mirrors domain, example: `1700471141;ftp.lysator.liu.se`
    • From what I know, our largest issue with mirror security (or rather the payloads on those mirrors) at this point in time is, that we do not sign the repository sync databases.

      How does cryptographic proof of the sync process improve on this, if an attacker can alter the repository sync database after a (signed) sync? More specifically: Which attack vector does this cryptographic proof improve upon?

Please register or sign in to reply
2. Arch Linux then validates the mirror continuously
Mirrors are then automatically hidden if signatures or content is considered invalid. This can also be managed via ``mirrorctl``.
Arch Linux commitment
^^^^^^^^^^^^^^^^^^^^^
- Provide *(rootless)* container based images for different tiers *(with docker-compose)*
    • Wouldn't rootless imply not using docker?

      • Providing container images is orthogonal to using management tools like docker-compose and providing config files for those. While both might be useful in practice, they should be described as separate things. Or even better, the commitment should be generic without needlessly referring to specific tools. The motivation section even promises to define "Arch Linux commitment towards community in improving security goals".

      • Please register or sign in to reply
Please register or sign in to reply
Mirror Tooling
^^^^^^^^^^^^^^
To facilitate the changes outlined in this RFC, tooling would not only be beneficial but crucial for managing things going forward. To preface, all features workflows of ``mirrorctl`` should also be possible to do manually from the mirror administrators.
This RFC in combination with `RFC 0029`_ proposes a tool called ``mirrorctl`` which should have at least the following features:
- Handle signing of proof-of-domain-ownership
Please register or sign in to reply
Container images for setting up a mirror should be separated into the different responsibilities. And should if possible be rootless, for instance via `podman`_.
Please register or sign in to reply
Drawbacks
---------
The potential added workload should be offset by better tooling to make mirror operators life easier from a security standpoint.
The main drawback would be that old mirrors might not updated to the proposed changes. The RFC has taken this into account and it produces no unwanted results other than old mirrors not supporting new features.
    • The RFC has taken this into account and it produces no unwanted results other than old mirrors not supporting new features.

      Doesn't !35 (diffs) argue for decommissioning of mirrors not in line with these specs? :thinking:

Please register or sign in to reply
It would require some more maintenance from Arch Linux staff, specifically the devops team. However, the added workload is considered to be less than or equal to that of the current `mirror administrators`_.
Please register or sign in to reply
Unresolved Questions
--------------------
No outstanding questions, see `RFC 0029`_.
Alternatives Considered
-----------------------
Alternative discussions was not mentioned in the `Arch Summit 2023`_ as the conclusion was that this is a good step forward.
.. _RFC 0029: https://gitlab.archlinux.org/archlinux/rfcs/-/merge_requests/29
.. _mirror administrators: https://gitlab.archlinux.org/groups/archlinux/teams/mirror-administrator/-/group_members
.. _GitLab: https://gitlab.archlinux.org/
.. _domain hijacking: https://en.wikipedia.org/wiki/Domain_hijacking
.. _FQDN: https://en.wikipedia.org/wiki/Fully_qualified_domain_name
.. _pacman: https://wiki.archlinux.org/title/Pacman
.. _Arch Summit 2023: https://md.archlinux.org/5I61f0yFSmysBkV8QQFU-g
.. _podman: https://wiki.archlinux.org/title/Podman
.. _mirrorctl: https://gitlab.archlinux.org/archlinux/mirrorctl
\ No newline at end of file
Loading