Mirror hardening - complementing RFC 29

Complements RFC !29 (merged). The mirror team is proposing new hardening aspects to the current mirror infrastructure as there were none implemented in the past.

added 1 commit

• 3a0adc1b - Rename rfc file

Compare with previous version

added 1 commit

• 0efa67b2 - Update date and link

Compare with previous version

Business days or calendar days?

Calendar days should be preferred as it's hard to account for holidays, even what calendar the recipient is using.

But lets either increase this or add that there's leeway before actions are taken @pitastrudl & @wahrwolf

I would like to see the OpenSSL version defined, as a way of future-proofing the RFC against new algorithms that might be supported by OpenSSL in the future.

I have mentioned this wrt the opened mirrorctl merge request already: I don't see the need for OpenSSL in this context at all and I am beginning to worry about this being brought up again.

We already have an OpenPGP Public Key Infrastructure. I see no reason for us to introduce yet another one, especially given that it doesn't do authentication.

As such I am very much against using OpenSSL for this purpose.

changed this line in version 4 of the diff

I think e08fc32b resolves this. OpenSSL was simply an easy way for operators to generate and maintain keys. But if OpenPGP is the preferred way from our side then we'll go with that. But for the sake of simplicity and the discussion, I removed tooling and specifics, we simply want to be able to verify a signature with this RFC.

This doesn't make much sense without specifying which key types and lengths are acceptable, or referring to another document which does that.

This makes it harder to host a mirror and honestly what does it bring us? Encrypted communication sure, but it doesn't give us much more trust that person X is person Y.

changed this line in version 5 of the diff

It does tho? And we would only require a OpenPGP signature, not encryption. Because if the domain ownership is transferred, especially in a malicious way, the new owner will have the ability to setup e-mail with correct DNS/DKIM/etc and it would look good on paper. Except, hopefully, the PGP key should not be as easily forged. And gaining access to the GitLab account should also not be trivial (unless 2FA was not setup, which we should require).

Changed the phrasing according to @dvzrv's suggestion.

I've read through this again and I still think this is a bad idea where the consequences is us loosing most of the servers in our mirror pool. And for a consequence like that I'd expect a bit more thorough arguments around the downsides of such a proposal.

I can't put my finger on it but there is something amiss with the power dynamic here. The mirrors are doing us a favour, not the other way around. Most people host several mirrors and having requirements like "reponding within 2(!) days" and special TLS requirements is going to make it a lot harder to retain mirrors.

Requiring legacy stuff like "GPG messages" is archaic considering Gitlab works perfectly fine, and this proposal still doesn't detail any threat model nor explains why pacman isn't enough.

Generally my opinion is that all of this is severely misaligned with what mirrors actually are; they serve files and that is it. Unless there is a general push from several distributions for something similar this is just going to make us an extremely unpopular distribution to support.

Considering none of the suggested improvements are going into pacman I expect them to have limited value as well.

TL;DR: This proposal needs to be realistic and not expect mirrors to be beholden to whatever Arch Linux says.

the consequences is us loosing most of the servers in our mirror pool

Based on what data would we loose mirrors? We extracted statistics to make sure this was not the case so I'd like - for the sake of this particular argument - data to be presented so we can talk about it and find concrete solutions.

The mirrors are doing us a favour, not the other way around.

We're in agreement, and it's worth repeating as many times as needed. Mirror operators are on a volunteer basis, we've kept this in mind throughout this whole RFC process - and we're giving leeway, exemptions, help and guidance, based it on statistical data from the mirror database and a lot of other variables that I cannot recall from the top of my head at this point.

But whatever concerns there are, we should present and discuss before accepting anything as fact, obviously.

having requirements like "reponding within 2(!) days" and special TLS requirements is going to make it a lot harder to retain mirrors.

I agree, if the values are too strict - lets change them. But a value is needed, because we do have multiple instances where we don't get responses and there's currently no way for mirror administrators to feel okay about their choice of "when has enough time passed, that I can consider my following action justified" - as it's all guesswork and estimates at the moment.

I'm okay with writing "responding within 6 months" if that feels better, but then we'll have "dead mirrors" for 6 months. So we need to agree on a value of not-so-ridiculous-time-for-volunteers but also not a value not-so-long-that-mirror-admins-get-a-backlog-of-months. 2 days was a starting point, let's figure it out together!

Requiring legacy stuff like "GPG messages" is archaic considering Gitlab works perfectly fine, and this proposal still doesn't detail any threat model nor explains why pacman isn't enough.

We tried moving away from PGP more or less everywhere, but it keeps being the preferred method and in e-mail communication it's more or less the only common built-in method for signing e-mails.

they serve files and that is it.

Currently, yes, but we're trying to push for a change that enables them to keep doing that - but in a more secure manner. Not just be a dead-drop where we hope all 100% of our users follow written guidelines to the letter. See my other comment of why this is important here.

this is just going to make us an extremely unpopular distribution to support.

Only if we force unreasonable changes, which we don't want to. Especially without giving guidance, easy barrier to entry options, and we go outside of industry standards. I sincerely hope that us expecting HTTPS is not what would make us unreasonable? in 2024 where TLS certificates are easier to get a hold of than anything else in terms of web hosting.

TL;DR: This proposal needs to be realistic and not expect mirrors to be beholden to whatever Arch Linux says.

Could we outline what is unrealistic? I know you started by saying you can't put your finger on it, but perhaps we can start putting actual fingers on it so we can make the PR not unrealistic.

Based on what data would we loose mirrors? We extracted statistics to make sure this was not the case so I'd like - for the sake of this particular argument - data to be presented so we can talk about it and find concrete solutions.

What data could you possibly extract to make a case where mirrors wouldn't stop distributing Arch because of new requirements? Unless you did interviews with existing Tier 1 mirrors to figure out what their stance on this is.

There is no empirical data to represent here. It's a human factor of "this is too much effort".

We tried moving away from PGP more or less everywhere, but it keeps being the preferred method and in e-mail communication it's more or less the only common built-in method for signing e-mails.

You don't need either. Gitlab is perfectly fine for this.

Currently, yes, but we're trying to push for a change that enables them to keep doing that - but in a more secure manner. Not just be a dead-drop where we hope all 100% of our users follow written guidelines to the letter. See my other comment of why this is important here.

This is secure enough. The proposal still fails to describe the threat model we are defending against and why pacman itself is not enough.

Only if we force unreasonable changes, which we don't want to. Especially without giving guidance, easy barrier to entry options, and we go outside of industry standards. I sincerely hope that us expecting HTTPS is not what would make us unreasonable? in 2024 where TLS certificates are easier to get a hold of than anything else in terms of web hosting.

Then how have you figured out what unreasonable and reasonable changes are? HTTPS is the reasonable part, beyond hard requiring modern TLS1.3 because mirrors need to support distros with legacy TLS versions.

Everything else is the unreasonable part.

Could we outline what is unrealistic? I know you started by saying you can't put your finger on it, but perhaps we can start putting actual fingers on it so we can make the PR not unrealistic.

The proposal says "Before this proposal, there were a lack of security definitions and measures on mirrors within the Arch Linux community" but doesn't explain why. It seems to be in disagreement because of a poorly defined (if any) threat model.

Unless you did interviews with existing Tier 1 mirrors to figure out what their stance on this is.

This is not out of the question to be honest. Unless a survey is too much to ask as well.

We can rephrase things to say "recommendation" instead, but there are already requirements enacted. And as long as we can figure out a way to make this RFC balanced with equal effort of today - I think that would be the main goal at the moment.

Which is why we extracted data, to make sure that the things we can measure, is taken care of. For instance bandwidth limit, size requirements etc.

The "feeling of too much effort" is harder, the only thing we can do is put ourselves in the shoes of the mirror operators. And ask ourselves, "would this be too much for us as well?".

So lets eliminate "those feelings" from the RFC by rephrasing things, or cutting things out. But lets not do it "just because" before we've bounced some ideas around of:

1. What causes those kinds of concerns
2. Are the better options for those concerns
3. Would we lose less than 1% of mirrors - if so does the benefit outweigh the outcome (as an example of a realistic conversation that could occur, if we don't wanna loose a single mirror, then that's the choice we'll make)

Currently I don't even have a concept of what those concerns are tho, other than, there are concerns. So it's hard to have a discussion around it. Is TLS one such concern?

You don't need either. Gitlab is perfectly fine for this.

Unless they use e-mail to 'mirrors@archlinux.org` to create a ticket for changes to their mirrors, or e-mail us directly (sure, we could ask them to not do that i guess).

The proposal still fails to describe the threat model

I am pretty convinced we've described it here in the comment section tho. Would adding this comment in refined form help justify the threat model?

Everything else is the unreasonable part.

Everything is quite a lot heh, re-write the entire thing?

Are these topics not reasonable:

• "Active monitoring of tagged GitLab issues"
• (optional) “Service JSON manifest”
• "A new way of verifying (continued) ownership of a mirror domain is hereby proposed."
• "Provide (rootless) container based images for different tiers" (to make it easier for mirror operators to get going)

I can agree that the how can be considered unreasonable, but "everything" being unreasonable is quite a lot to deal with to be honest hehe.

Would we lose less than 1% of mirrors

Simply judging from the comments, my hunch is that arch linux can lose 30%-50% T1 mirrors, much more for T2.

Solved in 57acf72a

changed this line in version 6 of the diff

Solved in 57acf72a

changed this line in version 6 of the diff

Solved in 57acf72a (not sure why I can't mark any of these as solved, so commenting individually)

changed this line in version 6 of the diff

The new requirements will first be enacted only on new mirrors.

What does this mean? How would decommissioning be done on mirrors that are not new/ do not exist? Is this relevant information? If so, maybe it should be at the beginning of the section.

The idea was to not force requirements on existing mirrors, as they had no chance to agree on the requirements. But we'll attempt to convert existing mirrors to the new requirements.

But we also want to give leeway wherever needed, at the mirror admins discretion. Considering some mirror regions have different pre-existing conditions such as average speed in that region etc.

Perhaps we can emphasise this better, as it becomes more and more clear that the message we want to get across can still be interpreted in a variety of personal ways.

@wahrwolf & @pitastrudl - note for next meeting.

changed this line in version 7 of the diff

What does this proper tooling do? How does it improve on the status quo? This needs to be more detailed.

changed this line in version 7 of the diff

changed this line in version 7 of the diff

changed this line in version 7 of the diff

Wouldn't rootless imply not using docker?

Providing container images is orthogonal to using management tools like docker-compose and providing config files for those. While both might be useful in practice, they should be described as separate things. Or even better, the commitment should be generic without needlessly referring to specific tools. The motivation section even promises to define "Arch Linux commitment towards community in improving security goals".

I'm wondering whether what you are referring to as "domain hijacking" is actually guarded against at all, following the linked to Wikipedia definition:

Domain hijacking or domain theft is the act of changing the registration of a domain name without the permission of its original registrant, or by abuse of privileges on domain hosting and registrar software systems.

IIUC, what is proposed in this section is tooling, that is supposed to (automatically) sign a token file, which is used to evaluate when a given host has last synced. This means, that mirrorctl has to be used in an automated fashion on that host (as a post action to syncing). In consequence an attacker has access to these key material used to sign the token file and can sign anything they want.

When it comes to actual "domain hijacking" I think what you would want to have is a proof in the domain records that you can validate or something like that.

As is, this section appears to conflate two topics.

From what I know, our largest issue with mirror security (or rather the payloads on those mirrors) at this point in time is, that we do not sign the repository sync databases.

How does cryptographic proof of the sync process improve on this, if an attacker can alter the repository sync database after a (signed) sync? More specifically: Which attack vector does this cryptographic proof improve upon?

changed this line in version 7 of the diff

IIUC this is in fact proof-of-sync.

The RFC has taken this into account and it produces no unwanted results other than old mirrors not supporting new features.

Doesn't !35 (diffs) argue for decommissioning of mirrors not in line with these specs?