Reflector not fetching mirrors

Description:

Reflector seems to have problems fetching mirrors. I only get one mirror from china. Maybe it has something to do with completion rate, but inside the EndeavourOS forum some people also have this problem (I'm using vanilla arch).

Especially annoying when booting from an arch iso for installing arch.

---

package version: 2023-1

---

Steps to replicate:

Just run reflector.

added priority3-normal severity4-low statusunconfirmed labels

assigned to @morganamilo, @toolybird, and @thurstylark

changed the description

I confirm that I have exactly the same problem.

I found a workaround for archinstall cases - mask the reflector before install:

systemctl mask reflector.service

Something definitely seems amiss.. but reflector hasn't been updated in quite a while, so probably something has changed at the Arch end. cc @torxed @pitastrudl for visibility. Any ideas?

added statuswaiting-input label and removed statusunconfirmed label

assigned to @xyne and unassigned @morganamilo, @toolybird, and @thurstylark

Raising the priority because failing installations from the Arch ISO are not a good look..

added priority2-high scopebug severity2-high statusconfirmed labels and removed priority3-normal severity4-low statuswaiting-input labels

The problem was in percentage of fullness of the mirrors - only three of them were 100% full yesterday and only one of them has https protocol. Reflector only works with the mirrors are 100% full by default, so it only showed one server.

As a solution, the '--completion-percent 98' option can be added to the Arch install script, but it can lead to a more serious problem - partial updates

Even if it only showed one mirror, reflector.service should have ended up in a "completed" state right? Or what happened? Because archinstall monitors the service, waiting for it to be done, dead or similar states.

Yesterday there was a unique situation when all ~500 mirrors didn't have 100% completion.

https://bbs.archlinux.org/viewtopic.php?id=292010

Is there a way to check why all mirrors weren't 100% completed for a long time? Would be interesting to know why that happened.

This can probably be ignored now:

All I can say is that I've gotten multiple pings about it too, where people blame archinstall * (the symptoms just show up there)*.

Runnig reflector --verbose manually shows nothing and it hangs. Spamming Ctrl-C basically does nothing either.

The endpoint https://archlinux.org/mirrors/status/json/ appears to be working.

At the very least, Reflector could implement signal handling so outside influences could trigger signals (even via systemd that could be useful).

Further, I don't know how that xyne.dev "source code" repo works. But the issue appears to hit around line ~122, in Reflector.py (/usr/lib/Python3.11/Site-Packages/Reflector.py):

    with urllib.request.urlopen(url, None, connection_timeout) as handle:
        obj = json.loads(handle.read().decode())

A minimal test that represent what's run in the ISO would be:

import urllib.request
import json

url = "https://archlinux.org/mirrors/status/json/"
connection_timeout = 5
with urllib.request.urlopen(url, None, connection_timeout) as handle:
	obj = json.loads(handle.read().decode())

And that code works if run outside of the ISO. But inside the ISO it hangs. The root cause appears to be (in my test) the DNS lookup of archlinux.org, which is why the timeout=5 isn't effective.

Looking at journalctl -f -u systemd-resolved.service I got a bunch of:

Using degraded feature set UDP instead of TCP for DNS server <ip of dns server>
Using degraded feature set TCP instead of UDP for DNS server <ip of dns server>

Some weird loop that keeps going.

I wrote above, it wasn't a connectivity issue at all

I noted that we wrote at the exact same time, and hence why I edited the comment. However I'm wondering if the the default conf of resolved and the DNS lookup issue happens for others too. As I constantly get issues about reflector and hanging things.

I always refer to "git gud, fix your network" but perhaps there's some improvement to be made here too?

It seems to be working fine now. Though, not sure who/what fixed it.. or how.

closed

added resolutioncompleted label

Hey, just wanted to drop this bit of information. Seems that one of the location that archweb checks from (London located server) was unavailable for around 2 hours give or take. In that time all the checks from that location failed, making it seem like a mirror is unavailable which then in turn caused the Completion % to drop thus causing reflector to stop working. You can verify this by going to any server log history and looking at around 4AM on the 18th of January. You'll see pretty much everywhere that there was an issue. This data point will not be visible anymore in a couple of days as it rotates through.

mentioned in issue archlinux/arch-mirrors#52 (closed)