aur.archlinux.org bandwidth issue

Pamac now uses the AUR metadata to do searches which meant that we now almost reached our 20TB bandwidth limit, some information:

The metadata is quite large

-rw-r--r-- 1 aur aur 8.2M Feb 10 19:03 /srv/http/aurweb/web/html/packages-meta-ext-v1.json.gz

Collect all the Pamac requests to metadata.

awk '/packages-meta-ext-v1.json.gz/ { if ($12 ~ /Pamac/) map[$1]++ } END { for (k in map) printf "%s %s\n", k, map[k] }' < access.log > /tmp/ips.log

wc -l /tmp/ips.log
68055 /tmp/ips.log

Sorting by IP:

$ sort -k 2,2 /tmp/ips.log
X.X.X.X 9
Y.Y.Y.Y 9
Z.Z.Z.Z 9
W.W.W.W 9
A.A.A.A. 90

So doing some silly maths: 68055 * 9 * 8.8 = 5389956 M (~ 0.5 TB)

Our bandwidth usage:

   daily
                     rx      |     tx      |    total    |   avg. rate
     ------------------------+-------------+-------------+---------------
     yesterday     35.89 GiB |    1.66 TiB |    1.70 TiB |  172.89 Mbit/s
         today     27.12 GiB |    1.25 TiB |    1.28 TiB |  161.21 Mbit/s

Before the new AURweb deployment this was ~ 60-70G daily.

After blocking the Pamac user agent: