Skip to content
GitLab
Projects Groups Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
  • Sign in
  • infrastructure infrastructure
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Locked Files
  • Issues 106
    • Issues 106
    • List
    • Boards
    • Service Desk
    • Milestones
    • Iterations
  • Merge requests 14
    • Merge requests 14
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
    • Test Cases
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Code review
    • Insights
    • Issue
    • Repository
  • Wiki
    • Wiki
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • Arch LinuxArch Linux
  • infrastructureinfrastructure
  • Issues
  • #437
Closed
Open
Issue created Feb 10, 2022 by Jelle van der Waa@jelle🚧Developer

aur.archlinux.org bandwidth issue

Pamac now uses the AUR metadata to do searches which meant that we now almost reached our 20TB bandwidth limit, some information:

The metadata is quite large

-rw-r--r-- 1 aur aur 8.2M Feb 10 19:03 /srv/http/aurweb/web/html/packages-meta-ext-v1.json.gz

Collect all the Pamac requests to metadata.

awk '/packages-meta-ext-v1.json.gz/ { if ($12 ~ /Pamac/) map[$1]++ } END { for (k in map) printf "%s %s\n", k, map[k] }' < access.log > /tmp/ips.log
wc -l /tmp/ips.log
68055 /tmp/ips.log

Sorting by IP:

$ sort -k 2,2 /tmp/ips.log
X.X.X.X 9
Y.Y.Y.Y 9
Z.Z.Z.Z 9
W.W.W.W 9
A.A.A.A. 90

So doing some silly maths: 68055 * 9 * 8.8 = 5389956 M (~ 0.5 TB)

Our bandwidth usage:

   daily
                     rx      |     tx      |    total    |   avg. rate
     ------------------------+-------------+-------------+---------------
     yesterday     35.89 GiB |    1.66 TiB |    1.70 TiB |  172.89 Mbit/s
         today     27.12 GiB |    1.25 TiB |    1.28 TiB |  161.21 Mbit/s

Before the new AURweb deployment this was ~ 60-70G daily.

After blocking the Pamac user agent: image

Edited Feb 10, 2022 by Jelle van der Waa
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information
Assignee
Assign to
Time tracking