archwiki: Fight load spikes caused by crawlers

Christian Heuselrequested to merge
wiki-cpx52 into master

  • tf-stage1: Upgrade the Wiki box to CPX52 ➡️ Give the server more resources

  • archwiki: Write Mediawiki specific anubis rules ➡️ Enable Anubis selectively for expensive endpoints such as "Special:RecentChanges" etc.

  • anubis: Create config dropin structure ➡️ Create a config structure so that we can more easily configure anubis on a per-service basis.

  • anubis: Drop paid-only configuration ➡️ Drop config which requires thoth service (paid)

Undone again:

  • archwiki: Enable Extension:CrawlerProtection ➡️ make the expensive pages only available for logged in users Extension:CrawlerProtection

Cherry-picked to master:

  • archwiki: Fix the error_log in proxy location

See https://www.mediawiki.org/wiki/Manual:Handling_web_crawlers

cc @archlinux/teams/wiki/admins regarding the wiki config changes, as for example page history is only viewable for logged in users right now.

Edited by Christian Heusel

Merge request reports