Verified Commit 8691ab73 authored by Jelle van der Waa's avatar Jelle van der Waa 🚧
Browse files

Disallow scraping the search results on the AUR

As the queries for searching are the most 'heavy' queries we can
disallow bots to scrape through all search results. Archweb uses the
same robots.txt rules for disallowing indexing search results.
parent 1b6fad6a
Pipeline #1549 passed with stage
in 47 seconds
User-agent: *
Disallow: /packages/?
Disallow: /packages/?*
Crawl-delay: 2
......@@ -141,6 +141,9 @@
- name: make nginx log dir
file: path=/var/log/nginx/{{ aurweb_domain }} state=directory owner=root group=root mode=0755
- name: configure robots.txt
copy: src=robots.txt dest="{{ aurweb_dir }}/robots.txt" owner=aur group=http mode=0644
- name: configure php-fpm
src=php-fpm.conf.j2 dest="/etc/php/php-fpm.d/{{ aurweb_user }}.conf"
......@@ -40,6 +40,10 @@ server {
root {{ aurweb_dir }}/web/html;
index index.php;
location /robots.txt {
alias {{ aurweb_dir }}/robots.txt
location ~ ^/trusted-user/ {
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment