aurweb: Create robots.txt

This should i.e. forbid crawlers to index all of the git history pages which put's unneccessary load on the server.

Link: #610 Signed-off-by: Christian Heusel christian@heusel.eu

requested review from @jelle

This is also just a first version to get one in place, we should probably add some more things to it

mentioned in issue #610

requested changes

added 1 commit

• 84ccaeed - aurweb: Create robots.txt

Compare with previous version

resolved all threads

added 1 commit

• 50d9df48 - aurweb: Create robots.txt

Compare with previous version

added 1 commit

• d6c9a5c5 - aurweb: Create robots.txt

Compare with previous version

assigned to @svenstaro, @anthraxx, and @gromit

approved this merge request

marked this merge request as ready

added 1 commit

• f562b4e9 - aurweb: Create robots.txt

Compare with previous version

approved this merge request

merged manually