FSFE/fsfe-website
28
32
Fork 92
Code Issues 64 Pull Requests 15 Wiki Activity
Files
74eac0a55926ab30a0327f4c204367a3acc3de11
fsfe-website/robots.txt

6 lines
69 B
Plaintext
Raw Normal View History

add robots.txt files for fsfe.org and test.fsfe.org
2018-12-10 14:25:20 +01:00
User-agent: *
Disallow: /source/
block SemrushBot which is spamming our logs
2019-03-06 13:46:52 +01:00
User-agent: SemrushBot
Disallow: /
Reference in New Issue Copy Permalink