FSFE/fsfe-website
27
32
Fork 93
Code Issues 63 Pull Requests 18 Wiki Activity
Files
2b2f9da4c07feddd945bb90f6c4ce5e9249aa568
fsfe-website/fsfe.org/robots.txt

6 lines
69 B
Plaintext
Raw Normal View History

add robots.txt files for fsfe.org and test.fsfe.org
2018-12-10 14:25:20 +01:00
User-agent: *
Disallow: /source/
block SemrushBot which is spamming our logs
2019-03-06 13:46:52 +01:00
User-agent: SemrushBot
Disallow: /
Reference in New Issue Copy Permalink