Browse Source

block SemrushBot which is spamming our logs

master
Max Mehl 4 months ago
parent
commit
dafb337ed1
Signed by: max.mehl <max.mehl@fsfe.org> GPG Key ID: 2704E4AB371E2E92
1 changed files with 3 additions and 0 deletions
  1. 3
    0
      robots.txt

+ 3
- 0
robots.txt View File

@@ -1,2 +1,5 @@
1 1
 User-agent: *
2 2
 Disallow: /source/
3
+
4
+User-agent: SemrushBot
5
+Disallow: /

Loading…
Cancel
Save