Browse Source

block SemrushBot which is spamming our logs

pull/823/head
Max Mehl 7 months ago
parent
commit
dafb337ed1
Signed by: max.mehl <max.mehl@fsfe.org> GPG Key ID: 2704E4AB371E2E92
1 changed files with 3 additions and 0 deletions
  1. 3
    0
      robots.txt

+ 3
- 0
robots.txt View File

@@ -1,2 +1,5 @@
User-agent: *
Disallow: /source/

User-agent: SemrushBot
Disallow: /

Loading…
Cancel
Save