From 6c4c691fe781454d1e7f271f78bc5fa7225539a8 Mon Sep 17 00:00:00 2001 From: Zed Date: Mon, 9 Jan 2023 00:12:07 +0100 Subject: [PATCH] Add Crawl-delay to robots.txt --- public/robots.txt | 1 + 1 file changed, 1 insertion(+) diff --git a/public/robots.txt b/public/robots.txt index 7a74568..812656a 100644 --- a/public/robots.txt +++ b/public/robots.txt @@ -1,4 +1,5 @@ User-agent: * Disallow: / +Crawl-delay: 1 User-agent: Twitterbot Disallow: