mirrors/bookwyrm
1
0
Fork 1
mirror of https://github.com/bookwyrm-social/bookwyrm.git synced 2024-06-10 09:19:27 +00:00
Code Issues Projects Releases Wiki Activity

Expand robots.txt

Browse source
This commit is contained in:
Mouse Reeve 2022-11-19 10:03:40 -08:00
parent bbd2dc75f8
commit 7d51a69c71
1 changed files with 8 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

8
bookwyrm/templates/robots.txt
View file

@ -73,6 +73,14 @@ User-agent: PetalBot
Disallow: /
User-agent: DataForSeoBot
Disallow: /
User-agent: YisouSpider
Disallow: /
User-agent: *
Crawl-delay: 10
Disallow: /static/js/
Disallow: /static/css/