mirrors/nitter
1
0
Fork 0
mirror of https://github.com/zedeus/nitter.git synced 2024-05-19 17:48:10 +00:00
Code Issues Packages Projects Releases Wiki Activity
nitter/public/robots.txt

6 lines
74 B
Plaintext
Raw Normal View History

Block search engines via robots.txt (#631) Prevents instances from being rate limited due to being senselessly crawled by search engines. Since there is no reason to index Nitter instances, simply block all robots. Notably, this does *not* affect link previews (e.g. in various chat software).
2022-06-04 15:48:25 +00:00
User-agent: *
Disallow: /
Add Crawl-delay to robots.txt
2023-01-08 23:12:07 +00:00
Crawl-delay: 1
Fix Twitterbot rule in robots.txt
2022-06-04 15:55:35 +00:00
User-agent: Twitterbot
Disallow:
Reference in a new issue Copy permalink