mirrors/nitter
1
0
Fork 0
mirror of https://github.com/zedeus/nitter.git synced 2024-05-19 17:48:10 +00:00
Code Issues Packages Projects Releases Wiki Activity
1057 commits 4 branches 0 tags 8.3 MiB
Commit graph

4 commits

Author SHA1 Message Date
Zed 6c4c691fe7 Add Crawl-delay to robots.txt 2023-01-09 00:12:07 +01:00
Zed 138826fb4f Fix Twitterbot rule in robots.txt 2022-06-04 17:55:35 +02:00
Zed dfb2519870 Explicitly allow Twitterbot to generate previews 2022-06-04 17:52:47 +02:00
minus c543a1df8c
Block search engines via robots.txt (#631) 
Prevents instances from being rate limited due to being senselessly
crawled by search engines. Since there is no reason to index Nitter
instances, simply block all robots. Notably, this does *not* affect link
previews (e.g. in various chat software).
 2022-06-04 17:48:25 +02:00