FedericoSchonborn/website
1
0
Fork 0
forked from auxolotl/website
Code Pull requests Activity

chore: generate robots.txt

Browse source
This commit is contained in:
github-actions 2024-06-15 00:46:38 +00:00
parent 9abceac04e
commit eb9072d8ed
1 changed files with 6 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

6
public/robots.txt
View file

@ -16,6 +16,12 @@ Disallow: /
User-agent: Applebot User-agent: Applebot
Disallow: / Disallow: /
# AI Data Scraper
# https://darkvisitors.com/agents/applebot-extended
User-agent: Applebot-Extended
Disallow: /
# AI Data Scraper # AI Data Scraper
# https://darkvisitors.com/agents/bytespider # https://darkvisitors.com/agents/bytespider