7 lines
293 B
Plaintext
7 lines
293 B
Plaintext
# https://www.robotstxt.org/
|
|
|
|
# Crawling stays allowed on purpose. While the site is a prototype, every response carries `X-Robots-Tag: noindex, nofollow`,
|
|
# and a crawler barred here would never read it — it could still list the bare URLs it finds linked elsewhere.
|
|
User-agent: *
|
|
Disallow:
|