Robots.txt Generator for AI

Control how AI crawlers and LLMs interact with your website. Protect your intellectual property while staying visible to the right models.

AI Crawlers

Search Engines

Disallow Paths

/admin/
/private/
/tmp/

Crawl Delay

Time between requests for crawlers that support it (in seconds)

# Robots.txt generated by Temso AI Robots.txt Generator
# https://temso.ai/free-tools/robots-txt-generator

# Default rule for all robots
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /private/
Disallow: /tmp/
Crawl-delay: 10

Pro Tip

Place the generated content in a file named robots.txt in your website's root directory (e.g., yoursite.com/robots.txt).

Why control AI crawlers?

As AI models evolve, control over your data becomes critical. Some bots train on your content to build competing models, while others help users find your brand. Our generator helps you selectively allow or block access based on your strategy.

How to use this tool

Simply toggle the bots you wish to allow or block. Add any specific paths you want to keep private in the "Disallow Paths" section. Once ready, click "Copy to clipboard" and paste the content into arobots.txt file in your site's root directory.