robots.txt is a plain text file placed at the root of your website that instructs web crawlers which pages they can and cannot access. It follows the Robots Exclusion Protocol.

Does robots.txt guarantee bots won't access my pages?

No. robots.txt is advisory — well-behaved crawlers (Googlebot, Bingbot) respect it, but malicious bots can ignore it. Use server-side authentication for truly private content.

What is the 'Block AI Bots' preset?

This preset adds Disallow: / rules for known AI training crawlers including GPTBot (OpenAI), CCBot (Common Crawl), Google-Extended, and anthropic-ai, preventing them from training on your content.

Where should I place the robots.txt file?

Place it at the root of your domain: https://yourdomain.com/robots.txt — it must be at this exact path to be recognized by search engines.

Developer Gratuito Sem cadastro

Robots.txt Generator

Generate a robots.txt file with presets for common crawlers and custom user-agent rules.

Carregando ferramenta…

Sobre esta ferramenta

Build a valid robots.txt file visually without memorizing the syntax. Choose from quick presets — Allow All, Block All, Common Crawlers Only, or Block AI Bots — or craft fully custom rules. Add as many User-agent / Allow / Disallow rule groups as you need using the dynamic row builder. Optionally add a Sitemap URL. The generated robots.txt updates live and can be copied to clipboard or downloaded directly as `robots.txt` ready for your server's root directory.

Como usar

1 Choose a preset (Allow All, Block All, Common Crawlers, or Block AI Bots) or start custom.
2 Add rule groups by clicking 'Add User-agent Group'.
3 For each group, enter the User-agent and add Allow/Disallow rules.
4 Optionally add your sitemap URL at the bottom.
5 Click Copy or Download to save your robots.txt file.

Perguntas frequentes

Ferramentas relacionadas

Meta Tag Generator

Generate SEO-optimized HTML meta tags with live Google and Twitter card preview.

Analisador e Construtor de URL

Divida qualquer URL em seus componentes ou monte uma URL a partir de partes — com um editor de parâmetros de consulta em tempo real.

Minificador de HTML / CSS / JS

Minifique HTML, CSS e JavaScript para reduzir o tamanho do arquivo — funciona inteiramente no seu navegador.

Formatador e validador JSON

Valide dados JSON contra um Esquema JSON com mensagens de erro detalhadas no nível do caminho.