robots.txt is a plain text file placed at the root of your website that instructs web crawlers which pages they can and cannot access. It follows the Robots Exclusion Protocol.

Does robots.txt guarantee bots won't access my pages?

No. robots.txt is advisory — well-behaved crawlers (Googlebot, Bingbot) respect it, but malicious bots can ignore it. Use server-side authentication for truly private content.

What is the 'Block AI Bots' preset?

This preset adds Disallow: / rules for known AI training crawlers including GPTBot (OpenAI), CCBot (Common Crawl), Google-Extended, and anthropic-ai, preventing them from training on your content.

Where should I place the robots.txt file?

Place it at the root of your domain: https://yourdomain.com/robots.txt — it must be at this exact path to be recognized by search engines.

Developer Gratis Sin registro

Robots.txt Generator

Generate a robots.txt file with presets for common crawlers and custom user-agent rules.

Cargando la herramienta…

Acerca de esta herramienta

Build a valid robots.txt file visually without memorizing the syntax. Choose from quick presets — Allow All, Block All, Common Crawlers Only, or Block AI Bots — or craft fully custom rules. Add as many User-agent / Allow / Disallow rule groups as you need using the dynamic row builder. Optionally add a Sitemap URL. The generated robots.txt updates live and can be copied to clipboard or downloaded directly as `robots.txt` ready for your server's root directory.

Cómo usar

1 Choose a preset (Allow All, Block All, Common Crawlers, or Block AI Bots) or start custom.
2 Add rule groups by clicking 'Add User-agent Group'.
3 For each group, enter the User-agent and add Allow/Disallow rules.
4 Optionally add your sitemap URL at the bottom.
5 Click Copy or Download to save your robots.txt file.

Preguntas frecuentes

Herramientas relacionadas

Meta Tag Generator

Generate SEO-optimized HTML meta tags with live Google and Twitter card preview.

Analizador y Constructor de URL

Descomponga cualquier URL en sus componentes o ensamble una URL a partir de partes — con un editor de parámetros de consulta en vivo.

Minificador HTML / CSS / JS

Minifique HTML, CSS y JavaScript para reducir el tamaño del archivo — se ejecuta completamente en su navegador.

Formateador y validador JSON

Valide datos JSON contra un Esquema JSON con mensajes de error detallados a nivel de ruta.