robots-txt
Manage crawler permissions and site indexing rules through robots.txt setup and validation. This skill helps you configure directives that guide search engines and AI bots on which content to crawl and index, while auditing existing configurations for compliance and optimization.
robots-txt is configured by creating a plain text file named robots.txt and placing it in your site's root directory (e.g., example.com/robots.txt). The file uses simple directives like User-agent, Disallow, Allow, and Sitemap to specify crawler permissions. Each rule applies to a specific user-agent (crawler type) or all crawlers using User-agent: *. Save the file as UTF-8 text with no BOM, then verify it's accessible via your web server.
AI-generated summary based on this skill's SKILL.md
Install
kostja94/marketing-skills/robots
git clone https://github.com/kostja94/marketing-skills
cp -r marketing-skills/skills/seo/technical/robots ~/.claude/skills/robotsFrequently asked questions
AI-generated answers based on this skill's SKILL.md and metadata
How do I create a robots.txt file?
robots-txt is configured by creating a plain text file named robots.txt and placing it in your site's root directory (e.g., example.com/robots.txt). The file uses simple directives like User-agent, Disallow, Allow, and Sitemap to specify crawler permissions. Each rule applies to a specific user-agent (crawler type) or all crawlers using User-agent: *. Save the file as UTF-8 text with no BOM, then verify it's accessible via your web server.
What is the robots.txt disallow syntax and how does it work?
robots-txt uses Disallow directives to block crawlers from specific paths. The syntax is Disallow: /path/ where /path/ is relative to your domain root. For example, Disallow: /admin/ blocks the /admin/ folder and all its contents. Disallow: / blocks everything; Disallow: (blank) allows everything. Paths are case-sensitive and support wildcards ($, *). Each Disallow rule applies to the User-agent specified above it.
How can I block GPTBot and other AI crawlers with robots.txt?
robots-txt lets you block specific AI crawlers by targeting their user-agent names. Add rules like User-agent: GPTBot followed by Disallow: / to block OpenAI's crawler entirely. Similarly, use User-agent: ClaudeBot for Anthropic's bot, User-agent: PerplexityBot for Perplexity, and User-agent: CCBot for Common Crawl. You can also block all AI crawlers with a catch-all rule, or allow specific paths while disallowing others per crawler.
What is the difference between robots.txt and noindex?
robots-txt blocks crawlers from accessing content before they read it, preventing indexing and bandwidth use. The noindex meta tag or header tells crawlers to index the page but not display it in search results—crawlers still must reach the page first. Use robots-txt to prevent access entirely (admin areas, private content); use noindex when you want crawlers to see the page but exclude it from search indexes. They serve different purposes.
What are robots.txt allow vs disallow directives and when do I use each?
robots-txt Disallow: /path/ blocks crawlers from accessing a path; Allow: /path/ overrides a broader disallow for that specific path. For example, Disallow: /temp/ blocks everything in /temp/, but Allow: /temp/public/ lets crawlers access /temp/public/. Most crawlers respect Allow directives to create exceptions. Use Disallow for content you want hidden; use Allow when you need fine-grained exceptions within a blocked folder.
How do I audit and verify my robots.txt configuration?
robots-txt can be audited by checking it's accessible at yourdomain.com/robots.txt and reviewing syntax for errors. Use Google Search Console's robots.txt tester to simulate how Googlebot interprets your rules. Check that Disallow paths are correctly formatted, User-agent names match actual crawlers, and Sitemap URLs are valid. Test specific URLs against your rules to ensure crawlers can access intended content and are blocked from restricted areas.
SKILL.md
rendered from the published skill — quoted content, verbatim
SEO Technical: robots.txt
Guides configuration and auditing of robots.txt for search engine and AI crawler control.
When invoking: On first use, if helpful, open with 1–2 sentences on what this skill covers and why it matters, then provide the main output. On subsequent use or when the user asks to skip, go directly to the main output.
Scope (Technical SEO)
- Robots.txt: Configure Disallow/Allow, Sitemap, Clean-param; audit for accidental blocks
- Crawler access: Path-level crawl control; AI crawler allow/block strategy
- Differentiation: robots.txt = crawl control (who accesses what paths); noindex = index control (what gets indexed). See indexing for page-level exclusions.
Initial Assessment
**Check for project context
(truncated - see the full file via the links below)
Read as markdown · JSON record · Browse the source repository
File tree — 1 file
skills/seo/technical/robots/SKILL.md