{"enrichment":{"faq":[{"a":"robots-txt is configured by creating a plain text file named robots.txt and placing it in your site's root directory (e.g., example.com/robots.txt). The file uses simple directives like User-agent, Disallow, Allow, and Sitemap to specify crawler permissions. Each rule applies to a specific user-agent (crawler type) or all crawlers using User-agent: *. Save the file as UTF-8 text with no BOM, then verify it's accessible via your web server.","q":"How do I create a robots.txt file?"},{"a":"robots-txt uses Disallow directives to block crawlers from specific paths. The syntax is Disallow: /path/ where /path/ is relative to your domain root. For example, Disallow: /admin/ blocks the /admin/ folder and all its contents. Disallow: / blocks everything; Disallow: (blank) allows everything. Paths are case-sensitive and support wildcards ($, *). Each Disallow rule applies to the User-agent specified above it.","q":"What is the robots.txt disallow syntax and how does it work?"},{"a":"robots-txt lets you block specific AI crawlers by targeting their user-agent names. Add rules like User-agent: GPTBot followed by Disallow: / to block OpenAI's crawler entirely. Similarly, use User-agent: ClaudeBot for Anthropic's bot, User-agent: PerplexityBot for Perplexity, and User-agent: CCBot for Common Crawl. You can also block all AI crawlers with a catch-all rule, or allow specific paths while disallowing others per crawler.","q":"How can I block GPTBot and other AI crawlers with robots.txt?"},{"a":"robots-txt blocks crawlers from accessing content before they read it, preventing indexing and bandwidth use. The noindex meta tag or header tells crawlers to index the page but not display it in search results\u2014crawlers still must reach the page first. Use robots-txt to prevent access entirely (admin areas, private content); use noindex when you want crawlers to see the page but exclude it from search indexes. They serve different purposes.","q":"What is the difference between robots.txt and noindex?"},{"a":"robots-txt Disallow: /path/ blocks crawlers from accessing a path; Allow: /path/ overrides a broader disallow for that specific path. For example, Disallow: /temp/ blocks everything in /temp/, but Allow: /temp/public/ lets crawlers access /temp/public/. Most crawlers respect Allow directives to create exceptions. Use Disallow for content you want hidden; use Allow when you need fine-grained exceptions within a blocked folder.","q":"What are robots.txt allow vs disallow directives and when do I use each?"},{"a":"robots-txt can be audited by checking it's accessible at yourdomain.com/robots.txt and reviewing syntax for errors. Use Google Search Console's robots.txt tester to simulate how Googlebot interprets your rules. Check that Disallow paths are correctly formatted, User-agent names match actual crawlers, and Sitemap URLs are valid. Test specific URLs against your rules to ensure crawlers can access intended content and are blocked from restricted areas.","q":"How do I audit and verify my robots.txt configuration?"}],"shadow_tags":["crawler-control","seo-technical","ai-bot-blocking","path-level-exclusion","crawl-directive","search-engine-access","robots-protocol","content-protection","indexing-strategy"],"summary_rewrite":"Manage crawler permissions and site indexing rules through robots.txt setup and validation. This skill helps you configure directives that guide search engines and AI bots on which content to crawl and index, while auditing existing configurations for compliance and optimization."},"files":[{"bytes":7644,"path":"skills/seo/technical/robots/SKILL.md","sha256":"d7dac75c1e956a0f0675eef0fbc097bb3c96280c62d3c4a3646b70339f98001d","url":"https://skillfed.io/files/kostja94/marketing-skills/robots/9dbbc0e4/SKILL.md"}],"id":"kostja94/marketing-skills/robots","links":{"html":"https://skillfed.io/kostja94/marketing-skills/robots","md":"https://skillfed.io/kostja94/marketing-skills/robots.md","repo":"https://github.com/kostja94/marketing-skills"},"meta":{"agents_supported":[],"first_seen":"2026-07-28","forks":109,"language":null,"last_updated":"2026-06-09","license":"MIT","name":"robots-txt","publisher":"kostja94","stars":762},"relations":{"similar":[{"id":"zubair-trabzada/geo-seo-claude/geo-crawlers"},{"id":"kostja94/marketing-skills/indexing"},{"id":"AgriciDaniel/codex-seo/seo-technical"},{"id":"AgriciDaniel/claude-seo/seo-technical"},{"id":"schwepps/skills/seo-technical-audit"},{"id":"kostja94/marketing-skills/crawlability"},{"id":"alinaqi/maggy/site-architecture"},{"id":"rampstackco/claude-skills/seo-technical"},{"id":"tech-leads-club/agent-skills/tlc-generative-engine-optimization"},{"id":"zubair-trabzada/geo-seo-claude/geo-technical"}]},"slug":{"owner":"kostja94","repo":"marketing-skills","skill":"robots"},"version":"9dbbc0e4"}