geo-crawlers
geo-crawlers helps you understand how AI-powered search engines interact with your site. Detect which crawlers from ChatGPT, Claude, Perplexity, Gemini, and Google can access your content, then resolve any barriers preventing proper indexing. Stay ahead of AI search trends while preserving your traditional SEO performance.
geo-crawlers audits your site's accessibility to AI crawlers from ChatGPT, Claude, Perplexity, Gemini, and Google. It analyzes your robots.txt file, meta tags, and HTTP headers to identify which crawlers can reach your content and flags any blocking issues preventing proper indexing by AI search products.
AI-generated summary based on this skill's SKILL.md
Install
zubair-trabzada/geo-seo-claude/geo-crawlers · repository language: Python
git clone https://github.com/zubair-trabzada/geo-seo-claude
cp -r geo-seo-claude/skills/geo-crawlers ~/.claude/skills/geo-crawlersFrequently asked questions
AI-generated answers based on this skill's SKILL.md and metadata
How can I check if AI crawlers can access my website?
geo-crawlers audits your site's accessibility to AI crawlers from ChatGPT, Claude, Perplexity, Gemini, and Google. It analyzes your robots.txt file, meta tags, and HTTP headers to identify which crawlers can reach your content and flags any blocking issues preventing proper indexing by AI search products.
What robots.txt AI crawler rules should I configure?
geo-crawlers helps you configure robots.txt directives for specific AI crawlers like GPTBot, PerplexityBot, and Bytespider. You can strategically allow or block crawlers based on your content strategy—enabling visibility in ChatGPT and Claude while optionally restricting training-only crawlers or aggressive bots like Bytespider.
Why isn't my site appearing in ChatGPT search results?
geo-crawlers identifies blocking issues preventing ChatGPT-User bot and other AI search crawlers from accessing your content. Common causes include restrictive robots.txt rules, noindex meta tags, or X-Robots-Tag headers. The tool provides specific recommendations to resolve each barrier and maximize your AI search visibility.
How do I block AI crawlers from training on my content?
geo-crawlers supports multiple control methods: robots.txt Disallow rules for training-focused crawlers, meta noai tags, X-Robots-Tag headers, and llms.txt file configuration. You can distinguish between crawlers used for live AI search (which you may want to allow) versus those used only for model training (which you can block separately).
What is the difference between AI search visibility and model training access?
geo-crawlers helps you understand that some crawlers like GPTBot serve live ChatGPT search, while others like Google-Extended support Gemini training. You can allow live search crawlers to improve visibility in AI products while blocking training-only crawlers from using your content for model development, giving you granular control over both use cases.
How do I implement meta robots noai tags and content signals?
geo-crawlers guides you through adding meta noai tags to block AI crawlers, configuring X-Robots-Tag HTTP headers for fine-grained control, and setting up llms.txt files per IETF draft standards. The tool validates your implementation and provides an access map showing exactly which crawlers can reach each part of your site.
SKILL.md
rendered from the published skill — quoted content, verbatim
AI Crawler Access Analysis Skill
Purpose
This skill analyzes a website's accessibility to AI crawlers -- the bots that AI companies use to discover, index, and train on web content. If AI crawlers are blocked, the site's content cannot appear in AI-generated responses regardless of its quality. Crawler access is the foundational technical requirement for GEO.
Key Insight
As of early 2026, many websites inadvertently block AI crawlers through overly aggressive robots.txt rules, inherited from legacy SEO configurations. An Originality.ai 2025 study found that over 35% of the top 1,000 websites block at least one major AI crawler, and 5-10% block all AI crawlers. Blocking AI crawlers is the single fastest way to become invisible in AI-generated search results.
Complete AI Crawler Reference
Tier 1:
(truncated - see the full file via the links below)
Read as markdown · JSON record · Browse the source repository
File tree — 1 file
skills/geo-crawlers/SKILL.md