geo-crawlers
geo-crawlers helps you understand how AI-powered search engines interact with your site. Detect which crawlers from ChatGPT, Claude, Perplexity, Gemini, and Google can access your content, then resolve any barriers preventing proper indexing. Stay ahead of AI search trends while preserving your traditional SEO performance.
geo-crawlers audits your site's accessibility to AI crawlers from ChatGPT, Claude, Perplexity, Gemini, and Google. It analyzes your robots.txt file, meta tags, and HTTP headers to identify which crawlers can reach your content and flags any blocking issues preventing proper indexing by AI search products.
AI-generated summary based on this skill's SKILL.md
Decision gist · record as of 2026-05-27
geo-crawlers audits your site's accessibility to AI crawlers from ChatGPT, Claude, Perplexity, Gemini, and Google. It analyzes your robots.txt file, meta tags, and HTTP headers to identify which crawlers can reach your content and flags any blocking issues preventing proper indexing by AI search products.
Use it when
- geo-crawlers helps you configure robots.txt directives for specific AI crawlers like GPTBot, PerplexityBot, and Bytespider.
- geo-crawlers identifies blocking issues preventing ChatGPT-User bot and other AI search crawlers from accessing your content.
Verify before relying
Read SKILL.md below before installing (1 file). Open directory: indexed for reading, not audited.
Install
zubair-trabzada/geo-seo-claude/geo-crawlers · repository language: Python
Open directory. Skills are indexed for reading, not audited. Review a skill's body before installing it.
Frequently asked questions
AI-generated answers based on this skill's SKILL.md and metadata
How can I check if AI crawlers can access my website?
geo-crawlers audits your site's accessibility to AI crawlers from ChatGPT, Claude, Perplexity, Gemini, and Google. It analyzes your robots.txt file, meta tags, and HTTP headers to identify which crawlers can reach your content and flags any blocking issues preventing proper indexing by AI search products.
What robots.txt AI crawler rules should I configure?
geo-crawlers helps you configure robots.txt directives for specific AI crawlers like GPTBot, PerplexityBot, and Bytespider. You can strategically allow or block crawlers based on your content strategy—enabling visibility in ChatGPT and Claude while optionally restricting training-only crawlers or aggressive bots like Bytespider.
Why isn't my site appearing in ChatGPT search results?
geo-crawlers identifies blocking issues preventing ChatGPT-User bot and other AI search crawlers from accessing your content. Common causes include restrictive robots.txt rules, noindex meta tags, or X-Robots-Tag headers. The tool provides specific recommendations to resolve each barrier and maximize your AI search visibility.
How do I block AI crawlers from training on my content?
geo-crawlers supports multiple control methods: robots.txt Disallow rules for training-focused crawlers, meta noai tags, X-Robots-Tag headers, and llms.txt file configuration. You can distinguish between crawlers used for live AI search (which you may want to allow) versus those used only for model training (which you can block separately).
What is the difference between AI search visibility and model training access?
geo-crawlers helps you understand that some crawlers like GPTBot serve live ChatGPT search, while others like Google-Extended support Gemini training. You can allow live search crawlers to improve visibility in AI products while blocking training-only crawlers from using your content for model development, giving you granular control over both use cases.
How do I implement meta robots noai tags and content signals?
geo-crawlers guides you through adding meta noai tags to block AI crawlers, configuring X-Robots-Tag HTTP headers for fine-grained control, and setting up llms.txt files per IETF draft standards. The tool validates your implementation and provides an access map showing exactly which crawlers can reach each part of your site.
SKILL.md
Rendered from the published skill. Quoted content, verbatim.
AI Crawler Access Analysis Skill
Purpose
This skill analyzes a website's accessibility to AI crawlers -- the bots that AI companies use to discover, index, and train on web content. If AI crawlers are blocked, the site's content cannot appear in AI-generated responses regardless of its quality. Crawler access is the foundational technical requirement for GEO.
Key Insight
As of early 2026, many websites inadvertently block AI crawlers through overly aggressive robots.txt rules, inherited from legacy SEO configurations. An Originality.ai 2025 study found that over 35% of the top 1,000 websites block at least one major AI crawler, and 5-10% block all AI crawlers. Blocking AI crawlers is the single fastest way to become invisible in AI-generated search results.
Complete AI Crawler Reference
Tier 1:
(truncated - see the full file via the links below)
File tree — 1 file
skills/geo-crawlers/SKILL.md
Let your AI agent find skills like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 56,283 agent skills by what they can do, searchable in plain language.
wish › “Audit which AI crawlers can access my website and identify blocking issues”
Give your agent the search over MCP, or paste the wish link into any chat. No install? Search from any chat →
Related skills
Track and interpret how AI agents interact with your infrastructure by examining server logs for bot activity patterns. Gain actionable insights into crawl frequency, request timing, and resource usage to refine your content strategy and improve bot-to-human traffic balance. Essential for teams managing high-volume AI agent access.
Manage crawler permissions and site indexing rules through robots.txt setup and validation. This skill helps you configure directives that guide search engines and AI bots on which content to crawl and index, while auditing existing configurations for compliance and optimization.
Build a solid SEO foundation by implementing core technical elements like robots.txt directives, XML sitemaps, and structured meta tags. This skill guides you through configuring each component to ensure search engines can discover and properly index your site's content. Establish best practices that support long-term organic visibility.
geo-audit helps you evaluate whether your website is properly configured for discovery and indexing by AI systems and search crawlers. Run a comprehensive technical assessment to identify missing metadata, blocking directives, and accessibility issues that might prevent LLMs from effectively processing your content. Get actionable insights to optimize your site's AI-readiness.
This skill grants AI agents controlled access to your Shopify Admin API, letting tools like ChatGPT, Claude, and Perplexity crawl and interact with your store's data. It bridges the gap between conversational AI and live commerce operations, enabling agents to retrieve inventory, orders, and customer information while maintaining security boundaries. Perfect for stores looking to automate agent-driven workflows and real-time store management.
geo-technical helps you evaluate your website's technical foundation through the lens of both traditional search engines and AI-powered platforms like ChatGPT, Claude, and Perplexity. It identifies gaps in crawlability, indexing, and content structure that matter to modern discovery systems. Use this skill to shift your optimization strategy toward where traffic is actually heading.
More skills Seo Technical (NOASSERTION) · Seo Geo (NOASSERTION) · seo-geo (MIT) · seo-technical-audit (MIT) · seo-technical (MIT)