site-crawlability
This skill helps you identify and resolve technical barriers preventing search engines and AI crawlers from accessing your web pages. Run diagnostics on robots.txt, sitemap configuration, and indexation blockers, then apply targeted fixes to improve discoverability across search and AI platforms.
site-crawlability helps you identify and resolve technical barriers preventing search engines and AI crawlers from accessing your web pages. Start by auditing your robots.txt and sitemap configuration, then fix indexation blockers like broken internal links, orphan pages with no incoming links, and redirect chains. Optimize your site structure to reduce crawl budget waste and ensure all important content is reachable within 3 clicks from your homepage.
AI-generated summary based on this skill's SKILL.md
Decision gist · record as of 2026-06-09
site-crawlability helps you identify and resolve technical barriers preventing search engines and AI crawlers from accessing your web pages. Start by auditing your robots.txt and sitemap configuration, then fix indexation blockers like broken internal links, orphan pages with no incoming links, and redirect chains. Optimize your site structure to reduce crawl budget waste and ensure all important content is reachable within 3 clicks from your homepage.
Use it when
- site-crawlability identifies orphan pages—content with no incoming internal links that crawlers struggle to discover.
- site-crawlability optimizes site structure and internal linking to reduce crawl budget waste.
Verify before relying
Read SKILL.md below before installing (1 file). Open directory: indexed for reading, not audited.
Install
kostja94/marketing-skills/crawlability
Open directory. Skills are indexed for reading, not audited. Review a skill's body before installing it.
Frequently asked questions
AI-generated answers based on this skill's SKILL.md and metadata
How do I improve my site's crawlability for search engines?
site-crawlability helps you identify and resolve technical barriers preventing search engines and AI crawlers from accessing your web pages. Start by auditing your robots.txt and sitemap configuration, then fix indexation blockers like broken internal links, orphan pages with no incoming links, and redirect chains. Optimize your site structure to reduce crawl budget waste and ensure all important content is reachable within 3 clicks from your homepage.
What are orphan pages and how do I fix them for SEO?
site-crawlability identifies orphan pages—content with no incoming internal links that crawlers struggle to discover. Fix them by adding internal links from relevant pages, updating your sitemap to include them, or consolidating orphan content into existing pages. Resolving orphan pages improves indexation and ensures crawlers can reach all your valuable content.
How can site-crawlability help optimize my crawl budget?
site-crawlability optimizes site structure and internal linking to reduce crawl budget waste. It helps you eliminate redirect chains, fix broken internal links, and streamline your site architecture so search engines spend their crawl budget on important pages rather than errors or redundant URLs. This is especially critical for large sites where crawl efficiency directly impacts indexation.
Should I use infinite scroll or pagination for AI crawler optimization?
site-crawlability recommends pagination with rel="prev" and rel="next" tags for better AI crawler and search engine visibility. Infinite scroll and masonry layouts create crawl traps that prevent bots from discovering all content. Pagination ensures crawlers can navigate through your pages systematically and index all content without hitting URL generation traps.
How does site-crawlability help with AI bot indexation?
site-crawlability diagnoses why content isn't being indexed by AI crawlers like GPTBot and ClaudeBot. It implements server-side rendering (SSR) for better content visibility, optimizes your sitemap and canonical URLs, and fixes crawlable content issues. These improvements help AI platforms discover and index your pages alongside traditional search engines.
What technical issues does site-crawlability audit and fix?
site-crawlability audits robots.txt configuration, broken internal links, redirect chains, orphan pages, sitemap errors, and crawl traps. It resolves indexation blockers, optimizes site structure to improve discoverability, and implements pagination and SSR for better crawler access. The skill diagnoses why content isn't being crawled and provides targeted fixes for search and AI bot discovery.
SKILL.md
Rendered from the published skill. Quoted content, verbatim.
SEO Technical: Crawlability
Guides crawlability improvements: robots, X-Robots-Tag, site structure, and internal linking.
When invoking: On first use, if helpful, open with 1–2 sentences on what this skill covers and why it matters, then provide the main output. On subsequent use or when the user asks to skip, go directly to the main output.
Scope (Technical SEO)
- Redirect chains & loops: Fix multi-hop redirects; point directly to final URL
- Broken links (4xx): Fix broken internal/external links; 301 or remove
- Site architecture: Logical hierarchy; pages within 3–4 clicks from homepage
- Orphan pages: Add internal links to pages with no
(truncated - see the full file via the links below)
File tree — 1 file
skills/seo/technical/crawlability/SKILL.md
Let your AI agent find skills like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 56,283 agent skills by what they can do, searchable in plain language.
wish › “Audit and fix crawlability issues affecting search engine and AI bot discovery”
Give your agent the search over MCP, or paste the wish link into any chat. No install? Search from any chat →
Related skills
Selecting the right rendering method directly impacts your site's SEO performance and user experience. This skill guides you through the tradeoffs between static site generation, server-side rendering, client-side rendering, and incremental static regeneration—helping you match each approach to your specific content requirements and business goals.
This skill equips you to systematically evaluate how search engines discover, access, and interpret your web properties. Run comprehensive technical audits that surface crawlability barriers, indexation problems, and rendering gaps—then act on findings to strengthen your search presence.
This skill helps AI agents craft content strategies that rank prominently in generative search engines and AI-powered answer systems. It combines SEO fundamentals with techniques specifically designed for visibility in large language model outputs and AI summaries. Use it to ensure your content gets cited and featured when AI systems generate responses.
Manage crawler permissions and site indexing rules through robots.txt setup and validation. This skill helps you configure directives that guide search engines and AI bots on which content to crawl and index, while auditing existing configurations for compliance and optimization.
geo-technical helps you evaluate your website's technical foundation through the lens of both traditional search engines and AI-powered platforms like ChatGPT, Claude, and Perplexity. It identifies gaps in crawlability, indexing, and content structure that matter to modern discovery systems. Use this skill to shift your optimization strategy toward where traffic is actually heading.
Master the art of arranging images and content in dynamic, responsive masonry grids. This skill equips you with techniques to create visually balanced galleries and portfolio displays that adapt seamlessly across devices. Perfect for designers and developers building modern, gallery-driven interfaces.
More skills Seo Technical (NOASSERTION) · technical-seo-checker (Apache-2.0) · seo-technical (MIT) · technical-seo (MIT) · seo-audit (MIT) · indexing (MIT) · xml-sitemap (MIT)