site-crawlability
This skill helps you identify and resolve technical barriers preventing search engines and AI crawlers from accessing your web pages. Run diagnostics on robots.txt, sitemap configuration, and indexation blockers, then apply targeted fixes to improve discoverability across search and AI platforms.
site-crawlability helps you identify and resolve technical barriers preventing search engines and AI crawlers from accessing your web pages. Start by auditing your robots.txt and sitemap configuration, then fix indexation blockers like broken internal links, orphan pages with no incoming links, and redirect chains. Optimize your site structure to reduce crawl budget waste and ensure all important content is reachable within 3 clicks from your homepage.
AI-generated summary based on this skill's SKILL.md
Install
kostja94/marketing-skills/crawlability
git clone https://github.com/kostja94/marketing-skills
cp -r marketing-skills/skills/seo/technical/crawlability ~/.claude/skills/crawlabilityFrequently asked questions
AI-generated answers based on this skill's SKILL.md and metadata
How do I improve my site's crawlability for search engines?
site-crawlability helps you identify and resolve technical barriers preventing search engines and AI crawlers from accessing your web pages. Start by auditing your robots.txt and sitemap configuration, then fix indexation blockers like broken internal links, orphan pages with no incoming links, and redirect chains. Optimize your site structure to reduce crawl budget waste and ensure all important content is reachable within 3 clicks from your homepage.
What are orphan pages and how do I fix them for SEO?
site-crawlability identifies orphan pages—content with no incoming internal links that crawlers struggle to discover. Fix them by adding internal links from relevant pages, updating your sitemap to include them, or consolidating orphan content into existing pages. Resolving orphan pages improves indexation and ensures crawlers can reach all your valuable content.
How can site-crawlability help optimize my crawl budget?
site-crawlability optimizes site structure and internal linking to reduce crawl budget waste. It helps you eliminate redirect chains, fix broken internal links, and streamline your site architecture so search engines spend their crawl budget on important pages rather than errors or redundant URLs. This is especially critical for large sites where crawl efficiency directly impacts indexation.
Should I use infinite scroll or pagination for AI crawler optimization?
site-crawlability recommends pagination with rel="prev" and rel="next" tags for better AI crawler and search engine visibility. Infinite scroll and masonry layouts create crawl traps that prevent bots from discovering all content. Pagination ensures crawlers can navigate through your pages systematically and index all content without hitting URL generation traps.
How does site-crawlability help with AI bot indexation?
site-crawlability diagnoses why content isn't being indexed by AI crawlers like GPTBot and ClaudeBot. It implements server-side rendering (SSR) for better content visibility, optimizes your sitemap and canonical URLs, and fixes crawlable content issues. These improvements help AI platforms discover and index your pages alongside traditional search engines.
What technical issues does site-crawlability audit and fix?
site-crawlability audits robots.txt configuration, broken internal links, redirect chains, orphan pages, sitemap errors, and crawl traps. It resolves indexation blockers, optimizes site structure to improve discoverability, and implements pagination and SSR for better crawler access. The skill diagnoses why content isn't being crawled and provides targeted fixes for search and AI bot discovery.
SKILL.md
rendered from the published skill — quoted content, verbatim
SEO Technical: Crawlability
Guides crawlability improvements: robots, X-Robots-Tag, site structure, and internal linking.
When invoking: On first use, if helpful, open with 1–2 sentences on what this skill covers and why it matters, then provide the main output. On subsequent use or when the user asks to skip, go directly to the main output.
Scope (Technical SEO)
- Redirect chains & loops: Fix multi-hop redirects; point directly to final URL
- Broken links (4xx): Fix broken internal/external links; 301 or remove
- Site architecture: Logical hierarchy; pages within 3–4 clicks from homepage
- Orphan pages: Add internal links to pages with no
(truncated - see the full file via the links below)
Read as markdown · JSON record · Browse the source repository
File tree — 1 file
skills/seo/technical/crawlability/SKILL.md