{"enrichment":{"faq":[{"a":"Site Crawler systematically traverses your entire website to uncover all pages, links, and resources in one pass. It's perfect for auditing site architecture, finding orphaned content, or preparing for migrations and SEO analysis. Site Crawler gives you a comprehensive map of your digital property without manual exploration.","q":"What does Site Crawler do?"},{"a":"Yes. Site Crawler is designed to crawl entire websites, discovering all pages and content across your domain. By traversing the full structure of your site, Site Crawler enables you to see every page, link, and resource that exists, making it ideal for complete site audits and comprehensive content inventories.","q":"Can Site Crawler crawl an entire website?"},{"a":"Site Crawler automatically extracts links and URLs as it traverses your website. During the crawling process, Site Crawler identifies and collects all URLs found on each page, allowing you to analyze link patterns, discover broken links, and map your site's connectivity for SEO and structural analysis.","q":"How does Site Crawler extract links and URLs from a website?"},{"a":"Site Crawler maps your website's structure and navigation hierarchy by systematically exploring all pages and their relationships. This structural mapping helps you understand how your content is organized, identify navigation gaps, and visualize the complete architecture of your site\u2014essential for planning redesigns, migrations, or SEO optimization.","q":"What is Site Crawler useful for in terms of website structure mapping?"},{"a":"Yes. Site Crawler automates website exploration and data collection, eliminating the need for manual page-by-page review. By running automated crawls, Site Crawler efficiently discovers all pages, extracts content metadata, and compiles comprehensive reports\u2014saving time and ensuring no content is overlooked during site audits or content inventories.","q":"Can Site Crawler automate website exploration and data collection?"},{"a":"Site Crawler supports website monitoring and auditing by systematically scanning your site to identify all pages and content. Use Site Crawler to detect orphaned pages, verify content presence, audit navigation paths, and ensure your site structure remains healthy\u2014making it invaluable for ongoing site maintenance and compliance checks.","q":"How can Site Crawler help with website monitoring and auditing?"}],"shadow_tags":["web-automation","link-discovery","content-indexing","url-extraction","site-mapping","page-traversal","web-reconnaissance","domain-analysis"],"summary_rewrite":"Site Crawler systematically traverses your entire website to uncover all pages, links, and resources in one pass. Perfect for auditing site architecture, finding orphaned content, or preparing for migrations and SEO analysis. Get a comprehensive map of your digital property without manual exploration."},"gist":{"api_url":"https://skillfed.io/api/skills/mindmorass/reflex/site-crawler.json","as_of":"2026-02-12","description":"Site Crawler systematically traverses your entire website to uncover all pages, links. npx skillfed install mindmorass/reflex/site-crawler","install":{"manual":["git clone https://github.com/mindmorass/reflex","cp -r reflex ~/.claude/skills/site-crawler"],"primary":"npx skillfed install mindmorass/reflex/site-crawler","version":"367c4d6a"},"kind":"skill","mirror_url":"https://skillfed.io/mindmorass/reflex/site-crawler.md","similar":[{"id":"jamditis/claude-skills-journalism/web-scraping","name":"web-scraping","publisher":"jamditis/claude-skills-journalism","url":"https://skillfed.io/jamditis/claude-skills-journalism/web-scraping"},{"id":"Mindrally/skills/beautifulsoup-parsing","name":"beautifulsoup-parsing","publisher":"Mindrally/skills","url":"https://skillfed.io/Mindrally/skills/beautifulsoup-parsing"},{"id":"indranilbanerjee/digital-marketing-pro/technical-seo","name":"technical-seo","publisher":"indranilbanerjee/digital-marketing-pro","url":"https://skillfed.io/indranilbanerjee/digital-marketing-pro/technical-seo"},{"id":"alinaqi/maggy/site-architecture","name":"site-architecture","publisher":"alinaqi/maggy","url":"https://skillfed.io/alinaqi/maggy/site-architecture"}],"title":"Site Crawler by mindmorass: Crawl and index an entire \u2014 SkillFed","use":{"when":["Yes.","Site Crawler automatically extracts links and URLs as it traverses your website."]},"what":{"lead":"Site Crawler systematically traverses your entire website to uncover all pages, links, and resources in one pass. It's perfect for auditing site architecture, finding orphaned content, or preparing for migrations and SEO analysis. Site Crawler gives you a comprehensive map of your digital property without manual exploration."}},"id":"mindmorass/reflex/site-crawler","install":{"mode":"external","repo":"https://github.com/mindmorass/reflex"},"links":{"html":"https://skillfed.io/mindmorass/reflex/site-crawler","md":"https://skillfed.io/mindmorass/reflex/site-crawler.md","repo":"https://github.com/mindmorass/reflex"},"meta":{"agents_supported":[],"first_seen":"2026-07-28","forks":0,"language":"Python","last_updated":"2026-02-12","license":null,"name":"Site Crawler","publisher":"mindmorass","stars":2},"relations":{"similar":[{"id":"mindmorass/reflex/knowledge-ingestion-patterns"},{"id":"jamditis/claude-skills-journalism/web-scraping"},{"id":"johnlindquist/claude/spider"},{"id":"KnockOutEZ/wigolo/wigolo-crawl"},{"id":"nicepkg/auto-company/web-scraping"},{"id":"browser-act/skills/webcrawler-deep-crawl"},{"id":"Mindrally/skills/beautifulsoup-parsing"},{"id":"indranilbanerjee/digital-marketing-pro/technical-seo"},{"id":"Aradotso/marketing-skills/seo-content-marketing-skill-suite"},{"id":"alinaqi/maggy/site-architecture"}]},"slug":{"owner":"mindmorass","repo":"reflex","skill":"site-crawler"},"version":"367c4d6a"}