Site Crawler
Site Crawler systematically traverses your entire website to uncover all pages, links, and resources in one pass. Perfect for auditing site architecture, finding orphaned content, or preparing for migrations and SEO analysis. Get a comprehensive map of your digital property without manual exploration.
Site Crawler systematically traverses your entire website to uncover all pages, links, and resources in one pass. It's perfect for auditing site architecture, finding orphaned content, or preparing for migrations and SEO analysis. Site Crawler gives you a comprehensive map of your digital property without manual exploration.
AI-generated summary based on this skill's SKILL.md
Decision gist · record as of 2026-02-12
Site Crawler systematically traverses your entire website to uncover all pages, links, and resources in one pass. It's perfect for auditing site architecture, finding orphaned content, or preparing for migrations and SEO analysis. Site Crawler gives you a comprehensive map of your digital property without manual exploration.
Use it when
- Yes.
- Site Crawler automatically extracts links and URLs as it traverses your website.
Install
mindmorass/reflex/site-crawler · repository language: Python
generated, unverified - the skill's exact subdirectory could not be determined; check the repository on GitHub
Open directory. Skills are indexed for reading, not audited. Review a skill's body before installing it.
Frequently asked questions
AI-generated answers based on this skill's SKILL.md and metadata
What does Site Crawler do?
Site Crawler systematically traverses your entire website to uncover all pages, links, and resources in one pass. It's perfect for auditing site architecture, finding orphaned content, or preparing for migrations and SEO analysis. Site Crawler gives you a comprehensive map of your digital property without manual exploration.
Can Site Crawler crawl an entire website?
Yes. Site Crawler is designed to crawl entire websites, discovering all pages and content across your domain. By traversing the full structure of your site, Site Crawler enables you to see every page, link, and resource that exists, making it ideal for complete site audits and comprehensive content inventories.
How does Site Crawler extract links and URLs from a website?
Site Crawler automatically extracts links and URLs as it traverses your website. During the crawling process, Site Crawler identifies and collects all URLs found on each page, allowing you to analyze link patterns, discover broken links, and map your site's connectivity for SEO and structural analysis.
What is Site Crawler useful for in terms of website structure mapping?
Site Crawler maps your website's structure and navigation hierarchy by systematically exploring all pages and their relationships. This structural mapping helps you understand how your content is organized, identify navigation gaps, and visualize the complete architecture of your site—essential for planning redesigns, migrations, or SEO optimization.
Can Site Crawler automate website exploration and data collection?
Yes. Site Crawler automates website exploration and data collection, eliminating the need for manual page-by-page review. By running automated crawls, Site Crawler efficiently discovers all pages, extracts content metadata, and compiles comprehensive reports—saving time and ensuring no content is overlooked during site audits or content inventories.
How can Site Crawler help with website monitoring and auditing?
Site Crawler supports website monitoring and auditing by systematically scanning your site to identify all pages and content. Use Site Crawler to detect orphaned pages, verify content presence, audit navigation paths, and ensure your site structure remains healthy—making it invaluable for ongoing site maintenance and compliance checks.
Let your AI agent find skills like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 56,283 agent skills by what they can do, searchable in plain language.
wish › “Crawl and index an entire website to discover all pages and content”
Give your agent the search over MCP, or paste the wish link into any chat. No install? Search from any chat →
Related skills
This skill teaches you BeautifulSoup's core parsing capabilities for extracting data from HTML and XML documents. You'll learn how to navigate document trees, select elements efficiently, and transform raw markup into structured data for your Python projects.
This skill enables reliable extraction of public web content using layered fallback approaches and robust error management. Built for journalists, researchers, and media professionals who need dependable data collection workflows. Integrates seamlessly with Claude-based journalism and academic research pipelines.
This skill provides a systematic approach to identifying and resolving Core Web Vitals performance bottlenecks that impact user experience and SEO rankings. Run structured diagnostics to pinpoint LCP delays, INP responsiveness problems, and CLS layout shifts, then apply targeted fixes across your site's critical pages.
Build a solid SEO foundation by implementing core technical elements like robots.txt directives, XML sitemaps, and structured meta tags. This skill guides you through configuring each component to ensure search engines can discover and properly index your site's content. Establish best practices that support long-term organic visibility.
Web Recon automates reconnaissance workflows for identifying hidden web content, detecting technology stacks, and discovering virtual hosts. It orchestrates multiple specialized tools—from WAF detection and HTTP probing to recursive directory scanning and application crawling—to build a comprehensive picture of web targets.
scrapling enables automated data extraction from sites defended by Cloudflare and similar anti-scraping systems. It uses sophisticated evasion methods to bypass protection layers and retrieve content that would otherwise be inaccessible to standard scrapers. Ideal for researchers and developers who need reliable access to protected web data.
More skills Firecrawl Crawl (unlicensed)