firecrawl-scraping
Extract individual web pages into clean, structured markdown using Firecrawl's API. The skill manages JavaScript rendering, anti-bot defenses, and dynamic content automatically, supporting multiple output formats including markdown, HTML, summaries, and screenshots. Configure proxy modes and timeouts to handle everything from standard sites to protected news content.
Firecrawl Scraping extracts web page content and converts it to clean, LLM-ready markdown while handling JavaScript rendering and anti-bot protection.
AI-generated summary based on this skill's SKILL.md
Install
Casper-Studios/casper-marketplace/firecrawl-scraping · repository language: Python
git clone https://github.com/Casper-Studios/casper-marketplace
cp -r casper-marketplace/plugins/bizdev/research/skills/firecrawl-scraping ~/.claude/skills/firecrawl-scrapingFrequently asked questions
AI-generated answers based on this skill's SKILL.md and metadata
How do I scrape website content with firecrawl?
firecrawl-scraping extracts web pages into clean markdown using its API. Pass a URL to the skill, and it handles JavaScript rendering, anti-bot defenses, and dynamic content automatically. The skill returns structured markdown output ready for processing, with options to configure proxy modes and timeouts based on the target site's protection level.
Can firecrawl-scraping convert web pages to markdown?
Yes. firecrawl-scraping specializes in converting web pages to clean markdown format. It processes HTML, renders JavaScript, and extracts readable content while stripping unnecessary elements. You can also request alternative output formats including raw HTML, summaries, or screenshots depending on your workflow needs.
Does firecrawl-scraping handle JavaScript-heavy websites?
firecrawl-scraping includes built-in JavaScript rendering to scrape dynamic content from sites that load data client-side. The skill manages rendering automatically without requiring additional configuration, making it suitable for modern single-page applications and content-heavy blogs that traditional scrapers cannot access.
How does firecrawl-scraping bypass anti-bot protection?
firecrawl-scraping uses configurable proxy modes and stealth techniques to handle anti-bot defenses on protected sites. You can adjust timeout settings and proxy strategies to work around rate limiting and detection systems. This enables scraping of paywalled articles and news content that blocks standard requests.
What output formats does firecrawl-scraping support?
firecrawl-scraping delivers extracted content in multiple formats: clean markdown (primary), raw HTML, summaries, and screenshots. This flexibility lets you choose the best format for your use case—markdown for LLM processing, HTML for preservation, summaries for quick overviews, or screenshots for visual verification.
Can I automate batch scraping with firecrawl-scraping?
firecrawl-scraping supports API-based automation for scraping multiple pages. You can integrate it into workflows to extract blog posts, documentation, or article collections at scale. Configure proxy and timeout settings once, then apply them across batch jobs for consistent, automated web content extraction.
SKILL.md
rendered from the published skill — quoted content, verbatim
Firecrawl Scraping
Overview
Scrape individual web pages and convert them to clean, LLM-ready markdown. Handles JavaScript rendering, anti-bot protection, and dynamic content.
Quick Decision Tree
What are you scraping?
│
├── Single page (article, blog, docs)
│ └── references/single-page.md
│ └── Script: scripts/firecrawl_scrape.py
│
└── Entire website (multiple pages, crawling)
└── references/website-crawler.md
└── (Use Apify Website Content Crawler for multi-page)
Environment Setup
# Required in .env
FIRECRAWL_API_KEY=fc-your-api-key-here
Get your API key: https://firecrawl.dev/app/api-keys
Common Usage
Simple Scrape
python scripts/firecrawl_scrape.py "https://example.com/article"
With Options
```bash python scripts/firecrawl_scrape.py "https://wsj.com/article" \
(truncated - see the full file via the links below)
Read as markdown · JSON record · Browse the source repository
File tree — 4 files
plugins/bizdev/research/skills/firecrawl-scraping/SKILL.md
plugins/bizdev/research/skills/firecrawl-scraping/references/single-page.md
plugins/bizdev/research/skills/firecrawl-scraping/references/website-crawler.md
plugins/bizdev/research/skills/firecrawl-scraping/scripts/firecrawl_scrape.py