---
id: Casper-Studios/casper-marketplace/firecrawl-scraping
version: "e48d06e0"
license: MIT
install: manual
updated: 2026-07-27
---

# firecrawl-scraping — Extract individual web pages into clean, structured markdown using Firecrawl's API. The skill manages JavaScript rendering, anti-bot defenses, and dynamic content automatically, supporting multiple output formats including markdown, HTML, summaries, and screenshots. Configure proxy modes and timeouts to handle everything from standard sites to protected news content.

Publisher: Casper-Studios · Stars: 11 · Updated: 2026-07-27

Install (manual): `git clone https://github.com/Casper-Studios/casper-marketplace`

## SKILL.md

# Firecrawl Scraping

## Overview

Scrape individual web pages and convert them to clean, LLM-ready markdown. Handles JavaScript rendering, anti-bot protection, and dynamic content.

## Quick Decision Tree

```
What are you scraping?
│
├── Single page (article, blog, docs)
│   └── references/single-page.md
│   └── Script: scripts/firecrawl_scrape.py
│
└── Entire website (multiple pages, crawling)
    └── references/website-crawler.md
    └── (Use Apify Website Content Crawler for multi-page)
```

## Environment Setup

```bash
# Required in .env
FIRECRAWL_API_KEY=fc-your-api-key-here
```

Get your API key: https://firecrawl.dev/app/api-keys

## Common Usage

### Simple Scrape
```bash
python scripts/firecrawl_scrape.py "https://example.com/article"
```

### With Options
```bash
python scripts/firecrawl_scrape.py "https://wsj.com/article" \
  --proxy stealth \
  --format markdown summary \
  --timeout 60000
```

## Proxy Modes

| Mode | Use Case |
|------|----------|
| `basic` | Standard sites, fastest |
| `stealth` | Anti-bot protection, premium content (WSJ, NYT) |
| `auto` | Let Firecrawl decide (recommended) |

## Output Formats

- `markdown` - Clean markdown content (default)
- `html` - Raw HTML
- `summary` - AI-generated summary
- `screenshot` - Page screenshot
- `links` - All links on page

## Cost

~1 credit per page. Stealth proxy may use additional credits.

## Security Notes

### Credential Handling
- Store `FIRECRAWL_API_KEY` in `.env` file (never commit to git)
- API keys can be regenerated at https://firecrawl.dev/app/api-keys
- Never log or print API keys in script output
- Use environment variables, not hardcoded values

### Data Privacy
- Only scrapes publicly accessible web pages
- Scraped content is processed by Firecrawl servers temporarily
- Markdown output stored locally in `.tmp/` directory
- Screenshots (if requested) are stored locally
- No persistent data retention by Firecrawl after request

### Access Scopes
- API key provides full access to scraping features
- No granular permission scopes available
- Monitor usage via Firecrawl dashboard

### Compliance Considerations
- **Robots.txt**: Firecrawl respects robots.txt by default
- **Public Content Only**: Only scrape publicly accessible pages
- **Terms of Service**: Respect target site ToS
- **Rate Limiting**: Built-in rate limiting prevents abuse
- **Stealth Proxy**: Use stealth mode only when necessary (paywalled news, not auth bypass)
- **GDPR**: Scraped content may contain PII - handle accordingly
- **Copyright**: Respect intellectual property rights of scraped content

## Troubleshooting

### Common Issues

#### Issue: Credits exhausted
**Symptoms:** API returns "insufficient credits" or quota exceeded error
**Cause:** Account credits depleted
**Solution:**
- Check credit balance at https://firecrawl.dev/app
- Upgrade plan or purchase additional credits
- Reduce scraping frequency
- Use `basic` proxy mode to conserve credits

#### Issue: Page not rendering correctly
**Symptoms:** Empty content or partial HTML returned
**Cause:** JavaScript-heavy page not fully loading
**Solution:**
- Enable JavaScript rendering with `--js-render` flag
- Increase timeout with `--timeout 60000` (60 seconds)
- Try `stealth` proxy mode for protected sites
- Wait for specific elements with `--wait-for` selector

#### Issue: 403 Forbidden error
**Symptoms:** Script returns 403 status code
**Cause:** Site blocking automated access
**Solution:**
- Enable `stealth` proxy mode
- Add delay between requests
- Try at different times (some sites rate limit by time)
- Check if site requires login (not supported)

#### Issue: Empty markdown output
**Symptoms:** Scrape succeeds but markdown is empty or malformed
**Cause:** Dynamic content loaded after page load, or unusual page structure
**Solution:**
- Increase wait time for JavaScript to execute
- Use `--wait-for` to wait for specific content
- Try `html` format to see raw content
- Check if content is in an iframe (not always supported)

#### Issue: Timeout errors
**Symptoms:** Request times out before completion
**Cause:** Slow page load or large page content
**Solution:**
- Increase timeout value (up to 120000ms)
- Use `basic` proxy for faster response
- Target specific page sections if possible
- Check if site is experiencing issues

## Resources

- **references/single-page.md** - Single page scraping details
- **references/website-crawler.md** - Multi-page website crawling

## Integration Patterns

### Scrape and Analyze
**Skills:** firecrawl-scraping → parallel-research
**Use case:** Scrape competitor pages, then analyze content strategy
**Flow:**
1. Scrape competitor website pages with Firecrawl
2. Convert to clean markdown
3. Use parallel-research to analyze positioning, messaging, features

### Scrape and Document
**Skills:** firecrawl-scraping → content-generation
**Use case:** Create summary documents from web research
**Flow:**
1. Scrape multiple article pages on a topic
2. Combine markdown content
3. Generate summary document via content-generation

### Scrape and Enrich CRM
**Skills:** firecrawl-scraping → attio-crm
**Use case:** Enrich company records with website data
**Flow:**
1. Scrape company website (about page, team page, product pages)
2. Extract key information (funding, team size, products)
3. Update company record in Attio CRM with enriched data


[View on SkillFed](https://skillfed.io/Casper-Studios/casper-marketplace/firecrawl-scraping) · [View on GitHub](https://github.com/Casper-Studios/casper-marketplace)