---
id: firecrawl
version: "4.35.0"
license: MIT License
license_treatment: permissive
maintenance: active
---

# firecrawl — Python SDK for the Firecrawl API: web scraping, crawling, web search, and scientific literature search over a research paper index of PubMed, bioRxiv, medRxiv and arXiv abstracts

License: permissive · Maintenance: active · Downloads: 1.5M/mo

## What it is and what it does

Firecrawl is a Python client for the Firecrawl web-scraping and search service. It wraps HTTP calls to a remote API that handles the complexity of fetching and parsing web content, returning results in multiple formats (Markdown, HTML, structured JSON, video, product data, menu data) or as typed Pydantic models. The SDK supports both synchronous and asynchronous operations, making it suitable for AI agents and data pipelines that need to extract clean, machine-readable content from websites.

Beyond simple scraping, it offers site-wide crawling with depth and URL limits, web search with optional result scraping, and a specialized research-paper index search covering PubMed, bioRxiv, medRxiv, and arXiv abstracts. File parsing (HTML, PDF, DOCX) is supported via multipart upload. Pagination is handled automatically by default but can be controlled manually. The package depends on requests, httpx, websockets, aiohttp, pydantic, python-dotenv, and nest-asyncio for HTTP, async, and environment handling.

Use it for:

- Extract clean Markdown from web pages for ingestion into RAG pipelines or LLM context windows.
- Crawl entire websites to build searchable indexes or knowledge bases with configurable depth and URL limits.
- Search academic papers across PubMed, bioRxiv, medRxiv, and arXiv, then retrieve specific passages answering a research question.
- Scrape product pages to extract structured title, price, availability, and variant data deterministically.
- Parse uploaded HTML, PDF, or DOCX files locally via the parse method for document extraction workflows.
- Perform web search with optional simultaneous scraping of results to gather and process information in one call.

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

A Python SDK for web scraping, crawling, searching, and extracting structured data from websites and research papers via the Firecrawl API, returning results as clean Markdown, HTML, or typed objects.

Yes. Active maintenance (release 2 days old), no known vulnerabilities, low install friction, permissive MIT license, and support for Python 3.8+ make it a safe choice. The package is production-stable and well-suited for AI agents, research workflows, and data extraction pipelines. The main constraint is the dependency on an external Firecrawl API service and its associated costs and rate limits—verify those fit your use case before committing.

## Install

pip install firecrawl
uv add firecrawl
poetry add firecrawl

## Installing firecrawl

Before you install: Low install friction with a pure-Python wheel and common async/HTTP dependencies. Active maintenance with a release 2 days old and 167387 repository stars signal strong ongoing development.

License in practice: MIT License permits free use, modification, and distribution with minimal restrictions—suitable for both open-source and commercial projects.

Quickstart:

from firecrawl import Firecrawl
from firecrawl.types import ScrapeOptions

firecrawl = Firecrawl(api_key="fc-YOUR_API_KEY")
data = firecrawl.scrape('https://firecrawl.dev', formats=['markdown', 'html'])
print(data)

Requires a Firecrawl API key from firecrawl.dev; can be set via FIRECRAWL_API_KEY environment variable or passed directly to the Firecrawl class.

Verify before relying:

- Rate limits or quota restrictions on API calls via the Firecrawl service.
- Latency and reliability characteristics of the underlying Firecrawl API.
- Whether video, product, and menu extraction formats work reliably across all target sites.

## Package facts

- License: MIT License (permissive)
- Python support: supports_current
- Install friction: low
- Maintenance: active
- Downloads: 1.5M/month (top 5,000 on PyPI)
- Known vulnerabilities: none known

## Tags

web scraping SDK, web crawler API client, research paper search, structured data extraction, markdown web scraping, firecrawl API wrapper, academic literature search, web-scraping, research-papers, async-capable

[View on SkillFed](https://skillfed.io/packages/firecrawl) · [View on PyPI](https://pypi.org/project/firecrawl/)