firecrawl

What it is and what it does

Firecrawl is a Python client for the Firecrawl web-scraping and search service. It wraps HTTP calls to a remote API that handles the complexity of fetching and parsing web content, returning results in multiple formats (Markdown, HTML, structured JSON, video, product data, menu data) or as typed Pydantic models. The SDK supports both synchronous and asynchronous operations, making it suitable for AI agents and data pipelines that need to extract clean, machine-readable content from websites.

Beyond simple scraping, it offers site-wide crawling with depth and URL limits, web search with optional result scraping, and a specialized research-paper index search covering PubMed, bioRxiv, medRxiv, and arXiv abstracts. File parsing (HTML, PDF, DOCX) is supported via multipart upload. Pagination is handled automatically by default but can be controlled manually. The package depends on requests, httpx, websockets, aiohttp, pydantic, python-dotenv, and nest-asyncio for HTTP, async, and environment handling.

Use it for:

• Extract clean Markdown from web pages for ingestion into RAG pipelines or LLM context windows.
• Crawl entire websites to build searchable indexes or knowledge bases with configurable depth and URL limits.
• Search academic papers across PubMed, bioRxiv, medRxiv, and arXiv, then retrieve specific passages answering a research question.
• Scrape product pages to extract structured title, price, availability, and variant data deterministically.
• Parse uploaded HTML, PDF, or DOCX files locally via the parse method for document extraction workflows.
• Perform web search with optional simultaneous scraping of results to gather and process information in one call.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

A Python SDK for web scraping, crawling, searching, and extracting structured data from websites and research papers via the Firecrawl API, returning results as clean Markdown, HTML, or typed objects.

Yes. Active maintenance (release 2 days old), no known vulnerabilities, low install friction, permissive MIT license, and support for Python 3.8+ make it a safe choice. The package is production-stable and well-suited for AI agents, research workflows, and data extraction pipelines. The main constraint is the dependency on an external Firecrawl API service and its associated costs and rate limits—verify those fit your use case before committing.

Install

firecrawl on PyPI

pip install firecrawl

uv add firecrawl

poetry add firecrawl

Installing firecrawl

Before you install

Low install friction with a pure-Python wheel and common async/HTTP dependencies. Active maintenance with a release 2 days old and 167387 repository stars signal strong ongoing development.

License in practice

MIT License permits free use, modification, and distribution with minimal restrictions—suitable for both open-source and commercial projects.

Quickstart

from firecrawl import Firecrawl
from firecrawl.types import ScrapeOptions

firecrawl = Firecrawl(api_key="fc-YOUR_API_KEY")
data = firecrawl.scrape('https://firecrawl.dev', formats=['markdown', 'html'])
print(data)

Requires a Firecrawl API key from firecrawl.dev; can be set via FIRECRAWL_API_KEY environment variable or passed directly to the Firecrawl class.

Verify before relying

• Rate limits or quota restrictions on API calls via the Firecrawl service.
• Latency and reliability characteristics of the underlying Firecrawl API.
• Whether video, product, and menu extraction formats work reliably across all target sites.

Package facts

Evidence: firecrawl-4.35.0-py3-none-any.whl

Keywords: SDK, API, firecrawl, web scraping, crawler, web search, literature search, research, scientific papers, academic search, biomedical, life sciences, pubmed, biorxiv, medrxiv, arxiv, preprints, citations, bioinformatics

Tags

More Software Development packages