firecrawl-py

What it is and what it does

Firecrawl-py is a Python client for the Firecrawl web scraping and search service. It provides methods to scrape individual URLs, crawl entire websites, search the web, and query a research paper index covering PubMed, bioRxiv, medRxiv, and arXiv abstracts. The library returns content in multiple formats—Markdown, HTML, video, product data, menu data—and supports both synchronous and asynchronous operations. It depends on requests, httpx, websockets, aiohttp, pydantic, python-dotenv, and nest-asyncio for HTTP, async, and configuration handling.

The package is designed for AI agents and applications that need to extract structured or semi-structured data from web pages or search academic literature. It handles pagination automatically, supports interactive browsing actions, and provides specialized extraction for product pages and restaurant menus. The research paper search methods return raw JSON with camelCase keys, distinct from the rest of the SDK's snake_case convention.

Use it for:

• Scrape product pages to extract title, price, availability, and variants deterministically.
• Crawl a website up to a specified depth and format limit to build a searchable knowledge base.
• Search academic literature by querying abstracts from biomedical and physics preprint repositories.
• Extract Markdown from web pages for ingestion into RAG (retrieval-augmented generation) pipelines.
• Map a website's URL structure with optional sitemap and subdomain inclusion for discovery.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Client library for the Firecrawl API that scrapes, crawls, and searches the web, returning clean Markdown or structured data; also indexes research papers from PubMed, bioRxiv, medRxiv, and arXiv.

Yes. The package is actively maintained, has no known vulnerabilities, uses a permissive MIT license, and installs with low friction. It is well-suited for developers building AI agents or applications that need web scraping, crawling, or research paper search. The primary constraint is the requirement for a Firecrawl API key and service account.

Install

firecrawl-py on PyPI

pip install firecrawl-py

uv add firecrawl-py

poetry add firecrawl-py

Installing firecrawl-py

Before you install

Low install friction; pure Python wheel with common async and HTTP dependencies. Active maintenance with a release 2 days old and high repository activity (167374 stars).

License in practice

MIT License permits commercial and private use with minimal restrictions; suitable for most projects.

Quickstart

pip install firecrawl-py

from firecrawl import Firecrawl
firecrawl = Firecrawl(api_key="fc-YOUR_API_KEY")
data = firecrawl.scrape('https://example.com', formats=['markdown', 'html'])
print(data)

Requires a Firecrawl API key from firecrawl.dev; can be set via FIRECRAWL_API_KEY environment variable or passed to the Firecrawl constructor.

Verify before relying

• Whether the paper index (~43M abstracts) is kept current and how frequently it is updated.
• Rate limits and quota behavior for the Firecrawl API service itself.
• Performance characteristics for large-scale crawls or batch scrapes.

Package facts

Evidence: firecrawl_py-4.35.0-py3-none-any.whl

Keywords: SDK, API, firecrawl, web scraping, crawler, web search, literature search, research, scientific papers, academic search, biomedical, life sciences, pubmed, biorxiv, medrxiv, arxiv, preprints, citations, bioinformatics

Tags

More Software Development packages