---
id: fastwarc
version: "1.0.9"
license: Apache-2.0
license_treatment: permissive
maintenance: active
---

# FastWARC — The world's fastest WARC parsing library written in Rust with bindings for Python.

License: permissive · Maintenance: active · Downloads: 1.4M/mo

## What it is and what it does

FastWARC is a Python library that reads WARC archive files—the standard format for storing web crawl data—using a Rust implementation for speed. It handles both compressed (Gzip, Zstd, LZ4) and uncompressed WARC/1.0 and WARC/1.1 streams. The library is part of the ChatNoir Resiliparse toolkit for web data processing and is designed for workloads that need to ingest and parse large volumes of web archive data efficiently.

You install it via pip and import it to iterate over records in WARC files. It depends on brotli, click, tqdm, and typing_extensions. The package requires Python 3.10 or later and is actively maintained with recent releases and no known security issues.

Use it for:

- Parse Common Crawl WARC dumps or other web archive collections for research or data extraction.
- Build web crawl pipelines that need to read compressed WARC files without decompressing to disk first.
- Extract HTTP responses, metadata, or payloads from archived web data at scale.
- Integrate WARC parsing into information retrieval or web analytics workflows.
- Process historical web snapshots stored in WARC format for archival or compliance tasks.

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

FastWARC parses WARC (Web ARChive) files—compressed or uncompressed—at high speed using Rust-backed bindings, supporting WARC/1.0 and WARC/1.1 with Gzip, Zstd, and LZ4 compression.

Yes. FastWARC is actively maintained, has no known vulnerabilities, is licensed permissively, and offers pre-built wheels for modern Python versions on all major platforms. Install it if you work with WARC files or web archives; the Rust backend and broad compression support make it a solid choice for this specific task.

## Install

pip install fastwarc
uv add fastwarc
poetry add fastwarc

## Installing FastWARC

Before you install: Pre-built wheels cover modern Python versions and major platforms (macOS, Linux, Windows), reducing compile friction. The package is actively maintained with a recent release and no known vulnerabilities.

License in practice: Apache-2.0 is permissive; you may use, modify, and distribute FastWARC freely in proprietary or open-source projects with minimal restriction.

Quickstart:

pip install fastwarc

from fastwarc.warc import ArchiveIterator

with open('archive.warc.gz', 'rb') as f:
    for record in ArchiveIterator(f):
        print(record.headers)

Requires Python 3.10 or later; pre-built wheels available for macOS (arm64, x86_64), Linux (aarch64, x86_64), and Windows (amd64).

Verify before relying:

- Exact performance improvement over pure-Python WARC parsers is not quantified in the fact sheet.
- Memory overhead or streaming behavior for very large WARC files is not documented in the excerpt.
- Whether the library supports incremental parsing or requires loading entire records into memory.

## Package facts

- License: Apache-2.0 (permissive)
- Python support: supports_current
- Install friction: medium
- Maintenance: active
- Downloads: 1.4M/month (top 5,000 on PyPI)
- Known vulnerabilities: none known

## Tags

warc file parsing, web archive reader, warc compression support, rust warc library, common crawl processing, web data extraction, archive stream parsing, web-archiving, rust-bindings, data-processing

[View on SkillFed](https://skillfed.io/packages/fastwarc) · [View on PyPI](https://pypi.org/project/fastwarc/)