warcio

What it is and what it does

Warcio is a streaming library for reading and writing WARC and ARC archive formats used in web preservation. It provides fast, low-level access to web archival content through an iterator-based API, allowing you to process records one at a time without loading entire archives into memory. The library supports both WARC 1.0 and WARC 1.1 ISO standards, handles on-the-fly ARC-to-WARC conversion, and automatically decompresses and de-chunks HTTP payloads. It also includes HTTP capture functionality that monkey-patches Python's http.client to write requests and responses directly to WARC files, integrating well with the requests library.

The package is designed for web archivists, digital preservation workflows, and applications that need to read or create web archives. It originated as a component of the Webrecorder project and is maintained as a standalone library. With only six as an external dependency and pure Python distribution, it installs cleanly across Python 3.8+ versions.

Use it for:

• Extract URLs and content from archived WARC files for analysis or migration to other systems
• Capture HTTP/HTTPS traffic from web scraping scripts directly into WARC files for reproducible archiving
• Convert legacy ARC archives to modern WARC format while preserving record metadata
• Stream remote WARC files over HTTP without downloading them entirely to disk first
• Build web archival pipelines that read, filter, and rewrite WARC records in a single pass

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Reads and writes WARC (Web ARChive) and legacy ARC format files used in web archiving, with streaming support for both local and remote archives.

Yes. Warcio is production-stable, actively maintained, has no known vulnerabilities, and installs with minimal friction. Install it if you need to read or write WARC/ARC archives or capture HTTP traffic into web archives.

Install

warcio on PyPI

pip install warcio

uv add warcio

poetry add warcio

Installing warcio

Before you install

Low friction: pure Python wheel with only six as a runtime dependency. Active maintenance with a recent release (136 days ago) and steady repository activity.

License in practice

Apache 2.0 permissive license allows commercial and private use with minimal restrictions—suitable for most projects.

Quickstart

from warcio.archiveiterator import ArchiveIterator

with open('path/to/file', 'rb') as stream:
    for record in ArchiveIterator(stream):
        if record.rec_type == 'response':
            print(record.rec_headers.get_header('WARC-Target-URI'))

Verify before relying

• Whether the package supports Python 3.7 as stated in the description, given classifiers list Python 3.8 as minimum

Package facts

Evidence: warcio-1.8.1-py2.py3-none-any.whl

Tags

More Python Modules packages