warc3-wet

What it is and what it does

warc3-wet is a Python library for reading and parsing WARC and WET archive files—the standard formats used to store web crawls. It provides a simple interface to open these files and iterate over records, extracting metadata headers like WARC-Target-URI and Content-Length. The library is a Python 3 port of an older warc package; this fork adds support for WET files and seeking within archives.

The package has no external runtime dependencies, making it lightweight to install. It is classified as Beta-status software. The repository remains accessible but has not seen a release in over two years, suggesting the package is in a stable, maintenance-only state.

Use it for:

• Extract and process records from web crawl archives stored in WARC format for data analysis or research.
• Parse WET files to access extracted text content from archived web pages.
• Build data pipelines that read WARC archives and filter or transform records based on metadata.
• Access specific records within large WARC files using seek functionality for efficient random access.
• Integrate web archive data into workflows that require crawled web content.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Read and iterate over WARC (Web ARChive) and WET (WARC Extracted Text) files, extracting record metadata and content from web crawl archives.

Yes, if you need to work with WARC or WET files. The package is stable, has no dependencies, and handles a specific file format well. The dormant maintenance status is not a blocker for read-only use, but verify GPLv2 compatibility with your project first. No known security vulnerabilities.

Install

warc3-wet on PyPI

pip install warc3-wet

uv add warc3-wet

poetry add warc3-wet

Installing warc3-wet

Before you install

Low install friction with no runtime dependencies. Maintenance is dormant—last release was 758 days ago—but the repository remains active.

License in practice

GPLv2 copyleft license means any derivative work or distribution must also be licensed under GPLv2 and have source code available; verify compatibility with your project's licensing before use.

Quickstart

pip install warc3-wet

with warc3_wet.open("test.warc") as f:
    for record in f:
        print(record['WARC-Target-URI'], record['Content-Length'])

Verify before relying

• Whether Python 3.10+ is fully supported (changelog mentions 3.10 compatibility in 0.2.4, but requires_python is unspecified)
• Current stability and whether dormant status indicates maintenance-only mode or active development

Package facts

Evidence: warc3_wet-0.2.5-py3-none-any.whl

Tags

More WWW/HTTP packages