warc3-wet-clueweb09

What it is and what it does

warc3-wet-clueweb09 is a Python library for reading WARC (Web ARChive) and WET (WARC Extracted Text) files, which are standard formats for storing web crawls. It provides an interface to iterate over archive records and access their metadata fields and content. The package is a Python 3 port of an older library, with modifications to handle parsing issues specific to ClueWeb09 archives.

The library has no runtime dependencies and is designed for straightforward use: open an archive file, iterate over records, and extract fields like WARC-Target-URI and Content-Length. However, the package is abandoned (last updated 2020-12-07 with no commits since 2021-12-06), so it receives no maintenance, bug fixes, or updates for compatibility with newer Python versions.

Use it for:

• Parse ClueWeb09 web crawl archives to extract URLs and content metadata for research or analysis.
• Iterate over WARC files to build indexes or extract specific records for information retrieval tasks.
• Read WET files to access extracted text content from archived web pages.
• Migrate legacy Python 2 WARC processing pipelines to Python 3 using this ported library.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Reads and iterates over WARC and WET archive files, extracting records with their metadata and content for web crawl analysis.

No, unless you have a specific, narrow need to read ClueWeb09 archives and cannot use an actively maintained alternative. The package is abandoned, Python version support is unspecified, and GPLv2 copyleft licensing restricts use in proprietary projects. High install friction and no maintenance path make it a poor choice for new projects.

Install

warc3-wet-clueweb09 on PyPI

pip install warc3-wet-clueweb09

uv add warc3-wet-clueweb09

poetry add warc3-wet-clueweb09

Installing warc3-wet-clueweb09

Before you install

Installation requires building from source with no runtime dependencies, which adds friction. The package is abandoned (last release 2020-12-07, no commits since 2021-12-06), so expect no maintenance or bug fixes going forward.

License in practice

Licensed under GPLv2 (copyleft), which requires that any derivative work or distribution must also be open-source under compatible terms—a significant constraint for proprietary or closed-source projects.

Quickstart

pip install warc3-wet-clueweb09

with warc3_wet_clueweb09.open("test.warc") as f:
    for record in f:
        print(record['WARC-Target-URI'], record['Content-Length'])

Python version support is unspecified; verify compatibility with your Python version before relying on this abandoned package.

Verify before relying

• Whether the package actually works with modern Python versions (support unspecified in metadata).
• Extent of ClueWeb09-specific fixes and whether they address all known parsing issues.
• Whether seeking support (mentioned in changelog for 0.2.3) is fully functional.

Package facts

Evidence: warc3-wet-clueweb09-0.2.5.tar.gz

Tags

More WWW/HTTP packages