---
id: warc3-wet-clueweb09
version: "0.2.5"
license: GPLv2
license_treatment: copyleft
maintenance: abandoned
---

# warc3-wet-clueweb09 — Python library to work with ARC and WARC files, with fixes for ClueWeb09

License: copyleft · Maintenance: abandoned · Downloads: 142.3K/mo

## What it is and what it does

warc3-wet-clueweb09 is a Python library for reading WARC (Web ARChive) and WET (WARC Extracted Text) files, which are standard formats for storing web crawls. It provides an interface to iterate over archive records and access their metadata fields and content. The package is a Python 3 port of an older library, with modifications to handle parsing issues specific to ClueWeb09 archives.

The library has no runtime dependencies and is designed for straightforward use: open an archive file, iterate over records, and extract fields like WARC-Target-URI and Content-Length. However, the package is abandoned (last updated 2020-12-07 with no commits since 2021-12-06), so it receives no maintenance, bug fixes, or updates for compatibility with newer Python versions.

Use it for:

- Parse ClueWeb09 web crawl archives to extract URLs and content metadata for research or analysis.
- Iterate over WARC files to build indexes or extract specific records for information retrieval tasks.
- Read WET files to access extracted text content from archived web pages.
- Migrate legacy Python 2 WARC processing pipelines to Python 3 using this ported library.

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

Reads and iterates over WARC and WET archive files, extracting records with their metadata and content for web crawl analysis.

No, unless you have a specific, narrow need to read ClueWeb09 archives and cannot use an actively maintained alternative. The package is abandoned, Python version support is unspecified, and GPLv2 copyleft licensing restricts use in proprietary projects. High install friction and no maintenance path make it a poor choice for new projects.

## Install

pip install warc3-wet-clueweb09
uv add warc3-wet-clueweb09
poetry add warc3-wet-clueweb09

## Installing warc3-wet-clueweb09

Before you install: Installation requires building from source with no runtime dependencies, which adds friction. The package is abandoned (last release 2020-12-07, no commits since 2021-12-06), so expect no maintenance or bug fixes going forward.

License in practice: Licensed under GPLv2 (copyleft), which requires that any derivative work or distribution must also be open-source under compatible terms—a significant constraint for proprietary or closed-source projects.

Quickstart:

pip install warc3-wet-clueweb09

with warc3_wet_clueweb09.open("test.warc") as f:
    for record in f:
        print(record['WARC-Target-URI'], record['Content-Length'])

Python version support is unspecified; verify compatibility with your Python version before relying on this abandoned package.

Verify before relying:

- Whether the package actually works with modern Python versions (support unspecified in metadata).
- Extent of ClueWeb09-specific fixes and whether they address all known parsing issues.
- Whether seeking support (mentioned in changelog for 0.2.3) is fully functional.

## Package facts

- License: GPLv2 (copyleft)
- Python support: unspecified
- Install friction: high
- Maintenance: abandoned
- Downloads: 142.3K/month (top 15,000 on PyPI)
- Known vulnerabilities: none known

## Tags

warc file reader python, web archive parsing, warc wet file handling, clueweb09 archive support, warc record extraction, web crawl archive library, warc metadata access, web-archive, abandoned

[View on SkillFed](https://skillfed.io/packages/warc3-wet-clueweb09) · [View on PyPI](https://pypi.org/project/warc3-wet-clueweb09/)