pybloom-live

What it is and what it does

pybloom_live provides two Bloom filter implementations for Python: a fixed-capacity BloomFilter for known dataset sizes, and a ScalableBloomFilter that automatically expands as elements are added. Bloom filters are probabilistic data structures that answer set membership queries with certainty for negative results (element definitely not in set) and probabilistic results for positive results (element might be in set, with a tunable false positive rate). The package uses xxHash for fast non-cryptographic hashing and supports set operations (union, intersection), serialization to disk, and both small and large growth modes for scalable filters.

The library is designed for scenarios where memory efficiency and speed matter more than perfect accuracy—web crawlers tracking visited URLs, caching systems doing pre-filters before expensive lookups, database query optimization, and distributed system reconciliation. It has no runtime dependencies and works with Python 3.6 and later.

Use it for:

• Web crawlers: Track visited URLs to avoid re-crawling the same pages repeatedly
• Cache pre-filtering: Quick membership checks before expensive database or network lookups
• Database optimization: Pre-filter query results to avoid unnecessary disk reads
• Spell checkers: Fast dictionary lookups for word validation
• Distributed systems: Efficient set reconciliation and membership queries across nodes
• Network packet filtering: Fast classification of packets in routers and firewalls

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Implements Bloom filters—space-efficient probabilistic data structures for fast set membership testing with controllable false positive rates and no false negatives.

Yes, if you need efficient set membership testing and can tolerate the source-only install friction. The package is stable and permissively licensed, but maintenance is aging (no releases since October 2022). Install only if your use case genuinely benefits from Bloom filter semantics; for simple set membership in memory-unconstrained applications, a plain Python set is simpler and faster.

Install

pybloom-live on PyPI

pip install pybloom-live

uv add pybloom-live

poetry add pybloom-live

Installing pybloom-live

Before you install

High install friction: the package distributes as a source tarball with no pre-built wheels, requiring compilation at install time. Maintenance status is aging—the latest release dates to October 2022, over three years old, though the repository remains active with recent commits.

License in practice

MIT License (permissive): you may use, modify, and distribute this package freely in both open-source and commercial projects, provided you include the license notice.

Quickstart

pip install pybloom-live

import pybloom_live

# Create a fixed-capacity filter
f = pybloom_live.BloomFilter(capacity=1000, error_rate=0.001)
f.add("apple")
print("apple" in f)  # True
print("grape" in f)  # False

Source-only distribution requires a C compiler and build tools at install time; no pre-built wheels are available.

Verify before relying

• Whether the package works with current Python versions (requires_python is unspecified in metadata)
• Performance characteristics relative to alternative Bloom filter libraries
• Whether xxHash is bundled or requires a system dependency

Package facts

Evidence: pybloom_live-4.0.0.tar.gz

Keywords: data structures, bloom filter, bloom, filter, big data, probabilistic, set

Tags

More Python Modules packages