HLL

What it is and what it does

HLL is a C-based Python module implementing the 64-bit HyperLogLog algorithm for cardinality estimation. It trades accuracy for memory efficiency, allowing you to estimate how many unique elements exist in a dataset without storing all of them—useful when datasets are too large to fit in memory or when you need fast approximate counts in streaming contexts.

The package uses a Murmur64A hash and stores registers in a hybrid sparse-dense representation: sparse when few registers are set (using a sorted dynamic array), switching to dense when memory would be wasted. Version 3.0.0 adds intersection cardinality estimation via Ertl's JMLE method, bulk insertion via add_range(), and fixes memory leaks and type errors from earlier releases. Requires Python >= 3.9 and a C compiler.

Use it for:

• Estimate unique visitor counts or unique IDs in high-volume streaming logs without storing all values
• Merge cardinality estimates from multiple data sources to approximate total unique items across distributed systems
• Estimate set intersection size (e.g., overlapping users between two datasets) using intersection_cardinality()
• Profile memory usage of large datasets by approximating cardinality with minimal RAM overhead
• Bulk-insert sequential integers efficiently using add_range() to avoid Python-C call overhead

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Estimates the cardinality (unique count) of very large datasets using the HyperLogLog algorithm, implemented in C for memory efficiency and speed.

Yes, if you need cardinality estimation and can tolerate a build-time dependency. The package is actively maintained, production-stable, has no known vulnerabilities, and solves a specific algorithmic problem well. Install friction is real (requires C compilation and dev headers), but that is inherent to the algorithm's performance. Not worth installing if you need exact counts or cannot set up a build environment.

Install

hll on PyPI

pip install hll

uv add hll

poetry add hll

Installing HLL

Before you install

High install friction: requires C compilation and Python development headers. The package is actively maintained (last commit 2026-02-25) and production-stable, but installation will demand a build environment on your system.

License in practice

MIT license is permissive and places no restrictions on use, modification, or distribution in proprietary or open-source projects.

Quickstart

pip install HLL

from HLL import HyperLogLog

hll = HyperLogLog(10)  # 2^10 registers
hll.add('some data')
print(hll.cardinality())

Requires C compiler and Python development headers to build the C extension.

Verify before relying

• Whether the sparse-to-dense conversion strategy in 3.0.0 materially improves memory usage for typical workloads compared to prior versions
• Accuracy degradation specifics when Jaccard similarity falls below 0.05 in intersection_cardinality() estimates

Package facts

Evidence: hll-3.0.0.tar.gz

Keywords: hyperloglog, cardinality, cardinality estimate, approximate counting, probabilistic data structures, sketch, data science, big data, streaming algorithms, memory efficient, set cardinality, unique count

Tags

More Scientific/Engineering packages