lilcom

What it is and what it does

lilcom is a lossy compression utility that encodes NumPy floating-point arrays into compact byte strings, trading accuracy for storage efficiency. It uses a regression-based algorithm that models each element relative to its neighbors, then quantizes and compresses the residuals. The core implementation is in C++, and the compression ratio is controlled by a tick_power parameter that sets the quantization step size; the default tick_power of -8 yields a maximum error of 1/512 per element. It is intended for machine learning workflows where storing large volumes of training data or model weights benefits from reduced disk footprint and memory usage.

The package requires numpy as its only runtime dependency and supports Python 3.6 and later. Installation via pip uses prebuilt wheels for modern Python versions (3.10–3.14) on common platforms, but building from source requires a C++ compiler. The repository is maintained but aging, with no recent feature development; it remains suitable for stable, production use in contexts where the compression trade-offs are acceptable.

Use it for:

• Store large training datasets on disk with reduced footprint while maintaining acceptable numerical precision for model training.
• Compress neural network model weights for efficient distribution and deployment in bandwidth-constrained environments.
• Archive historical experimental results and intermediate model checkpoints with controlled accuracy loss.
• Reduce memory usage when loading large arrays into RAM during batch processing or inference pipelines.
• Serialize floating-point data for transmission over networks where bandwidth is a bottleneck.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Lossily compresses floating-point NumPy arrays into byte strings with user-specified accuracy, designed for storing training data and models in machine learning workflows.

Yes, if you need lossy compression of NumPy arrays in machine learning contexts and can accept the accuracy trade-offs. The package is stable, has no known vulnerabilities, and carries a permissive MIT license. Install friction is moderate due to C++ compilation, but prebuilt wheels are available for modern Python versions. The aging maintenance status is not a blocker for stable use, but expect no active feature development.

Install

lilcom on PyPI

pip install lilcom

uv add lilcom

poetry add lilcom

Installing lilcom

Before you install

Medium install friction due to C++ compilation requirement. Package is aging (211 days since last release) but repository remains active with recent commits; maintenance status suggests limited ongoing development.

License in practice

MIT license permits free use, modification, and distribution with minimal restrictions, making it suitable for both open-source and commercial projects.

Quickstart

import numpy as np
import lilcom

a = np.random.randn(300, 500)
a_compressed = lilcom.compress(a)
a_decompressed = lilcom.decompress(a_compressed)

Requires a C++ compiler (e.g., g++ or clang) to build from source; prebuilt wheels available for Python 3.10–3.14 on macOS, Linux, and Windows.

Verify before relying

• Typical compression ratio or file size reduction achieved in practice across different array types and sizes.
• Performance characteristics (compression/decompression speed) relative to alternative compression methods.
• Whether the tick_power parameter and resulting accuracy trade-offs are well-documented for different use cases.

Package facts

Evidence: lilcom-1.8.2-cp310-cp310-macosx_10_9_universal2.whl; lilcom-1.8.2-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.whl; lilcom-1.8.2-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl; lilcom-1.8.2-cp310-cp310-win_amd64.whl; lilcom-1.8.2-cp311-cp311-macosx_10_9_universal2.whl; lilcom-1.8.2-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl; lilcom-1.8.2-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl; lilcom-1.8.2-cp311-cp311-win_amd64.whl; lilcom-1.8.2-cp312-cp312-macosx_10_13_universal2.whl; lilcom-1.8.2-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl; lilcom-1.8.2-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl; lilcom-1.8.2-cp312-cp312-win_amd64.whl; lilcom-1.8.2-cp313-cp313-macosx_10_13_universal2.whl; lilcom-1.8.2-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl; lilcom-1.8.2-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl; lilcom-1.8.2-cp313-cp313-win_amd64.whl; lilcom-1.8.2-cp314-cp314-macosx_10_15_universal2.whl; lilcom-1.8.2-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl; lilcom-1.8.2-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl; lilcom-1.8.2-cp314-cp314-win_amd64.whl

Keywords: compression, numpy

Tags

More Compression packages

Further reading