lilcom

What it is and what it does

lilcom is a lossy compression utility that encodes NumPy floating-point arrays into compact byte strings, trading accuracy for storage efficiency. It uses a regression-based algorithm that models each element relative to its neighbors, then quantizes and compresses the residuals. The core implementation is in C++, and the compression ratio is controlled by a tick_power parameter that sets the quantization step size; the default tick_power of -8 yields a maximum error of 1/512 per element. It is intended for machine learning workflows where storing large volumes of training data or model weights benefits from reduced disk footprint and memory usage.

The package requires numpy as its only runtime dependency and supports Python 3.6 and later. Installation via pip uses prebuilt wheels for modern Python versions (3.10–3.14) on common platforms, but building from source requires a C++ compiler. The repository is maintained but aging, with no recent feature development; it remains suitable for stable, production use in contexts where the compression trade-offs are acceptable.

Use it for

• Store large training datasets on disk with reduced footprint while maintaining acceptable numerical precision for model training.
• Compress neural network model weights for efficient distribution and deployment in bandwidth-constrained environments.
• Archive historical experimental results and intermediate model checkpoints with controlled accuracy loss.
• Reduce memory usage when loading large arrays into RAM during batch processing or inference pipelines.
• Serialize floating-point data for transmission over networks where bandwidth is a bottleneck.

Worth the install?

AI-flagged interpretation of the facts on this page. Verify before relying on it.

Install

lilcom on PyPI

Before you install

Medium install friction due to C++ compilation requirement. Package is aging (211 days since last release) but repository remains active with recent commits; maintenance status suggests limited ongoing development.

Requires a C++ compiler (e.g., g++ or clang) to build from source; prebuilt wheels available for Python 3.10–3.14 on macOS, Linux, and Windows.

License in practice

MIT license permits free use, modification, and distribution with minimal restrictions, making it suitable for both open-source and commercial projects.

Quickstart

import numpy as np
import lilcom

a = np.random.randn(300, 500)
a_compressed = lilcom.compress(a)
a_decompressed = lilcom.decompress(a_compressed)

Package facts

Evidence: lilcom-1.8.2-cp310-cp310-macosx_10_9_universal2.whl; lilcom-1.8.2-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.whl; lilcom-1.8.2-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl; lilcom-1.8.2-cp310-cp310-win_amd64.whl; lilcom-1.8.2-cp311-cp311-macosx_10_9_universal2.whl; lilcom-1.8.2-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl; lilcom-1.8.2-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl; lilcom-1.8.2-cp311-cp311-win_amd64.whl; lilcom-1.8.2-cp312-cp312-macosx_10_13_universal2.whl; lilcom-1.8.2-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl; lilcom-1.8.2-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl; lilcom-1.8.2-cp312-cp312-win_amd64.whl; lilcom-1.8.2-cp313-cp313-macosx_10_13_universal2.whl; lilcom-1.8.2-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl; lilcom-1.8.2-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl; lilcom-1.8.2-cp313-cp313-win_amd64.whl; lilcom-1.8.2-cp314-cp314-macosx_10_15_universal2.whl; lilcom-1.8.2-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl; lilcom-1.8.2-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl; lilcom-1.8.2-cp314-cp314-win_amd64.whl

Tags

More Compression packages

See also ml-dtypes · python-neo-lzf · pcodec · pylibjpeg-openjpeg · pylerc · pylibjpeg-libjpeg · descript-audio-codec · lzstring · encodec · tifffile

Further reading