partd

What it is and what it does

Partd is a composable key-value store designed to handle append-heavy workloads, particularly shuffling operations common in distributed data processing. It stores raw bytes under string keys and supports appending new bytes to existing values, then retrieving the concatenated result. The package excels because it offers multiple interchangeable backends: an in-memory dictionary for small datasets, a file-based store for persistence, a buffered layer that spills to disk when memory runs low, and a client-server model for multi-process coordination.

The package's main strength is its composability—you can layer compression (BZ2, Blosc, ZLib, Snappy), encoding (Pickle, NumPy, Pandas), and storage backends together by nesting them, so a single append or get call transparently handles serialization, compression, and I/O. This design makes it useful for data pipelines that need to shuffle intermediate results efficiently without writing custom serialization or buffering logic. It depends on locket for file-based locking and toolz for utility functions.

Use it for

• Buffering intermediate shuffle results in a distributed data processing pipeline where data must be reorganized by key across multiple workers.
• Storing and retrieving compressed, serialized Python objects (lists, arrays, dataframes) on disk without manual encoding or decompression.
• Coordinating multi-process writes to a single key-value store via a server process, ensuring consistency across workers.
• Implementing a spill-to-disk cache that keeps hot data in memory and automatically moves cold data to files when memory pressure rises.
• Composing custom storage layers by wrapping a base store (Dict or File) with compression and encoding transformations.

Worth the install?

AI-flagged interpretation of the facts on this page. Verify before relying on it.

Install

partd on PyPI

Before you install

Low friction installation with only two runtime dependencies (locket and toolz). Dormant maintenance status—last release was in May 2024 and the repository has not been archived, but activity is minimal. Suitable for stable, established use cases rather than active development.

Requires Python 3.9 or later; file-backed stores need write access to the specified directory.

License in practice

BSD permissive license allows broad use, modification, and distribution with minimal restrictions, making it suitable for commercial and open-source projects.

Quickstart

import partd

# File-backed store
p = partd.File('/path/to/dataset/')
p.append({'x': b'Hello ', 'y': b'123'})
p.append({'x': b'world!', 'y': b'456'})
result = p.get('x')  # b'Hello world!'
p.drop()

Package facts

Evidence: partd-1.4.2-py3-none-any.whl

Tags

More Database packages

See also pickleshare · pyppmd · fcache · DrissionRecord · numcodecs · speedict · cobs · ase-db-backends · historydict · sqlitedict