vineyard-bdist

What it is and what it does

Vineyard is an in-memory immutable data manager designed to solve the data-sharing bottleneck in distributed big data pipelines. It provides zero-copy sharing of complex data structures (tensors, dataframes, graphs) across systems like Mars, PyTorch, and GraphScope by leveraging shared memory, eliminating the serialization, deserialization, and IO costs that typically occur when exchanging data between different frameworks in a distributed environment.

The package offers out-of-the-box high-level data abstractions and stream pipelining capabilities, allowing jobs to read and write data chunks without waiting for all preceding results. It is positioned as a CNCF sandbox project and supports Python 3.6 through 3.11 on POSIX systems and macOS. The binary distribution (vineyard-bdist) provides pre-compiled wheels for Linux platforms, reducing installation complexity compared to building from source.

Use it for:

• Preprocessing large datasets with Mars and training models with PyTorch without intermediate file I/O
• Sharing distributed graphs between GraphScope and downstream analytics or ML systems
• Building data pipelines where multiple frameworks need to exchange tensors or dataframes with minimal overhead
• Reducing memory consumption and latency in multi-stage distributed computations via stream pipelining
• Integrating heterogeneous big data systems (SQL, tensor, graph) into a single coherent pipeline

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Vineyard is an in-memory immutable data manager that enables zero-copy sharing of distributed data across big data systems like graph analytics, numerical computing, and machine learning frameworks.

Yes, if you are building distributed data pipelines that exchange data between multiple big data frameworks (Mars, PyTorch, GraphScope, etc.) and want to eliminate serialization and IO overhead. The package is production-stable, permissively licensed, and actively maintained. Install friction is moderate due to platform-specific binaries, but the zero-copy sharing benefit justifies it for large-scale workloads. Not necessary for single-machine or simple data-passing scenarios.

Install

vineyard-bdist on PyPI

pip install vineyard-bdist

uv add vineyard-bdist

poetry add vineyard-bdist

Installing vineyard-bdist

Before you install

Medium install friction due to compiled binary wheels for specific platforms (manylinux2014 aarch64 and x86_64). The package is marked as aging with last commit on 2026-01-22, but remains actively maintained and marked Production/Stable.

License in practice

Licensed under Apache License 2.0 (permissive), allowing commercial use, modification, and distribution with minimal restrictions—suitable for most production and research contexts.

Quickstart

pip install vineyard-bdist

import vineyard
client = vineyard.connect(vineyard_ipc_socket)
data = client.get(vineyard_object_id)

Requires a running Vineyard server instance and an IPC socket path; binary wheels are platform-specific (Linux x86_64 or aarch64).

Verify before relying

• Whether vineyard-bdist is the correct distribution for your use case versus the main vineyard package
• Specific performance gains or latency improvements in your distributed workload
• Compatibility with your particular big data framework (Mars, GraphScope, PyTorch, etc.)

Package facts

Evidence: vineyard_bdist-0.24.2-py3-none-manylinux2014_aarch64.whl; vineyard_bdist-0.24.2-py3-none-manylinux2014_x86_64.whl

Tags

More Libraries packages