{"categories":[{"label":"Libraries","url":"https://skillfed.io/packages/category/software-development-libraries/9"},{"label":"Distributed Computing","url":"https://skillfed.io/packages/category/system-distributed-computing/3"}],"enrichment":{"capability":"Vineyard is an in-memory immutable data manager that enables zero-copy sharing of distributed data across big data systems like graph analytics, numerical computing, and machine learning frameworks.","skillfed_tags":["distributed-data-sharing","zero-copy-memory","big-data-integration"],"use_cases":["Preprocessing large datasets with Mars and training models with PyTorch without intermediate file I/O","Sharing distributed graphs between GraphScope and downstream analytics or ML systems","Building data pipelines where multiple frameworks need to exchange tensors or dataframes with minimal overhead","Reducing memory consumption and latency in multi-stage distributed computations via stream pipelining","Integrating heterogeneous big data systems (SQL, tensor, graph) into a single coherent pipeline"],"what_it_does":"Vineyard is an in-memory immutable data manager designed to solve the data-sharing bottleneck in distributed big data pipelines. It provides zero-copy sharing of complex data structures (tensors, dataframes, graphs) across systems like Mars, PyTorch, and GraphScope by leveraging shared memory, eliminating the serialization, deserialization, and IO costs that typically occur when exchanging data between different frameworks in a distributed environment.\n\nThe package offers out-of-the-box high-level data abstractions and stream pipelining capabilities, allowing jobs to read and write data chunks without waiting for all preceding results. It is positioned as a CNCF sandbox project and supports Python 3.6 through 3.11 on POSIX systems and macOS. The binary distribution (vineyard-bdist) provides pre-compiled wheels for Linux platforms, reducing installation complexity compared to building from source.","worth_installing":"Yes, if you are building distributed data pipelines that exchange data between multiple big data frameworks (Mars, PyTorch, GraphScope, etc.) and want to eliminate serialization and IO overhead. The package is production-stable, permissively licensed, and actively maintained. Install friction is moderate due to platform-specific binaries, but the zero-copy sharing benefit justifies it for large-scale workloads. Not necessary for single-machine or simple data-passing scenarios."},"id":"vineyard-bdist","links":{"html":"https://skillfed.io/packages/vineyard-bdist","md":"https://skillfed.io/packages/vineyard-bdist.md","pypi":"https://pypi.org/project/vineyard-bdist/"},"maintenance":{"status":"aging"},"meta":{"latest_release":"2024-08-29","license_spdx":null,"license_treatment":"permissive","name":"vineyard-bdist","python_support":"unspecified","summary":"An in-memory immutable data manager"},"popularity":{"monthly_downloads":102682,"position":12854,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"0.24.2"}