apache-beam
Apache Beam SDK for Python
Decision gist · record as of 2026-08-14
Yes, if you need to write data pipelines that can run on multiple backends without code changes. The unified model and active maintenance make it solid for production ETL and streaming work. Be aware of medium install friction from 24 dependencies and the Python >=3.10 requirement. Not necessary if you only need simple single-machine data processing or are locked into a specific runner ecosystem.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- Requires Python >=3.10.
- Running pipelines on distributed backends requires additional setup and configuration of the target runner environment.
- Medium install friction due to 24 runtime dependencies including heavy packages like numpy, pyarrow, protobuf, and grpcio.
License · maintenance · safety
Apache License, Version 2.0 (permissive) — Apache License, Version 2.0 is permissive; you can use this package freely in commercial and private projects, modify it, and distribute it with minimal restrictions.
last release 2026-07-08 (37 days)
0 known vulnerabilities (OSV.dev, 2026-08-14) · 6,301,370 downloads/mo, #1,934 on PyPI
Alternatives
Verify before relying
pip install apache-beam
from apache_beam import Pipeline, Create, Map
with Pipeline() as p:
(p | Create([1, 2, 3]) | Map(lambda x: x * 2))- Whether the 24 runtime dependencies install cleanly together on all supported platforms without version conflicts.
- Performance characteristics and memory overhead when running large-scale pipelines locally versus on distributed backends.
- Specific limitations of Python streaming pipeline execution mentioned in the documentation.
What it is and what it does
Apache Beam provides a unified programming model for expressing both batch and streaming data-parallel processing pipelines. It abstracts the underlying execution engine through core concepts—PCollection (data), PTransform (computation), Pipeline (DAG), and PipelineRunner (execution target)—allowing the same code to run on multiple distributed backends. The Python SDK lets you write pipelines using familiar Python syntax and type hints, then execute them locally via DirectRunner or on production systems like Spark, Flink, or Google Cloud Dataflow without rewriting your logic.
The framework supports streaming pipelines, type safety through Python type hints, custom I/O connectors, and machine learning inference integration via the RunInference API for PyTorch and Scikit-learn models. It is designed for end users writing pipelines, SDK developers creating language-specific bindings, and runner developers implementing execution environments.
Use it for
- Build ETL pipelines that transform and load data from multiple sources, running locally for testing and on Spark or Dataflow for production.
- Process real-time streaming data with windowing and aggregation, then replay the same logic on historical batch data.
- Develop machine learning feature engineering pipelines that preprocess data at scale and deploy to production runners.
- Create data validation and quality-check pipelines that run on streaming data or in batch mode on historical datasets.
- Build multi-language data processing workflows by combining transforms written in different Beam SDK languages in one pipeline.
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes, if you need to write data pipelines that can run on multiple backends without code changes.
The unified model and active maintenance make it solid for production ETL and streaming work. Be aware of medium install friction from 24 dependencies and the Python >=3.10 requirement. Not necessary if you only need simple single-machine data processing or are locked into a specific runner ecosystem.
Install
apache-beam on PyPI
Before you install
Medium install friction due to 24 runtime dependencies including heavy packages like numpy, pyarrow, protobuf, and grpcio. Actively maintained with recent releases. Supports Python 3.10, 3.11, 3.12, 3.13, 3.14 across multiple platforms.
Requires Python >=3.10. Running pipelines on distributed backends requires additional setup and configuration of the target runner environment.
License in practice
Apache License, Version 2.0 is permissive; you can use this package freely in commercial and private projects, modify it, and distribute it with minimal restrictions.
Quickstart
pip install apache-beam
from apache_beam import Pipeline, Create, Map
with Pipeline() as p:
(p | Create([1, 2, 3]) | Map(lambda x: x * 2))Verify before relying
- Whether the 24 runtime dependencies install cleanly together on all supported platforms without version conflicts.
- Performance characteristics and memory overhead when running large-scale pipelines locally versus on distributed backends.
- Specific limitations of Python streaming pipeline execution mentioned in the documentation.
Package facts
| License | Apache License, Version 2.0 permissive |
| Python support | Supports the current Python release >=3.10 |
| Install friction | Medium. Platform-specific wheel |
| Runtime dependencies | 24 packagescryptographyenvoy-data-planefastavrofastenersgrpciohttplib2jsonpicklenumpyobjsizepackagingpillowpymongoproto-plusprotobufpython-dateutilpytzrequestssortedcontainerstyping-extensionszstandardpyyamlbeartypepyarrowpyarrow-hotfix |
| Maintenance | Actively maintained 37 days since the last release |
| First released | |
| Downloads | 6,301,370 / month, #1,934 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
| Classifiers | Intended Audience :: End Users/DesktopLicense :: OSI Approved :: Apache Software LicenseOperating System :: POSIX :: LinuxProgramming Language :: Python :: 3.10Programming Language :: Python :: 3.11Programming Language :: Python :: 3.12Programming Language :: Python :: 3.13Programming Language :: Python :: 3.14Topic :: Software Development :: LibrariesTopic :: Software Development :: Libraries :: Python Modules |
Evidence: apache_beam-2.75.0-cp310-cp310-macosx_11_0_arm64.whl; apache_beam-2.75.0-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl; apache_beam-2.75.0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl; apache_beam-2.75.0-cp310-cp310-win32.whl; apache_beam-2.75.0-cp310-cp310-win_amd64.whl; apache_beam-2.75.0-cp311-cp311-macosx_11_0_arm64.whl; apache_beam-2.75.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl; apache_beam-2.75.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl; apache_beam-2.75.0-cp311-cp311-win32.whl; apache_beam-2.75.0-cp311-cp311-win_amd64.whl; apache_beam-2.75.0-cp312-cp312-macosx_11_0_arm64.whl; apache_beam-2.75.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl; apache_beam-2.75.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl; apache_beam-2.75.0-cp312-cp312-win32.whl; apache_beam-2.75.0-cp312-cp312-win_amd64.whl; apache_beam-2.75.0-cp313-cp313-macosx_11_0_arm64.whl; apache_beam-2.75.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl; apache_beam-2.75.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl; apache_beam-2.75.0-cp313-cp313-win32.whl; apache_beam-2.75.0-cp313-cp313-win_amd64.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “batch and streaming data processing”
- apache-beamApache Beam is a unified framework for defining and executing batch…
- apache-flinkPython API for Apache Flink that builds stateful batch and streaming…
- cosmos-xennaCosmos-xenna is a Python framework for building and running…
Give your agent the search over MCP, or paste the wish link into any chat.
More Libraries packages
urllib3 is an HTTP client library that provides thread-safe connection pooling, SSL/TLS verification, multipart file uploads, request retries, compression support, and proxy handling for Python applications.
Requests is a Python HTTP library that simplifies sending HTTP/1.1 requests with automatic handling of headers, authentication, cookies, and response parsing.
Pluggy provides a plugin system that lets you define hook specifications and register implementations to be called in sequence, enabling extensible Python applications without tight coupling.
Install it if you're building an extensible application or framework.
Provides parsing, arithmetic, and recurrence rule computation for dates and times, with timezone support and iCalendar RFC compliance.
Install it if you need to parse flexible date strings, compute relative dates, handle timezones, or work with recurrence rules—it's the de facto choice for these tasks.
Six provides utility functions to write Python code that runs on both Python 2.7 and Python 3.3+, smoothing over language differences between the two versions.
pytest is a testing framework that lets you write test functions using plain assert statements and automatically discovers and runs them, with detailed failure reporting.
See also apache-airflow-providers-apache-beam · beam-nuggets · apache-flink · apache-flink-libraries · koheesio · kfp · hazelcast-python-client · sf-hamilton · raydp · jina