maggma
Framework to develop datapipelines from files on disk to full dissemenation API
Decision gist · record as of 2026-08-14
Yes. Maggma is actively maintained, has no known vulnerabilities, and solves a real problem for scientific and data-engineering workflows: abstracting storage backend differences. The dependency footprint is substantial but justified for a pipeline framework. Install it if you're building multi-stage data processing that needs to work across heterogeneous storage systems or if you want to parallelize transformations cleanly.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- Requires a running MongoDB instance (or mongomock-ng for testing).
- Python 3.11 or later.
- Low install friction with a pure-Python wheel.
License · maintenance · safety
(unclear) — Licensed under a BSD 3-Clause variant with institutional copyright held by Lawrence Berkeley National Laboratory. The license permits redistribution and derivative works under standard BSD terms; no commercial restrictions.
last release 2026-06-22 (53 days) · last repo commit 2026-08-10 · 43 stars
0 known vulnerabilities (OSV.dev, 2026-08-14) · 659,963 downloads/mo, #5,461 on PyPI
Alternatives
Verify before relying
pip install maggma
from maggma.stores import MongoStore
store = MongoStore(
database="my_db",
collection_name="my_collection",
host="localhost",
key="name"
)
with store:
store.update([{"name": "item"}])
print(store.count())- Whether the package's async capabilities (aioitertools dependency) are production-ready or experimental.
- Performance characteristics when working with very large datasets across multiple store backends.
- Whether REST API dissemination mentioned in the summary is fully implemented or a planned feature.
What it is and what it does
Maggma is a framework for building modular data processing pipelines in Python. It abstracts away the differences between storage backends—MongoDB, Amazon S3, GridFS, local files, and others—behind a unified Store interface that mimics PyMongo's query syntax. This lets you write pipeline code once and swap storage backends without rewriting your data access logic.
The framework's core is built around two concepts: Stores (which provide consistent read/write/query access to data) and Builders (which represent transformation steps analogous to ETL operations). Builders break work into three phases—fetching items, processing them (without I/O), and writing results—so that processing can be parallelized. Pipelines can be chained together and serialized to JSON for production deployment. It's designed for scientific data workflows and is maintained by the Materials Project team.
Use it for
- Build a data pipeline that reads raw experimental data from S3, transforms it with a Builder, and stores results in MongoDB without rewriting query code.
- Query and aggregate data across multiple storage backends (local files, databases, cloud storage) using a single consistent interface.
- Chain multiple transformation steps into a reproducible pipeline that can be version-controlled and deployed to production as JSON.
- Parallelize data processing by separating I/O-heavy retrieval and storage from stateless transformation logic.
- Migrate data between storage systems (e.g., files to MongoDB) by swapping Store implementations while keeping pipeline logic unchanged.
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes.
Maggma is actively maintained, has no known vulnerabilities, and solves a real problem for scientific and data-engineering workflows: abstracting storage backend differences. The dependency footprint is substantial but justified for a pipeline framework. Install it if you're building multi-stage data processing that needs to work across heterogeneous storage systems or if you want to parallelize transformations cleanly.
Install
maggma on PyPI
Before you install
Low install friction with a pure-Python wheel. Active maintenance with a recent release and ongoing commits. Supports Python 3.11, 3.12, and 3.13. Twenty runtime dependencies cover database connectivity, data serialization, and async utilities—a substantial but manageable dependency footprint for a data pipeline framework.
Requires a running MongoDB instance (or mongomock-ng for testing). Python 3.11 or later.
License in practice
Licensed under a BSD 3-Clause variant with institutional copyright held by Lawrence Berkeley National Laboratory. The license permits redistribution and derivative works under standard BSD terms; no commercial restrictions.
Quickstart
pip install maggma
from maggma.stores import MongoStore
store = MongoStore(
database="my_db",
collection_name="my_collection",
host="localhost",
key="name"
)
with store:
store.update([{"name": "item"}])
print(store.count())Verify before relying
- Whether the package's async capabilities (aioitertools dependency) are production-ready or experimental.
- Performance characteristics when working with very large datasets across multiple store backends.
- Whether REST API dissemination mentioned in the summary is fully implemented or a planned feature.
Package facts
| License | Not declared unclear |
| Python support | Supports the current Python release >=3.11 |
| Install friction | Low. Pure-Python wheel |
| Runtime dependencies | 20 packagesruamel.yamlpydanticpydantic-settingspymongomontymongomock-ngpydashjsonschematqdmpandasjsonlinesaioitertoolsnumpypyzmqdnspythonparamikomsgpackorjsonboto3python-dateutil |
| Maintenance | Actively maintained 53 days since the last release |
| Last repo commit | |
| First released | |
| Downloads | 659,963 / month, #5,461 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
| Classifiers | Development Status :: 5 - Production/StableIntended Audience :: Information TechnologyIntended Audience :: Science/ResearchIntended Audience :: System AdministratorsOperating System :: OS IndependentProgramming Language :: Python :: 3Programming Language :: Python :: 3.11Programming Language :: Python :: 3.12Programming Language :: Python :: 3.13Topic :: Database :: Front-EndsTopic :: Other/Nonlisted TopicTopic :: Scientific/Engineering |
Evidence: maggma-0.74.0-py3-none-any.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “mongodb abstraction layer”
- maggmaMaggma provides a unified interface for building data processing…
- pydantic-mongoProvides a Repository pattern for MongoDB with Pydantic model…
- cachelibCacheLib provides a unified API for interacting with multiple cache…
Give your agent the search over MCP, or paste the wish link into any chat.
More Scientific/Engineering packages
NumPy provides an N-dimensional array object and a comprehensive suite of mathematical, linear algebra, Fourier transform, and random number functions for scientific computing in Python.
pandas provides fast, flexible data structures (Series and DataFrame) for loading, cleaning, transforming, and analyzing labeled or relational data in Python.
scipy provides numerical algorithms for mathematics, science, and engineering—including optimization, integration, linear algebra, Fourier transforms, signal and image processing, and ODE solvers—built on numpy arrays.
scikit-learn provides a comprehensive Python library for supervised and unsupervised machine learning, including classification, regression, clustering, dimensionality reduction, and model evaluation tools built on NumPy and SciPy.
Install it if you need to train, evaluate, or deploy supervised or unsupervised learning models.
dill extends Python's pickle module to serialize and deserialize a much wider range of Python objects, including functions, lambdas, classes, and interpreter sessions, to byte streams for storage or network transmission.
Multiprocess is an enhanced fork of Python's standard multiprocessing library that uses dill for better serialization, allowing you to spawn processes with a threading-like API and share complex objects between them.
Install it if you use multiprocessing and encounter pickle serialization limits with lambdas or complex objects.
See also pymongo · multi-storage-client · hdmf · emmet-core · odmantic · mongodb-migrations · pymatgen · sqlmesh · Flask-PyMongo · unstructured-ingest