---
id: lsst-daf-butler
version: "30.2026.3300"
license: BSD-3-Clause OR GPL-3.0-or-later
license_treatment: unclear
maintenance: active
---

# lsst-daf-butler — An abstraction layer for reading and writing astronomical data to datastores.

License: unclear · Maintenance: active · Downloads: 176.6K/mo

## What it is and what it does

lsst-daf-butler is a data access framework designed for the Vera C. Rubin Observatory's LSST project, providing a Python abstraction layer that sits between scientific applications and underlying datastores. It handles the storage and retrieval of astronomical image data and related metadata, allowing researchers to work with large datasets without needing to know the details of the underlying storage system—whether files are stored locally, in cloud buckets, or in databases.

The package is built on a stack of mature dependencies: SQLAlchemy for database operations, PyArrow for efficient columnar data handling, NumPy for numerical work, and Pydantic for data validation. It requires Python 3.11 or higher and is actively maintained by the LSST collaboration. The documentation notes a known memory leak issue in PyArrow version 18 and later when reading Parquet files with the default allocator; the workaround is to set the ARROW_DEFAULT_MEMORY_POOL environment variable to jemalloc.

Use it for:

- Access and query large astronomical survey datasets from the Rubin Observatory without managing storage details directly
- Build data pipelines that read and write image metadata and provenance information in a standardized way
- Integrate with LSST processing systems that rely on the butler for consistent data organization
- Retrieve Parquet-formatted astronomical data while managing memory efficiently in long-running analysis jobs

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

Provides an abstraction layer for reading and writing astronomical data to datastores, enabling structured access to large scientific datasets through a unified interface.

Yes, if you are working within the LSST/Rubin Observatory ecosystem or need to integrate with their data pipelines. The package is actively maintained, has no known security vulnerabilities, and installs cleanly. However, it is a specialized tool tightly coupled to a specific observatory's data model; it is not a general-purpose data access layer. Verify that your use case aligns with LSST's data organization before adopting it.

## Install

pip install lsst-daf-butler
uv add lsst-daf-butler
poetry add lsst-daf-butler

## Installing lsst-daf-butler

Before you install: Active maintenance with a release within the last day. Low install friction with a pure-Python wheel. Requires Python 3.11 or higher and depends on a moderately sized stack of scientific and database libraries (SQLAlchemy, PyArrow, NumPy, Pydantic).

License in practice: Dual-licensed under GPL-3.0-or-later and BSD-3-Clause with unclear license treatment in the package metadata. Users may choose which license to apply, but should review both gpl-3.0.txt and bsd_license.txt files to understand their obligations.

Quickstart:

pip install lsst-daf-butler

from lsst.daf.butler import Butler

butler = Butler(config='path/to/butler.yaml')
dataset = butler.get('dataset_type', dataId={})

Requires Python 3.11 or higher. If using PyArrow version 18 or later, set ARROW_DEFAULT_MEMORY_POOL=jemalloc environment variable to avoid memory leaks when reading Parquet files.

Verify before relying:

- Whether the memory leak workaround (ARROW_DEFAULT_MEMORY_POOL=jemalloc) is still necessary for current PyArrow versions
- Whether the package is suitable for non-LSST/Rubin Observatory use cases or if it is tightly coupled to that ecosystem

## Package facts

- License: BSD-3-Clause OR GPL-3.0-or-later (unclear)
- Python support: supports_current
- Install friction: low
- Maintenance: active
- Downloads: 176.6K/month (top 15,000 on PyPI)
- Known vulnerabilities: none known

## Tags

astronomical data access framework, datastore abstraction layer, lsst data butler, scientific data management, rubin observatory data access, parquet data retrieval, astronomy data pipeline, astronomy, data-access, lsst

[View on SkillFed](https://skillfed.io/packages/lsst-daf-butler) · [View on PyPI](https://pypi.org/project/lsst-daf-butler/)