lsst-daf-butler

What it is and what it does

lsst-daf-butler is a data access framework designed for the Vera C. Rubin Observatory's LSST project, providing a Python abstraction layer that sits between scientific applications and underlying datastores. It handles the storage and retrieval of astronomical image data and related metadata, allowing researchers to work with large datasets without needing to know the details of the underlying storage system—whether files are stored locally, in cloud buckets, or in databases.

The package is built on a stack of mature dependencies: SQLAlchemy for database operations, PyArrow for efficient columnar data handling, NumPy for numerical work, and Pydantic for data validation. It requires Python 3.11 or higher and is actively maintained by the LSST collaboration. The documentation notes a known memory leak issue in PyArrow version 18 and later when reading Parquet files with the default allocator; the workaround is to set the ARROW_DEFAULT_MEMORY_POOL environment variable to jemalloc.

Use it for:

• Access and query large astronomical survey datasets from the Rubin Observatory without managing storage details directly
• Build data pipelines that read and write image metadata and provenance information in a standardized way
• Integrate with LSST processing systems that rely on the butler for consistent data organization
• Retrieve Parquet-formatted astronomical data while managing memory efficiently in long-running analysis jobs

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Provides an abstraction layer for reading and writing astronomical data to datastores, enabling structured access to large scientific datasets through a unified interface.

Yes, if you are working within the LSST/Rubin Observatory ecosystem or need to integrate with their data pipelines. The package is actively maintained, has no known security vulnerabilities, and installs cleanly. However, it is a specialized tool tightly coupled to a specific observatory's data model; it is not a general-purpose data access layer. Verify that your use case aligns with LSST's data organization before adopting it.

Install

lsst-daf-butler on PyPI

pip install lsst-daf-butler

uv add lsst-daf-butler

poetry add lsst-daf-butler

Installing lsst-daf-butler

Before you install

Active maintenance with a release within the last day. Low install friction with a pure-Python wheel. Requires Python 3.11 or higher and depends on a moderately sized stack of scientific and database libraries (SQLAlchemy, PyArrow, NumPy, Pydantic).

License in practice

Dual-licensed under GPL-3.0-or-later and BSD-3-Clause with unclear license treatment in the package metadata. Users may choose which license to apply, but should review both gpl-3.0.txt and bsd_license.txt files to understand their obligations.

Quickstart

pip install lsst-daf-butler

from lsst.daf.butler import Butler

butler = Butler(config='path/to/butler.yaml')
dataset = butler.get('dataset_type', dataId={})

Requires Python 3.11 or higher. If using PyArrow version 18 or later, set ARROW_DEFAULT_MEMORY_POOL=jemalloc environment variable to avoid memory leaks when reading Parquet files.

Verify before relying

• Whether the memory leak workaround (ARROW_DEFAULT_MEMORY_POOL=jemalloc) is still necessary for current PyArrow versions
• Whether the package is suitable for non-LSST/Rubin Observatory use cases or if it is tightly coupled to that ecosystem

Package facts

Evidence: lsst_daf_butler-30.2026.3300-py3-none-any.whl

Keywords: lsst

Tags

More Astronomy packages