koalas

What it is and what it does

Koalas bridges pandas and Apache Spark by exposing a pandas-compatible DataFrame API that executes on Spark clusters. If you know pandas, you can write similar code against Koalas and have it run on distributed data without learning Spark's native API. The package wraps Spark's distributed execution while mimicking pandas' single-node interface, making it useful for teams that want to scale pandas workflows to big data without rewriting code.

However, Koalas is now in maintenance mode and officially superseded by PySpark's native pandas API layer in Spark 3.2+. The last release was in October 2021, and the repository has seen minimal activity since. For new projects targeting Spark 3.2 or later, the description recommends using PySpark directly instead.

Use it for:

• Scale existing pandas code to distributed Spark clusters without rewriting logic or learning Spark's native API
• Write a single codebase that works with pandas on small datasets (for testing) and Koalas on large Spark clusters
• Migrate legacy pandas workflows to big data infrastructure when locked into Spark 3.1 or below
• Prototype data transformations in pandas, then deploy them on Spark using Koalas with minimal code changes

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Koalas implements the pandas DataFrame API on top of Apache Spark, letting you write pandas-like code that runs on distributed Spark clusters instead of a single machine.

No, unless you are locked into Spark 3.1 or below and cannot upgrade. Koalas is dormant (last release October 2021, minimal commits since) and officially deprecated in favor of PySpark's native pandas API layer in Spark 3.2+. For new projects or upgradeable environments, use PySpark directly. For legacy systems on Spark 3.1, Koalas remains functional but will not receive updates.

Install

koalas on PyPI

pip install koalas

uv add koalas

poetry add koalas

Installing koalas

Before you install

Low install friction with a pure-Python wheel. However, the package is dormant—last release was 2021-10-19 and last commit 2024-03-20—and the description explicitly states it is in maintenance mode, superseded by PySpark's native pandas API layer in Spark 3.2+.

License in practice

Licensed under Apache License 2.0 (permissive), which allows commercial and private use with minimal restrictions, though you must include a copy of the license and state significant changes.

Quickstart

pip install koalas

import pandas as pd
import pyarrow
import numpy

pdf = pd.DataFrame({'x': range(3), 'y': ['a', 'b', 'b']})
# Convert to Koalas DataFrame and perform operations

Requires Apache Spark 3.1 or below; for Spark 3.2+, use PySpark directly. Also requires pandas, pyarrow, and numpy as runtime dependencies.

Verify before relying

• Exact import path and API surface for Koalas DataFrame creation and operations
• Whether existing code will continue to work without modification if Spark is upgraded to 3.2+
• Compatibility with Python versions beyond 3.9 (classifiers support up to 3.9)

Package facts

Evidence: koalas-1.8.2-py3-none-any.whl

Tags

More Information Analysis packages