---
id: flashoptim
version: "0.1.4"
license: Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for… (full text in the JSON record)
license_treatment: permissive
maintenance: active
---

# flashoptim — Memory-Efficient PyTorch optimizers

License: permissive · Maintenance: active · Downloads: 82.2K/mo

## What it is and what it does

FlashOptim is a library of PyTorch optimizer implementations that reduce peak training memory by compressing optimizer states, master weights, and gradients through quantization and fused Triton kernels. It provides drop-in replacements for standard optimizers—FlashSGD, FlashSGDW, FlashAdam, FlashAdamW, and FlashLion—that follow the standard PyTorch optimizer API, so you can swap them in with minimal code changes.

The library works by splitting weight representation and quantizing optimizer moments to 8-bit while maintaining master weights at configurable precision (24-bit or 32-bit by default). All compression operations are fused into the update kernel to avoid overhead. It supports optional gradient release for further memory reduction and can produce checkpoints with quantized optimizer states. Training in reduced precision (bf16/fp16) does not degrade convergence according to the documentation.

Use it for:

- Fine-tuning large language models (8B+) on memory-constrained GPUs to fit larger batch sizes or longer sequences.
- Reducing peak memory during training to enable training on smaller GPUs or with larger models.
- Storing compressed checkpoints that are substantially smaller than standard optimizer state files.
- Training workflows where gradient release timing is critical to minimize intermediate memory peaks.
- Migrating existing PyTorch training code to use memory-efficient optimizers without rewriting training loops.

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

FlashOptim provides drop-in replacements for PyTorch optimizers (SGD, Adam, Lion variants) that reduce training memory by quantizing optimizer states, master weights, and gradients while maintaining model convergence.

Yes, with conditions. Install if you train on NVIDIA CUDA GPUs and need to reduce training memory for large models. The low install friction, active maintenance, permissive license, and zero known vulnerabilities support adoption. However, it is early-stage (Alpha), Linux/CUDA-only, and the first optimizer step incurs Triton JIT overhead. Verify convergence and performance on your specific models and hardware before production use.

## Install

pip install flashoptim
uv add flashoptim
poetry add flashoptim

## Installing flashoptim

Before you install: Low install friction with a pure Python wheel. Active maintenance (last commit 2026-07-09) and early-stage status (Alpha, first release 2026-02-28) suggest ongoing development. Requires torch and triton as runtime dependencies.

License in practice: Apache License 2.0 is permissive, allowing commercial and private use with minimal restrictions—you may use, modify, and distribute FlashOptim freely provided you include license notices and disclaim warranties.

Quickstart:

pip install flashoptim

import torch
from flashoptim import FlashAdamW, cast_model

model = torch.nn.Linear(128, 10).cuda()
cast_model(model, dtype=torch.bfloat16)
optimizer = FlashAdamW(model.parameters(), lr=1e-3)

x = torch.randn(32, 128, device="cuda", dtype=torch.bfloat16)
loss = model(x).sum()
loss.backward()
optimizer.step()

Requires NVIDIA CUDA GPU on Linux; torch and triton must be installed; first optimizer step is slower due to Triton kernel JIT compilation.

Verify before relying:

- Actual memory savings percentages (35% peak, 57% checkpoint reduction) on representative models and hardware configurations beyond the 8B finetuning example.
- Convergence equivalence claims across different model architectures, scales, and training regimes.
- Performance overhead of quantization and fused kernels on different GPU generations and batch sizes.
- Compatibility with distributed training frameworks (DDP, FSDP) and mixed-precision training workflows.

## Package facts

- License: Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for… (full text in the JSON record) (permissive)
- Python support: supports_current
- Install friction: low
- Maintenance: active
- Downloads: 82.2K/month (top 15,000 on PyPI)
- Known vulnerabilities: none known

## Tags

memory-efficient pytorch optimizers, reduce training memory footprint, quantized optimizer states, low-precision gradient training, pytorch adam sgd alternatives, fused triton kernel optimizers, compressed checkpoint storage, memory-optimization, gpu-training, quantization

[View on SkillFed](https://skillfed.io/packages/flashoptim) · [View on PyPI](https://pypi.org/project/flashoptim/)