flashoptim

What it is and what it does

FlashOptim is a library of PyTorch optimizer implementations that reduce peak training memory by compressing optimizer states, master weights, and gradients through quantization and fused Triton kernels. It provides drop-in replacements for standard optimizers—FlashSGD, FlashSGDW, FlashAdam, FlashAdamW, and FlashLion—that follow the standard PyTorch optimizer API, so you can swap them in with minimal code changes.

The library works by splitting weight representation and quantizing optimizer moments to 8-bit while maintaining master weights at configurable precision (24-bit or 32-bit by default). All compression operations are fused into the update kernel to avoid overhead. It supports optional gradient release for further memory reduction and can produce checkpoints with quantized optimizer states. Training in reduced precision (bf16/fp16) does not degrade convergence according to the documentation.

Use it for:

• Fine-tuning large language models (8B+) on memory-constrained GPUs to fit larger batch sizes or longer sequences.
• Reducing peak memory during training to enable training on smaller GPUs or with larger models.
• Storing compressed checkpoints that are substantially smaller than standard optimizer state files.
• Training workflows where gradient release timing is critical to minimize intermediate memory peaks.
• Migrating existing PyTorch training code to use memory-efficient optimizers without rewriting training loops.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

FlashOptim provides drop-in replacements for PyTorch optimizers (SGD, Adam, Lion variants) that reduce training memory by quantizing optimizer states, master weights, and gradients while maintaining model convergence.

Yes, with conditions. Install if you train on NVIDIA CUDA GPUs and need to reduce training memory for large models. The low install friction, active maintenance, permissive license, and zero known vulnerabilities support adoption. However, it is early-stage (Alpha), Linux/CUDA-only, and the first optimizer step incurs Triton JIT overhead. Verify convergence and performance on your specific models and hardware before production use.

Install

flashoptim on PyPI

pip install flashoptim

uv add flashoptim

poetry add flashoptim

Installing flashoptim

Before you install

Low install friction with a pure Python wheel. Active maintenance (last commit 2026-07-09) and early-stage status (Alpha, first release 2026-02-28) suggest ongoing development. Requires torch and triton as runtime dependencies.

License in practice

Apache License 2.0 is permissive, allowing commercial and private use with minimal restrictions—you may use, modify, and distribute FlashOptim freely provided you include license notices and disclaim warranties.

Quickstart

pip install flashoptim

import torch
from flashoptim import FlashAdamW, cast_model

model = torch.nn.Linear(128, 10).cuda()
cast_model(model, dtype=torch.bfloat16)
optimizer = FlashAdamW(model.parameters(), lr=1e-3)

x = torch.randn(32, 128, device="cuda", dtype=torch.bfloat16)
loss = model(x).sum()
loss.backward()
optimizer.step()

Requires NVIDIA CUDA GPU on Linux; torch and triton must be installed; first optimizer step is slower due to Triton kernel JIT compilation.

Verify before relying

• Actual memory savings percentages (35% peak, 57% checkpoint reduction) on representative models and hardware configurations beyond the 8B finetuning example.
• Convergence equivalence claims across different model architectures, scales, and training regimes.
• Performance overhead of quantization and fused kernels on different GPU generations and batch sizes.
• Compatibility with distributed training frameworks (DDP, FSDP) and mixed-precision training workflows.

Package facts

Evidence: flashoptim-0.1.4-py3-none-any.whl

Keywords: pytorch, optimizer, memory-efficient, deep-learning, training

Tags

More Artificial Intelligence packages

Further reading