flashoptim
Memory-Efficient PyTorch optimizers
Decision gist · record as of 2026-08-14
Yes, with conditions. Install if you train on NVIDIA CUDA GPUs and need to reduce training memory for large models. The low install friction, active maintenance, permissive license, and zero known vulnerabilities support adoption. However, it is early-stage (Alpha), Linux/CUDA-only, and the first optimizer step incurs Triton JIT overhead. Verify convergence and performance on your specific models and hardware before production use.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- Requires NVIDIA CUDA GPU on Linux; torch and triton must be installed; first optimizer step is slower due to Triton kernel JIT compilation.
- Low install friction with a pure Python wheel.
- Active maintenance (last commit 2026-07-09) and early-stage status (Alpha, first release 2026-02-28) suggest ongoing development.
License · maintenance · safety
permissive license (permissive) — Apache License 2.0 is permissive, allowing commercial and private use with minimal restrictions—you may use, modify, and distribute FlashOptim freely provided you include license notices and disclaim warranties.
last release 2026-04-17 (119 days) · last repo commit 2026-07-09 · 259 stars
0 known vulnerabilities (OSV.dev, 2026-08-14) · 82,187 downloads/mo, #14,177 on PyPI
Alternatives
Verify before relying
pip install flashoptim
import torch
from flashoptim import FlashAdamW, cast_model
model = torch.nn.Linear(128, 10).cuda()
cast_model(model, dtype=torch.bfloat16)
optimizer = FlashAdamW(model.parameters(), lr=1e-3)
x = torch.randn(32, 128, device="cuda", dtype=torch.bfloat16)
loss = model(x).sum()
loss.backward()
optimizer.step()- Actual memory savings percentages (35% peak, 57% checkpoint reduction) on representative models and hardware configurations beyond the 8B finetuning example.
- Convergence equivalence claims across different model architectures, scales, and training regimes.
- Performance overhead of quantization and fused kernels on different GPU generations and batch sizes.
- Compatibility with distributed training frameworks (DDP, FSDP) and mixed-precision training workflows.
What it is and what it does
FlashOptim is a library of PyTorch optimizer implementations that reduce peak training memory by compressing optimizer states, master weights, and gradients through quantization and fused Triton kernels. It provides drop-in replacements for standard optimizers—FlashSGD, FlashSGDW, FlashAdam, FlashAdamW, and FlashLion—that follow the standard PyTorch optimizer API, so you can swap them in with minimal code changes.
The library works by splitting weight representation and quantizing optimizer moments to 8-bit while maintaining master weights at configurable precision (24-bit or 32-bit by default). All compression operations are fused into the update kernel to avoid overhead. It supports optional gradient release for further memory reduction and can produce checkpoints with quantized optimizer states. Training in reduced precision (bf16/fp16) does not degrade convergence according to the documentation.
Use it for
- Fine-tuning large language models (8B+) on memory-constrained GPUs to fit larger batch sizes or longer sequences.
- Reducing peak memory during training to enable training on smaller GPUs or with larger models.
- Storing compressed checkpoints that are substantially smaller than standard optimizer state files.
- Training workflows where gradient release timing is critical to minimize intermediate memory peaks.
- Migrating existing PyTorch training code to use memory-efficient optimizers without rewriting training loops.
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes, with conditions.
Install if you train on NVIDIA CUDA GPUs and need to reduce training memory for large models. The low install friction, active maintenance, permissive license, and zero known vulnerabilities support adoption. However, it is early-stage (Alpha), Linux/CUDA-only, and the first optimizer step incurs Triton JIT overhead. Verify convergence and performance on your specific models and hardware before production use.
Install
flashoptim on PyPI
Before you install
Low install friction with a pure Python wheel. Active maintenance (last commit 2026-07-09) and early-stage status (Alpha, first release 2026-02-28) suggest ongoing development. Requires torch and triton as runtime dependencies.
Requires NVIDIA CUDA GPU on Linux; torch and triton must be installed; first optimizer step is slower due to Triton kernel JIT compilation.
License in practice
Apache License 2.0 is permissive, allowing commercial and private use with minimal restrictions—you may use, modify, and distribute FlashOptim freely provided you include license notices and disclaim warranties.
Quickstart
pip install flashoptim
import torch
from flashoptim import FlashAdamW, cast_model
model = torch.nn.Linear(128, 10).cuda()
cast_model(model, dtype=torch.bfloat16)
optimizer = FlashAdamW(model.parameters(), lr=1e-3)
x = torch.randn(32, 128, device="cuda", dtype=torch.bfloat16)
loss = model(x).sum()
loss.backward()
optimizer.step()Verify before relying
- Actual memory savings percentages (35% peak, 57% checkpoint reduction) on representative models and hardware configurations beyond the 8B finetuning example.
- Convergence equivalence claims across different model architectures, scales, and training regimes.
- Performance overhead of quantization and fused kernels on different GPU generations and batch sizes.
- Compatibility with distributed training frameworks (DDP, FSDP) and mixed-precision training workflows.
Package facts
| License | permissive license permissive |
| Python support | Supports the current Python release >=3.9 |
| Install friction | Low. Pure-Python wheel |
| Runtime dependencies | 2 packagestorchtriton |
| Maintenance | Actively maintained 119 days since the last release |
| Last repo commit | |
| First released | |
| Downloads | 82,187 / month, #14,177 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
| Classifiers | Development Status :: 3 - AlphaProgramming Language :: Python :: 3Topic :: Scientific/Engineering :: Artificial Intelligence |
Evidence: flashoptim-0.1.4-py3-none-any.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “memory-efficient pytorch optimizers”
- flashoptimFlashOptim provides drop-in replacements for PyTorch optimizers (SGD,…
- torch-optimizerProvides a collection of alternative optimization algorithms for…
- pytorch_optimizerProvides a collection of modern optimizers, learning rate schedulers,…
Give your agent the search over MCP, or paste the wish link into any chat.
More Artificial Intelligence packages
LiteLLM provides a unified Python interface to call 100+ LLM providers (OpenAI, Anthropic, Gemini, Bedrock, Azure, and others) using OpenAI-compatible API format, available as both a Python SDK and a self-hosted AI Gateway proxy server.
Install it if you need to work with multiple LLM providers or want to centralize LLM routing in your organization.
Client library and CLI tool for downloading, uploading, and managing models, datasets, and repositories on the Hugging Face Hub platform.
Install it if you work with Hugging Face Hub models or datasets.
LangChain provides a framework for building agents and LLM-powered applications by composing language models, tools, and memory through a unified API that abstracts over multiple model providers.
hf-xet provides chunk-based deduplication and efficient file transfer for the Hugging Face Hub, enabling faster uploads and downloads of large files with local disk caching.
Tokenizers converts raw text into token sequences for NLP models, with support for training custom vocabularies and using pre-built tokenizers (BPE, WordPiece) optimized for speed via Rust.
Transformers provides a unified framework for loading, fine-tuning, and running state-of-the-art pretrained models across text, vision, audio, video, and multimodal tasks using PyTorch, JAX, or TensorFlow.
Install it if you need to run or train any transformer-based model for NLP, vision, audio, or multimodal tasks.
See also lion-pytorch · schedulefree · prodigyopt · adam-atan2-pytorch · torch-optimizer · pytorch-ranger · pytorch_optimizer · optimum-quanto · comfy-aimdo · codeflash