lion-pytorch

What it is and what it does

Lion is a PyTorch optimizer implementing an evolved sign momentum algorithm that the description positions as a potential successor to AdamW. It depends only on torch and integrates directly into PyTorch's optimizer interface. The optimizer requires careful hyperparameter tuning: learning rates should typically be 3–10x smaller than AdamW equivalents, weight decay values 3–10x larger, and default β1 and β2 values differ from AdamW (0.9 and 0.99 versus 0.9 and 0.999). The description documents mixed empirical results—positive outcomes reported for language modeling and text-to-image training when tuned correctly, but negative results outside the paper's tested domains (reinforcement learning, feedforward networks, hybrid architectures). The author recommends Lion primarily for high batch sizes (64 or above) and notes sensitivity to batch size, data volume, and augmentation.

Use it for:

• Training large language models where a 3x smaller learning rate than AdamW yields better convergence
• Text-to-image model training when hyperparameters are carefully tuned to the specific architecture
• Vision transformer training with cosine decay learning rate schedules
• Scenarios with high batch sizes (64+) where Lion's sign momentum approach may outperform adaptive methods

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Lion is a PyTorch optimizer that implements an evolved sign momentum algorithm discovered by Google Brain as an alternative to AdamW for training neural networks.

Yes, with conditions. Lion is permissively licensed, has low install friction, and is actively maintained. Install it if you are training models in domains the paper evaluated (language modeling, vision transformers, text-to-image) and are willing to invest in hyperparameter tuning. The description's own updates acknowledge it performs worse than Adam without careful learning rate adjustment and shows negative results outside tested architectures. Not recommended for reinforcement learning, standard feedforward networks, or exploratory work where you cannot afford tuning overhead.

Install

lion-pytorch on PyPI

pip install lion-pytorch

uv add lion-pytorch

poetry add lion-pytorch

Installing lion-pytorch

Before you install

Low install friction with a single runtime dependency on torch. Actively maintained with a recent release 36 days ago.

License in practice

MIT License permits unrestricted use, modification, and distribution in both open-source and commercial projects with minimal restrictions.

Quickstart

pip install lion-pytorch

import torch
from torch import nn
from lion_pytorch import Lion

model = nn.Linear(10, 1)
opt = Lion(model.parameters(), lr=1e-4, weight_decay=1e-2)

loss = model(torch.randn(10))
loss.backward()
opt.step()
opt.zero_grad()

Requires Python 3.9 or later. Learning rate and weight decay require careful tuning (typically 3–10x smaller/larger than AdamW); default hyperparameters may not work without adjustment.

Verify before relying

• Whether Lion's performance gains over AdamW hold across a broad range of model architectures and training regimes beyond those tested in the original paper
• Optimal learning rate schedules and hyperparameter tuning strategies for specific problem domains
• Stability and convergence behavior with batch sizes below 64, given the author's recommendation for high batch sizes

Package facts

Evidence: lion_pytorch-0.2.5-py3-none-any.whl

Keywords: artificial intelligence, deep learning, optimizers

Tags

More Artificial Intelligence packages

Further reading