schedulefree

What it is and what it does

Schedule-free learning replaces traditional learning rate schedules with a method that combines interpolation and averaging of optimizer iterates, eliminating the need to specify training duration in advance. The package provides PyTorch implementations of SGDScheduleFree, AdamWScheduleFree, and RAdamScheduleFree, each with a standard form and a reference form that trades memory for numerical stability. The core insight is that the optimizer maintains three sequences: one for gradient evaluation, one for parameter updates, and one for test/validation evaluation, with the averaging mechanism allowing the method to achieve fast early convergence without the instability of standard momentum-based approaches.

The method requires explicit calls to optimizer.train() and optimizer.eval() to switch between the gradient evaluation and validation sequences, and learning rates typically need to be larger than classical schedules. It works without a learning rate schedule but remains compatible with one, and supports warmup through a warmup_steps parameter. The approach is based on a research preprint and has active maintenance with community contributions.

Use it for

• Training deep neural networks without manually tuning learning rate decay schedules or specifying total training steps upfront.
• Experimenting with different training durations without retuning the optimizer configuration for each run length.
• Wrapping existing PyTorch optimizers via ScheduleFreeWrapper to add schedule-free behavior to custom optimization setups.
• Research projects requiring comparison with Polyak-Ruppert averaging or Nesterov acceleration methods.
• Image classification tasks like MNIST where provided examples demonstrate the integration pattern.

Worth the install?

AI-flagged interpretation of the facts on this page. Verify before relying on it.

Install

schedulefree on PyPI

Before you install

Low friction installation with only torch and typing-extensions as runtime dependencies. Active maintenance with recent commits and steady release cadence since initial release in April 2024.

Requires calling optimizer.train() and optimizer.eval() at the same points as model.train() and model.eval(); BatchNorm models need additional handling to update running statistics at the correct sequence point.

License in practice

Permissive Apache license allows commercial and private use with minimal restrictions, making it suitable for most projects.

Quickstart

pip install schedulefree

import torch
from schedulefree import AdamWScheduleFree

optimizer = AdamWScheduleFree(model.parameters())
for epoch in range(num_epochs):
    optimizer.train()
    # training loop
    optimizer.eval()
    # validation loop

Package facts

Evidence: schedulefree-1.4.1-py3-none-any.whl

Tags

More Artificial Intelligence packages

See also lion-pytorch · pytorch_optimizer · prodigyopt · flashoptim · pytorch-ranger · torch-optimizer · adam-atan2-pytorch · torch-ema · fsrs · accelerate

Further reading