schedulefree

What it is and what it does

Schedule-free learning replaces traditional learning rate schedules with a method that combines interpolation and averaging of optimizer iterates, eliminating the need to specify training duration in advance. The package provides PyTorch implementations of SGDScheduleFree, AdamWScheduleFree, and RAdamScheduleFree, each with a standard form and a reference form that trades memory for numerical stability. The core insight is that the optimizer maintains three sequences: one for gradient evaluation, one for parameter updates, and one for test/validation evaluation, with the averaging mechanism allowing the method to achieve fast early convergence without the instability of standard momentum-based approaches.

The method requires explicit calls to optimizer.train() and optimizer.eval() to switch between the gradient evaluation and validation sequences, and learning rates typically need to be larger than classical schedules. It works without a learning rate schedule but remains compatible with one, and supports warmup through a warmup_steps parameter. The approach is based on a research preprint and has active maintenance with community contributions.

Use it for:

• Training deep neural networks without manually tuning learning rate decay schedules or specifying total training steps upfront.
• Experimenting with different training durations without retuning the optimizer configuration for each run length.
• Wrapping existing PyTorch optimizers via ScheduleFreeWrapper to add schedule-free behavior to custom optimization setups.
• Research projects requiring comparison with Polyak-Ruppert averaging or Nesterov acceleration methods.
• Image classification tasks like MNIST where provided examples demonstrate the integration pattern.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Provides schedule-free optimizers for PyTorch that eliminate the need to specify stopping time in advance, replacing momentum with interpolation and averaging to achieve faster training without learning rate schedules.

Yes, if you want to eliminate learning rate schedule tuning and are willing to retune learning rate magnitude and the beta interpolation parameter. The low install friction, active maintenance, permissive license, and zero known vulnerabilities make it safe to try. Not worth installing if your workflow is heavily dependent on standard schedules or if you cannot modify optimizer.train()/eval() calls around your training loop.

Install

schedulefree on PyPI

pip install schedulefree

uv add schedulefree

poetry add schedulefree

Installing schedulefree

Before you install

Low friction installation with only torch and typing-extensions as runtime dependencies. Active maintenance with recent commits and steady release cadence since initial release in April 2024.

License in practice

Permissive Apache license allows commercial and private use with minimal restrictions, making it suitable for most projects.

Quickstart

pip install schedulefree

import torch
from schedulefree import AdamWScheduleFree

optimizer = AdamWScheduleFree(model.parameters())
for epoch in range(num_epochs):
    optimizer.train()
    # training loop
    optimizer.eval()
    # validation loop

Requires calling optimizer.train() and optimizer.eval() at the same points as model.train() and model.eval(); BatchNorm models need additional handling to update running statistics at the correct sequence point.

Verify before relying

• Whether the experimental ScheduleFreeWrapper is production-ready or remains research-stage only.
• Performance gains relative to standard schedules on specific model architectures or domains.
• Compatibility with distributed training frameworks and gradient accumulation patterns.
• Specific learning rate multiplier guidance for different model types beyond the 10x-50x range for SGD and 1x-10x for AdamW.

Package facts

Evidence: schedulefree-1.4.1-py3-none-any.whl

Tags

More Artificial Intelligence packages

Further reading