fev

What it is and what it does

fev is a benchmarking framework designed to standardize time series forecasting evaluation. It sits between standalone datasets (which offer no reproducibility guarantees) and monolithic end-to-end systems (which bundle models, data, and tasks tightly). The library provides Task objects that wrap datasets from Hugging Face Hub, define evaluation windows and horizons, and compute metrics consistently across different forecasting models.

You define a forecasting task once—specifying dataset, horizon, covariates, and metrics—then iterate over rolling evaluation windows to collect predictions from your model. fev handles the metric computation and produces an evaluation summary that uniquely identifies the task and captures results. Multiple summaries can be aggregated into leaderboards for model comparison. The library is built on top of datasets and scipy, keeping dependencies minimal while supporting point and probabilistic forecasting.

Use it for:

• Benchmark your own forecasting model against standardized tasks to compare performance with other models.
• Create reproducible forecasting benchmarks that other researchers can run identically, ensuring comparable results.
• Aggregate results from multiple forecasting models into a leaderboard to rank them by skill score and win rate.
• Evaluate forecasting models on datasets stored on Hugging Face Hub without downloading data locally.
• Define custom forecasting tasks with specific horizons, covariates, and metrics for domain-specific evaluation.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

fev is a lightweight benchmarking library for time series forecasting models that provides standardized evaluation workflows, reproducible task definitions, and metric computation without heavy dependencies.

Yes. fev is actively maintained, has low install friction, carries permissive licensing, and fills a genuine gap in forecasting evaluation tooling. It is well-suited for researchers and practitioners who need reproducible benchmarking without the overhead of monolithic systems. The recent release cadence and active repository signal ongoing support.

Install

fev on PyPI

pip install fev

uv add fev

poetry add fev

Installing fev

Before you install

Low install friction: pure Python wheel with only four runtime dependencies (datasets, numpy, pydantic, scipy). Active maintenance with recent releases; last commit 2026-08-14.

License in practice

Apache License 2.0 permits commercial and derivative use with attribution. No restrictions on bundling or modification.

Quickstart

pip install fev

import fev

task = fev.Task(
    dataset_path="autogluon/chronos_datasets",
    dataset_config="m4_hourly",
    horizon=24,
)

for window in task.iter_windows():
    past_data, future_data = window.get_input_data()

task.evaluation_summary(predictions, model_name="my_model")

Requires Python >=3.10. Datasets are loaded from Hugging Face Hub; internet access needed for default usage.

Verify before relying

• Whether adapters for popular forecasting libraries are production-ready or experimental.
• Performance characteristics when evaluating large-scale datasets or many models in sequence.
• Stability of the leaderboard infrastructure and submission process for contributed results.

Package facts

Evidence: fev-0.9.0-py3-none-any.whl

Tags

More Artificial Intelligence packages