seqio-nightly

What it is and what it does

SeqIO is a library for building scalable data pipelines for sequence models, originally extracted from the T5 project. It abstracts the common pattern of combining a data source (TFDS, text files, TFRecord, or custom functions), applying preprocessing steps, defining output features with vocabularies and tokenization, and registering evaluation metrics into named Tasks that can be composed into Mixtures. The library uses tf.data.Dataset internally but is framework-agnostic: you can convert the output to numpy iterators for use with jax, or other frameworks with a single line of code.

SeqIO is designed for sequence-based modalities—text and audio are natural fits, and images work when represented as sequences. It handles the boilerplate of task definition and dataset construction, letting you focus on defining what your data looks like and how to preprocess it. Tasks are typically registered globally so they can be referenced by name in model configs and training scripts.

Use it for:

• Build machine translation pipelines combining TFDS datasets with language-pair-specific preprocessing and evaluation metrics
• Define text-to-text tasks with separate input and target features, vocabularies, and custom postprocessors for evaluation
• Create reusable task mixtures that combine multiple datasets for multi-task learning with a single registry lookup
• Preprocess raw text or audio data from files or TFRecord formats into tokenized sequences ready for model training
• Evaluate model outputs using task-specific metrics by defining postprocessors that convert detokenized predictions back to evaluation format

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

SeqIO provides task-based data pipelines for sequence models, combining raw data sources, preprocessing, vocabularies, and evaluation metrics into reusable registered tasks that output tf.data.Dataset instances compatible with JAX, PyTorch, and other frameworks.

Yes, if you are building NLP or sequence-model training pipelines and want a structured, reusable way to define tasks and datasets. The library is actively maintained, has low install friction, and is permissively licensed. However, this is a nightly build (0.0.18.dev20250227); verify that the development version is appropriate for your use case. The 11 runtime dependencies add significant install time and disk space.

Install

seqio-nightly on PyPI

pip install seqio-nightly

uv add seqio-nightly

poetry add seqio-nightly

Installing seqio-nightly

Before you install

Low install friction with a pure-Python wheel distribution. Active maintenance with recent releases; last commit 2026-07-02. Depends on 11 runtime packages including jax, tensorflow-text, and tfds-nightly, which may require system-level dependencies or compilation time on first install.

License in practice

Apache 2.0 permissive license allows commercial and private use with minimal restrictions; you must retain license notices and may not hold the authors liable.

Quickstart

pip install seqio-nightly

import seqio

seqio.TaskRegistry.add(
    "my_task",
    seqio.TfdsDataSource(tfds_name="dataset_name"),
    preprocessors=[seqio.preprocessors.tokenize],
    output_features={
        'inputs': seqio.Feature(seqio.SentencePieceVocabulary('/path/to/vocab')),
        'targets': seqio.Feature(seqio.SentencePieceVocabulary('/path/to/vocab'))
    }
)

dataset = seqio.get_dataset("my_task")

Requires TensorFlow and jax as runtime dependencies; initial install of these packages may require compilation or system libraries.

Verify before relying

• Whether the nightly build (0.0.18.dev20250227) is suitable for production use versus requiring a stable release
• Specific Python version compatibility, as requires_python is unspecified in the metadata

Package facts

Evidence: seqio_nightly-0.0.18.dev20250227-py3-none-any.whl

Keywords: sequence, preprocessing, nlp, machinelearning

Tags

More Artificial Intelligence packages