---
id: seqio
version: "0.0.20"
license: Apache 2.0
license_treatment: permissive
maintenance: active
---

# seqio — SeqIO: Task-based datasets, preprocessing, and evaluation for sequence models.

License: permissive · Maintenance: active · Downloads: 301.4K/mo

## What it is and what it does

SeqIO is a library for constructing data pipelines for sequence models, built on top of TensorFlow's tf.data.Dataset. It abstracts away the complexity of loading raw data, applying preprocessing steps, tokenizing features with custom vocabularies, and computing evaluation metrics into a unified Task interface. The library was originally extracted from the T5 model's data pipeline and refactored for general use.

The package is designed to work primarily with sequential data—text and audio are naturally supported, and images can be used if represented as sequences. While it uses TensorFlow internally, SeqIO can output datasets as numpy iterators, making it fully compatible with JAX, PyTorch, and other frameworks. You define a Task by specifying a data source (TFDS, text files, TFRecord, or custom functions), preprocessing steps, output feature definitions with vocabularies, and metric functions, then use seqio.get_dataset to obtain a ready-to-use tf.data.Dataset.

Use it for:

- Building machine translation pipelines with preprocessing and BLEU evaluation for sequence-to-sequence models
- Creating text-to-text task datasets with custom tokenization and prompt formatting for transfer learning
- Preprocessing benchmark datasets from TensorFlow Datasets with task-specific metrics for model evaluation
- Combining multiple tasks into a Mixture for multi-task learning with unified data handling
- Converting raw text or audio files into tokenized sequences with vocabulary management for downstream models

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

SeqIO builds scalable data pipelines for sequence models using TensorFlow's tf.data.Dataset, with support for preprocessing, tokenization, and evaluation metrics, while remaining compatible with JAX, PyTorch, and other frameworks.

Yes, if you are building sequence models (NLP, audio, or sequence-based vision) and want a structured, reusable way to manage data pipelines with built-in preprocessing and evaluation. The low install friction and active maintenance support this. However, the 11 runtime dependencies—particularly TensorFlow, JAX, and TensorFlow Text—make it heavy for lightweight use cases; consider it only if you need task-based dataset abstraction and don't already have a simpler pipeline in place.

## Install

pip install seqio
uv add seqio
poetry add seqio

## Installing seqio

Before you install: Low friction installation with a pure-Python wheel. The package is actively maintained with recent releases and moderate popularity (top_15000 tier), though it carries 11 runtime dependencies including JAX, TensorFlow, and TensorFlow Text, which may add setup complexity in constrained environments.

License in practice: Licensed under Apache 2.0 (permissive), allowing free use, modification, and distribution with minimal restrictions—suitable for both open-source and commercial projects.

Quickstart:

pip install seqio

import seqio
import tensorflow as tf

task = seqio.TaskRegistry.add(
    "example_task",
    seqio.TfdsDataSource(tfds_name="dataset_name"),
    output_features={
        'inputs': seqio.Feature(seqio.PassThroughVocabulary(), dtype=tf.int32),
        'targets': seqio.Feature(seqio.PassThroughVocabulary(), dtype=tf.int32),
    }
)

dataset = seqio.get_dataset("example_task", batch_size=32)

Requires TensorFlow, JAX, and TensorFlow Text as runtime dependencies; compatible with other frameworks via numpy iterator conversion but TensorFlow installation is mandatory.

Verify before relying:

- Minimum Python version requirement (requires_python is unspecified in metadata)
- Whether all 11 runtime dependencies are truly required for basic usage or if some are optional
- Performance characteristics and scalability limits for very large datasets

## Package facts

- License: Apache 2.0 (permissive)
- Python support: unspecified
- Install friction: low
- Maintenance: active
- Downloads: 301.4K/month (top 15,000 on PyPI)
- Known vulnerabilities: none known

## Tags

sequence data preprocessing pipeline, nlp dataset preprocessing, machine learning data pipeline, text tokenization and evaluation, task-based dataset management, tensorflow data pipeline, sequence model training data, nlp-data-pipeline, tensorflow-ecosystem, sequence-models

[View on SkillFed](https://skillfed.io/packages/seqio) · [View on PyPI](https://pypi.org/project/seqio/)