torchtext

What it is and what it does

torchtext is a PyTorch utility library for text processing and NLP workflows. It bundles raw text dataset iterators for common benchmarks (WikiText, SQuAD, AG_NEWS, etc.), basic NLP building blocks (tokenization, vocabulary), text transformations, and pre-trained models (RoBERTa, T5, XLM-R). The library depends on torch, numpy, requests, and tqdm.

As of September 2023, active development has been paused and the repository is archived. The project will receive maintenance releases but no new features. This means the library is stable for existing use cases but will not evolve to support emerging NLP techniques or datasets.

Use it for:

• Loading and preprocessing standard NLP benchmarks like AG_NEWS or SST-2 for text classification experiments.
• Building machine translation pipelines using Multi30k or IWSLT datasets with transformer models.
• Language modeling tasks with WikiText2 or WikiText103 datasets for pretraining or fine-tuning.
• Applying scriptable tokenizers (SentencePiece, GPT-2 BPE, BERT) to raw text in production workflows.
• Accessing pre-trained RoBERTa or T5 models for transfer learning on downstream NLP tasks.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

torchtext provides text datasets, preprocessing transforms, and pre-trained models for natural language processing tasks in PyTorch, including tokenizers, vocabulary management, and example workflows.

Yes, but with caution. torchtext remains useful for standard NLP tasks and dataset loading, with no known security vulnerabilities and permissive licensing. However, the abandoned maintenance status (paused since September 2023, archived repository) means no new features or dataset support will be added. Install it if you need its specific datasets or models for existing workflows, but do not expect updates for modern NLP techniques or emerging benchmarks. For new projects, consider actively maintained alternatives.

Install

torchtext on PyPI

pip install torchtext

uv add torchtext

poetry add torchtext

Installing torchtext

Before you install

Medium install friction due to compiled dependencies (torch, numpy). Wheels are pre-built for Python 3.8–3.12 on macOS ARM64, Linux x86_64, and Windows. However, the project is archived and in abandoned maintenance status as of September 2023, with no active feature development planned.

License in practice

BSD license is permissive and poses no significant restrictions on use or redistribution in most contexts.

Quickstart

pip install torchtext

import torchtext
from torchtext.datasets import AG_NEWS

train_dataset = AG_NEWS(split='train')

Requires torch to be installed first; optional dependencies (spacy, sacremoses, sentencepiece) needed for specific tokenizers.

Verify before relying

• Whether pre-trained models (RoBERTa, T5, Flan-T5) are still downloadable and functional given the abandoned status.
• Current state of dataset availability and whether all listed datasets (WikiText2, SQuAD, IMDB, etc.) remain accessible.
• Compatibility with recent PyTorch versions beyond those explicitly documented in the version table.

Package facts

Evidence: torchtext-0.18.0-cp310-cp310-macosx_11_0_arm64.whl; torchtext-0.18.0-cp310-cp310-manylinux1_x86_64.whl; torchtext-0.18.0-cp310-cp310-win_amd64.whl; torchtext-0.18.0-cp311-cp311-macosx_11_0_arm64.whl; torchtext-0.18.0-cp311-cp311-manylinux1_x86_64.whl; torchtext-0.18.0-cp311-cp311-win_amd64.whl; torchtext-0.18.0-cp312-cp312-macosx_11_0_arm64.whl; torchtext-0.18.0-cp312-cp312-manylinux1_x86_64.whl; torchtext-0.18.0-cp312-cp312-win_amd64.whl; torchtext-0.18.0-cp38-cp38-macosx_11_0_arm64.whl; torchtext-0.18.0-cp38-cp38-manylinux1_x86_64.whl; torchtext-0.18.0-cp38-cp38-win_amd64.whl; torchtext-0.18.0-cp39-cp39-macosx_11_0_arm64.whl; torchtext-0.18.0-cp39-cp39-manylinux1_x86_64.whl; torchtext-0.18.0-cp39-cp39-win_amd64.whl

Tags

More Linguistic packages