tensorflow-text

What it is and what it does

TensorFlow Text is a library of text-related operations and tokenizers designed to run within TensorFlow's computation graph. It provides utilities for Unicode normalization, case folding, tokenization (whitespace, Unicode script-based, and character-level), n-gram generation, and other text preprocessing tasks commonly needed for NLP models. The key advantage is that all preprocessing happens inside the TensorFlow graph, ensuring identical behavior between training and inference without needing separate preprocessing scripts.

The library depends solely on TensorFlow and is distributed as precompiled wheels for multiple Python versions (3.9–3.13) and architectures (x86_64, aarch64, macOS ARM64). It handles UTF-8 strings natively and supports operations like byte offset tracking during tokenization, integration with tf.data pipelines, and Keras API compatibility for ragged tensor inputs.

Use it for:

• Normalize and tokenize text within a TensorFlow model pipeline to ensure consistent preprocessing at training and serving time.
• Split multilingual or punctuation-heavy text using Unicode script boundaries instead of simple whitespace splitting.
• Extract byte offsets during tokenization to trace tokens back to their original positions in source text.
• Build end-to-end NLP models in Keras that accept raw text and produce token sequences without external preprocessing.
• Handle non-ASCII Unicode text with case folding and normalization forms (NFKC, NFKD) for case-insensitive matching.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

TensorFlow Text provides text preprocessing operations and tokenizers that run within the TensorFlow computation graph, enabling consistent text handling across training and inference without external preprocessing scripts.

Yes. TensorFlow Text is actively maintained, widely used (top 5000 PyPI packages), has no known vulnerabilities, and solves a real problem—keeping text preprocessing inside the TensorFlow graph. Install it if you are building NLP models with TensorFlow and need tokenization or text normalization; ensure your tensorflow-text version matches your TensorFlow minor version.

Install

tensorflow-text on PyPI

pip install tensorflow-text

uv add tensorflow-text

poetry add tensorflow-text

Installing tensorflow-text

Before you install

Medium install friction due to compiled wheel dependencies across multiple Python versions and architectures. Active maintenance with recent releases; repository shows 1293 stars and last commit on 2026-08-12.

License in practice

Apache 2.0 permissive license allows commercial and private use with minimal restrictions; suitable for most projects.

Quickstart

pip install tensorflow-text==2.20.1

import tensorflow as tf
import tensorflow_text as text

tokenizer = text.WhitespaceTokenizer()
tokens = tokenizer.tokenize(['hello world'])
print(tokens.to_list())

Requires TensorFlow installed; version 2.20.1 of tensorflow-text should match the minor version of your TensorFlow installation (e.g., tensorflow==2.20.x).

Verify before relying

• Exact Python version requirements beyond the classifiers (3.9, 3.10, 3.11, 3.12, 3.13 listed but requires_python field is null)
• Whether all tokenizers and ops are available on all supported platforms or if some are platform-specific

Package facts

Evidence: tensorflow_text-2.20.1-cp310-cp310-macosx_11_0_arm64.whl; tensorflow_text-2.20.1-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.whl; tensorflow_text-2.20.1-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl; tensorflow_text-2.20.1-cp311-cp311-macosx_11_0_arm64.whl; tensorflow_text-2.20.1-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl; tensorflow_text-2.20.1-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl; tensorflow_text-2.20.1-cp312-cp312-macosx_11_0_arm64.whl; tensorflow_text-2.20.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl; tensorflow_text-2.20.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl; tensorflow_text-2.20.1-cp313-cp313-macosx_11_0_arm64.whl; tensorflow_text-2.20.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl; tensorflow_text-2.20.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl; tensorflow_text-2.20.1-cp39-cp39-macosx_11_0_arm64.whl; tensorflow_text-2.20.1-cp39-cp39-manylinux2014_aarch64.manylinux_2_17_aarch64.whl; tensorflow_text-2.20.1-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl

Keywords: tensorflow, text, machine, learning

Tags

More Software Development packages