---
id: tensorflow-text
version: "2.20.1"
license: Apache 2.0
license_treatment: permissive
maintenance: active
---

# tensorflow-text — TF.Text is a TensorFlow library of text related ops, modules, and subgraphs.

License: permissive · Maintenance: active · Downloads: 5.7M/mo

## What it is and what it does

TensorFlow Text is a library of text-related operations and tokenizers designed to run within TensorFlow's computation graph. It provides utilities for Unicode normalization, case folding, tokenization (whitespace, Unicode script-based, and character-level), n-gram generation, and other text preprocessing tasks commonly needed for NLP models. The key advantage is that all preprocessing happens inside the TensorFlow graph, ensuring identical behavior between training and inference without needing separate preprocessing scripts.

The library depends solely on TensorFlow and is distributed as precompiled wheels for multiple Python versions (3.9–3.13) and architectures (x86_64, aarch64, macOS ARM64). It handles UTF-8 strings natively and supports operations like byte offset tracking during tokenization, integration with tf.data pipelines, and Keras API compatibility for ragged tensor inputs.

Use it for:

- Normalize and tokenize text within a TensorFlow model pipeline to ensure consistent preprocessing at training and serving time.
- Split multilingual or punctuation-heavy text using Unicode script boundaries instead of simple whitespace splitting.
- Extract byte offsets during tokenization to trace tokens back to their original positions in source text.
- Build end-to-end NLP models in Keras that accept raw text and produce token sequences without external preprocessing.
- Handle non-ASCII Unicode text with case folding and normalization forms (NFKC, NFKD) for case-insensitive matching.

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

TensorFlow Text provides text preprocessing operations and tokenizers that run within the TensorFlow computation graph, enabling consistent text handling across training and inference without external preprocessing scripts.

Yes. TensorFlow Text is actively maintained, widely used (top 5000 PyPI packages), has no known vulnerabilities, and solves a real problem—keeping text preprocessing inside the TensorFlow graph. Install it if you are building NLP models with TensorFlow and need tokenization or text normalization; ensure your tensorflow-text version matches your TensorFlow minor version.

## Install

pip install tensorflow-text
uv add tensorflow-text
poetry add tensorflow-text

## Installing tensorflow-text

Before you install: Medium install friction due to compiled wheel dependencies across multiple Python versions and architectures. Active maintenance with recent releases; repository shows 1293 stars and last commit on 2026-08-12.

License in practice: Apache 2.0 permissive license allows commercial and private use with minimal restrictions; suitable for most projects.

Quickstart:

pip install tensorflow-text==2.20.1

import tensorflow as tf
import tensorflow_text as text

tokenizer = text.WhitespaceTokenizer()
tokens = tokenizer.tokenize(['hello world'])
print(tokens.to_list())

Requires TensorFlow installed; version 2.20.1 of tensorflow-text should match the minor version of your TensorFlow installation (e.g., tensorflow==2.20.x).

Verify before relying:

- Exact Python version requirements beyond the classifiers (3.9, 3.10, 3.11, 3.12, 3.13 listed but requires_python field is null)
- Whether all tokenizers and ops are available on all supported platforms or if some are platform-specific

## Package facts

- License: Apache 2.0 (permissive)
- Python support: unspecified
- Install friction: medium
- Maintenance: active
- Downloads: 5.7M/month (top 5,000 on PyPI)
- Known vulnerabilities: none known

## Tags

tensorflow text preprocessing, tokenization in tensorflow, text normalization tensorflow, unicode text processing, nlp text ops tensorflow, whitespace tokenizer, unicode script tokenizer, nlp, tensorflow-ecosystem, graph-based-preprocessing

[View on SkillFed](https://skillfed.io/packages/tensorflow-text) · [View on PyPI](https://pypi.org/project/tensorflow-text/)