tensorflow-text
TF.Text is a TensorFlow library of text related ops, modules, and subgraphs.
Decision gist · record as of 2026-08-14
Yes. TensorFlow Text is actively maintained, widely used (top 5000 PyPI packages), has no known vulnerabilities, and solves a real problem—keeping text preprocessing inside the TensorFlow graph. Install it if you are building NLP models with TensorFlow and need tokenization or text normalization; ensure your tensorflow-text version matches your TensorFlow minor version.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- Requires TensorFlow installed; version 2.20.1 of tensorflow-text should match the minor version of your TensorFlow installation (e.g., tensorflow==2.20.x).
- Medium install friction due to compiled wheel dependencies across multiple Python versions and architectures.
- Active maintenance with recent releases; repository shows 1293 stars and last commit on 2026-08-12.
License · maintenance · safety
Apache 2.0 (permissive) — Apache 2.0 permissive license allows commercial and private use with minimal restrictions; suitable for most projects.
last release 2026-03-10 (157 days) · last repo commit 2026-08-12 · 1,293 stars
0 known vulnerabilities (OSV.dev, 2026-08-14) · 5,745,982 downloads/mo, #2,043 on PyPI
Alternatives
Verify before relying
pip install tensorflow-text==2.20.1
import tensorflow as tf
import tensorflow_text as text
tokenizer = text.WhitespaceTokenizer()
tokens = tokenizer.tokenize(['hello world'])
print(tokens.to_list())- Exact Python version requirements beyond the classifiers (3.9, 3.10, 3.11, 3.12, 3.13 listed but requires_python field is null)
- Whether all tokenizers and ops are available on all supported platforms or if some are platform-specific
What it is and what it does
TensorFlow Text is a library of text-related operations and tokenizers designed to run within TensorFlow's computation graph. It provides utilities for Unicode normalization, case folding, tokenization (whitespace, Unicode script-based, and character-level), n-gram generation, and other text preprocessing tasks commonly needed for NLP models. The key advantage is that all preprocessing happens inside the TensorFlow graph, ensuring identical behavior between training and inference without needing separate preprocessing scripts.
The library depends solely on TensorFlow and is distributed as precompiled wheels for multiple Python versions (3.9–3.13) and architectures (x86_64, aarch64, macOS ARM64). It handles UTF-8 strings natively and supports operations like byte offset tracking during tokenization, integration with tf.data pipelines, and Keras API compatibility for ragged tensor inputs.
Use it for
- Normalize and tokenize text within a TensorFlow model pipeline to ensure consistent preprocessing at training and serving time.
- Split multilingual or punctuation-heavy text using Unicode script boundaries instead of simple whitespace splitting.
- Extract byte offsets during tokenization to trace tokens back to their original positions in source text.
- Build end-to-end NLP models in Keras that accept raw text and produce token sequences without external preprocessing.
- Handle non-ASCII Unicode text with case folding and normalization forms (NFKC, NFKD) for case-insensitive matching.
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes.
TensorFlow Text is actively maintained, widely used (top 5000 PyPI packages), has no known vulnerabilities, and solves a real problem—keeping text preprocessing inside the TensorFlow graph. Install it if you are building NLP models with TensorFlow and need tokenization or text normalization; ensure your tensorflow-text version matches your TensorFlow minor version.
Install
tensorflow-text on PyPI
Before you install
Medium install friction due to compiled wheel dependencies across multiple Python versions and architectures. Active maintenance with recent releases; repository shows 1293 stars and last commit on 2026-08-12.
Requires TensorFlow installed; version 2.20.1 of tensorflow-text should match the minor version of your TensorFlow installation (e.g., tensorflow==2.20.x).
License in practice
Apache 2.0 permissive license allows commercial and private use with minimal restrictions; suitable for most projects.
Quickstart
pip install tensorflow-text==2.20.1
import tensorflow as tf
import tensorflow_text as text
tokenizer = text.WhitespaceTokenizer()
tokens = tokenizer.tokenize(['hello world'])
print(tokens.to_list())Verify before relying
- Exact Python version requirements beyond the classifiers (3.9, 3.10, 3.11, 3.12, 3.13 listed but requires_python field is null)
- Whether all tokenizers and ops are available on all supported platforms or if some are platform-specific
Package facts
| License | Apache 2.0 permissive |
| Python support | Not specified |
| Install friction | Medium. Platform-specific wheel |
| Runtime dependencies | 1 packagetensorflow |
| Maintenance | Actively maintained 157 days since the last release |
| Last repo commit | |
| First released | |
| Downloads | 5,745,982 / month, #2,043 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
| Classifiers | Development Status :: 5 - Production/StableIntended Audience :: DevelopersIntended Audience :: EducationIntended Audience :: Science/ResearchLicense :: OSI Approved :: Apache Software LicenseProgramming Language :: Python :: 3Programming Language :: Python :: 3 :: OnlyProgramming Language :: Python :: 3.10Programming Language :: Python :: 3.11Programming Language :: Python :: 3.12Programming Language :: Python :: 3.13Programming Language :: Python :: 3.9Topic :: Scientific/EngineeringTopic :: Scientific/Engineering :: Artificial IntelligenceTopic :: Software DevelopmentTopic :: Software Development :: LibrariesTopic :: Software Development :: Libraries :: Python Modules |
Evidence: tensorflow_text-2.20.1-cp310-cp310-macosx_11_0_arm64.whl; tensorflow_text-2.20.1-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.whl; tensorflow_text-2.20.1-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl; tensorflow_text-2.20.1-cp311-cp311-macosx_11_0_arm64.whl; tensorflow_text-2.20.1-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl; tensorflow_text-2.20.1-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl; tensorflow_text-2.20.1-cp312-cp312-macosx_11_0_arm64.whl; tensorflow_text-2.20.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl; tensorflow_text-2.20.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl; tensorflow_text-2.20.1-cp313-cp313-macosx_11_0_arm64.whl; tensorflow_text-2.20.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl; tensorflow_text-2.20.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl; tensorflow_text-2.20.1-cp39-cp39-macosx_11_0_arm64.whl; tensorflow_text-2.20.1-cp39-cp39-manylinux2014_aarch64.manylinux_2_17_aarch64.whl; tensorflow_text-2.20.1-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “tensorflow text preprocessing”
- tensorflow-textTensorFlow Text provides text preprocessing operations and tokenizers…
- seqioSeqIO builds scalable data pipelines for sequence models using…
- urduhackUrduhack provides NLP preprocessing, tokenization, part-of-speech…
Give your agent the search over MCP, or paste the wish link into any chat.
More Software Development packages
Provides backported and experimental type hints for Python 3.9+, allowing use of newer typing features on older Python versions and enabling early experimentation with type system PEPs before they enter the standard library.
NumPy provides an N-dimensional array object and a comprehensive suite of mathematical, linear algebra, Fourier transform, and random number functions for scientific computing in Python.
FastAPI is a Python web framework for building REST APIs using type hints, with automatic request validation, serialization, and interactive API documentation.
Provides a way to document function parameters, class attributes, return types, and variables inline using Python's `Annotated` type hint syntax instead of traditional docstrings.
Typer builds command-line applications from Python functions using type hints, automatically generating help text, argument parsing, and shell completion.
Install it if you are building CLIs in Python.
Distlib provides low-level packaging utilities for building, distributing, and managing Python software—including metadata handling, version specifiers, wheel support, script installation, and dependency resolution.
See also match · tokenizer · proces · seqio · jieba3k · sentencepiece · segtok · tokenizers · urduhack · emot