tokenizer
A fast, compact pure-Python tokenizer for Icelandic text with sentence segmentation
Decision gist · record as of 2026-08-14
Yes. The package is actively maintained, has zero known vulnerabilities, installs with no dependencies, and is production-stable. Install it if you need to tokenize or segment Icelandic text. It is not useful for other languages.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- Requires Python 3.10 or later; Icelandic-specific, not useful for other languages.
- Low friction: pure-Python wheel with no runtime dependencies.
- Active maintenance—last release 35 days ago, repository updated 2026-07-10.
License · maintenance · safety
MIT (permissive) — MIT license (permissive): you can use, modify, and distribute this package with minimal restrictions, provided you include the license notice.
last release 2026-07-10 (35 days) · last repo commit 2026-07-10 · 32 stars
0 known vulnerabilities (OSV.dev, 2026-08-14) · 121,119 downloads/mo, #11,994 on PyPI
Alternatives
Verify before relying
pip install tokenizer
from tokenizer import split_into_sentences, tokenize
# Shallow tokenization (returns sentences as strings)
for sent in split_into_sentences("Halló heimur. Þetta er prófun."):
print(sent)
# Deep tokenization (returns token objects with type and value)
for token in tokenize("3. janúar 2026"):
print(token.kind, token.txt, token.val)- Whether the included Abbrev.conf dictionary covers domain-specific abbreviations beyond common Icelandic ones.
- Performance characteristics on non-ASCII or mixed-language input beyond the 1 MB Icelandic Gigaword Corpus benchmark.
What it is and what it does
Tokenizer is a pure-Python library and command-line tool for breaking Icelandic text into tokens (words, punctuation, numbers, dates, URLs, e-mails, measurements, and amounts) and segmenting them into sentences. It handles corner cases like abbreviations and dates mid-sentence. The package offers two modes: shallow tokenization, which returns space-separated tokens as strings, and deep tokenization, which returns annotated token objects with type information and extracted values (e.g., a date token yields a year/month/day tuple). It includes a built-in dictionary of common Icelandic abbreviations and runs on Python 3.10 or later, with both CPython and PyPy support.
The library is a spinoff from the Greynir natural language parser project and is designed as a preprocessing step for NLP tasks like word counting, parsing, spell checking, and corpus analysis. It can be used as a Python module via functions like `split_into_sentences()` and `tokenize()`, or invoked as a command-line tool with options for CSV/JSON output, punctuation normalization, and HTML escape handling.
Use it for
- Preprocess Icelandic text for downstream NLP pipelines such as parsing, spell checking, or statistical analysis.
- Extract structured token information (dates, amounts, measurements, URLs) from Icelandic documents in JSON or CSV format.
- Segment Icelandic text into sentences for corpus generation or document processing workflows.
- Normalize Icelandic punctuation and typography (quotes, ellipsis, dashes) in text processing applications.
- Build command-line text processing workflows using the tokenize tool to batch-process Icelandic documents.
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes.
The package is actively maintained, has zero known vulnerabilities, installs with no dependencies, and is production-stable. Install it if you need to tokenize or segment Icelandic text. It is not useful for other languages.
Install
tokenizer on PyPI
Before you install
Low friction: pure-Python wheel with no runtime dependencies. Active maintenance—last release 35 days ago, repository updated 2026-07-10. Supports Python 3.10 through 3.14 on CPython and PyPy.
Requires Python 3.10 or later; Icelandic-specific, not useful for other languages.
License in practice
MIT license (permissive): you can use, modify, and distribute this package with minimal restrictions, provided you include the license notice.
Quickstart
pip install tokenizer
from tokenizer import split_into_sentences, tokenize
# Shallow tokenization (returns sentences as strings)
for sent in split_into_sentences("Halló heimur. Þetta er prófun."):
print(sent)
# Deep tokenization (returns token objects with type and value)
for token in tokenize("3. janúar 2026"):
print(token.kind, token.txt, token.val)Verify before relying
- Whether the included Abbrev.conf dictionary covers domain-specific abbreviations beyond common Icelandic ones.
- Performance characteristics on non-ASCII or mixed-language input beyond the 1 MB Icelandic Gigaword Corpus benchmark.
Package facts
| License | MIT permissive |
| Python support | Supports the current Python release >=3.10 |
| Install friction | Low. Pure-Python wheel |
| Runtime dependencies | None |
| Maintenance | Actively maintained 35 days since the last release |
| Last repo commit | |
| First released | |
| Downloads | 121,119 / month, #11,994 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
| Classifiers | Development Status :: 5 - Production/StableIntended Audience :: DevelopersNatural Language :: IcelandicOperating System :: MacOSOperating System :: Microsoft :: WindowsOperating System :: POSIXOperating System :: UnixProgramming Language :: PythonProgramming Language :: Python :: 3Programming Language :: Python :: 3.10Programming Language :: Python :: 3.11Programming Language :: Python :: 3.12Programming Language :: Python :: 3.13Programming Language :: Python :: 3.14Programming Language :: Python :: Implementation :: CPythonProgramming Language :: Python :: Implementation :: PyPyTopic :: Software Development :: Libraries :: Python ModulesTopic :: Text Processing :: LinguisticTopic :: Utilities |
Evidence: tokenizer-3.6.4-py3-none-any.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “icelandic text tokenization”
- tokenizerTokenizes Icelandic text into words, punctuation, numbers, dates, and…
- blingfireBlingfire provides fast tokenization and text processing using finite…
- rjiebaA Python binding to the Rust-based jieba-rs Chinese text segmentation…
Give your agent the search over MCP, or paste the wish link into any chat.
More Python Modules packages
Converts domain names between Unicode and ASCII-compatible encoding (Punycode) according to IDNA 2008 and Unicode Technical Standard 46, with security validation and broader script coverage than the standard library.
Install it if you work with internationalized domain names, need to validate domains, or use HTTP clients that depend on it transitively.
Setuptools is a Python build backend and package management tool that handles building, distributing, and installing Python packages, including support for C/C++ extension modules.
PyYAML parses and emits YAML 1.1 data format, enabling serialization and deserialization of configuration files and Python objects to and from human-readable YAML text.
Pydantic validates Python data structures against type hints, coercing and checking input at runtime to ensure it matches a declared schema.
Provides reusable metadata objects for use with PEP-593 `typing.Annotated` to express common constraints like bounds, collection sizes, and predicates on types.
Install it if you use or build libraries that need to express type constraints in a standardized, inspectable way—or if you want to annotate your own types with…
Provides runtime tools to inspect and introspect Python type annotations, enabling programmatic examination of type hints at execution time.
See also sacremoses · segtok · tensorflow-text · segments · razdel · jieba3k · polyglot · indic-nlp-library · blingfire · tokenizers