tokenizer

What it is and what it does

Tokenizer is a pure-Python library and command-line tool for breaking Icelandic text into tokens (words, punctuation, numbers, dates, URLs, e-mails, measurements, and amounts) and segmenting them into sentences. It handles corner cases like abbreviations and dates mid-sentence. The package offers two modes: shallow tokenization, which returns space-separated tokens as strings, and deep tokenization, which returns annotated token objects with type information and extracted values (e.g., a date token yields a year/month/day tuple). It includes a built-in dictionary of common Icelandic abbreviations and runs on Python 3.10 or later, with both CPython and PyPy support.

The library is a spinoff from the Greynir natural language parser project and is designed as a preprocessing step for NLP tasks like word counting, parsing, spell checking, and corpus analysis. It can be used as a Python module via functions like `split_into_sentences()` and `tokenize()`, or invoked as a command-line tool with options for CSV/JSON output, punctuation normalization, and HTML escape handling.

Use it for:

• Preprocess Icelandic text for downstream NLP pipelines such as parsing, spell checking, or statistical analysis.
• Extract structured token information (dates, amounts, measurements, URLs) from Icelandic documents in JSON or CSV format.
• Segment Icelandic text into sentences for corpus generation or document processing workflows.
• Normalize Icelandic punctuation and typography (quotes, ellipsis, dashes) in text processing applications.
• Build command-line text processing workflows using the tokenize tool to batch-process Icelandic documents.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Tokenizes Icelandic text into words, punctuation, numbers, dates, and other token types, with optional sentence segmentation and deep annotation of token properties.

Yes. The package is actively maintained, has zero known vulnerabilities, installs with no dependencies, and is production-stable. Install it if you need to tokenize or segment Icelandic text. It is not useful for other languages.

Install

tokenizer on PyPI

pip install tokenizer

uv add tokenizer

poetry add tokenizer

Installing tokenizer

Before you install

Low friction: pure-Python wheel with no runtime dependencies. Active maintenance—last release 35 days ago, repository updated 2026-07-10. Supports Python 3.10 through 3.14 on CPython and PyPy.

License in practice

MIT license (permissive): you can use, modify, and distribute this package with minimal restrictions, provided you include the license notice.

Quickstart

pip install tokenizer

from tokenizer import split_into_sentences, tokenize

# Shallow tokenization (returns sentences as strings)
for sent in split_into_sentences("Halló heimur. Þetta er prófun."):
    print(sent)

# Deep tokenization (returns token objects with type and value)
for token in tokenize("3. janúar 2026"):
    print(token.kind, token.txt, token.val)

Requires Python 3.10 or later; Icelandic-specific, not useful for other languages.

Verify before relying

• Whether the included Abbrev.conf dictionary covers domain-specific abbreviations beyond common Icelandic ones.
• Performance characteristics on non-ASCII or mixed-language input beyond the 1 MB Icelandic Gigaword Corpus benchmark.

Package facts

Evidence: tokenizer-3.6.4-py3-none-any.whl

Keywords: tokenizer, tokenization, icelandic, nlp, natural-language-processing, text-processing, sentence-segmentation, parsing

Tags

More Python Modules packages