{"categories":[{"label":"Python Modules","url":"https://skillfed.io/packages/category/software-development-libraries-python-modules/19"},{"label":"Utilities","url":"https://skillfed.io/packages/category/utilities/9"},{"label":"Linguistic","url":"https://skillfed.io/packages/category/text-processing-linguistic/2"}],"enrichment":{"capability":"Tokenizes Icelandic text into words, punctuation, numbers, dates, and other token types, with optional sentence segmentation and deep annotation of token properties.","skillfed_tags":["icelandic-nlp","sentence-segmentation"],"use_cases":["Preprocess Icelandic text for downstream NLP pipelines such as parsing, spell checking, or statistical analysis.","Extract structured token information (dates, amounts, measurements, URLs) from Icelandic documents in JSON or CSV format.","Segment Icelandic text into sentences for corpus generation or document processing workflows.","Normalize Icelandic punctuation and typography (quotes, ellipsis, dashes) in text processing applications.","Build command-line text processing workflows using the tokenize tool to batch-process Icelandic documents."],"what_it_does":"Tokenizer is a pure-Python library and command-line tool for breaking Icelandic text into tokens (words, punctuation, numbers, dates, URLs, e-mails, measurements, and amounts) and segmenting them into sentences. It handles corner cases like abbreviations and dates mid-sentence. The package offers two modes: shallow tokenization, which returns space-separated tokens as strings, and deep tokenization, which returns annotated token objects with type information and extracted values (e.g., a date token yields a year/month/day tuple). It includes a built-in dictionary of common Icelandic abbreviations and runs on Python 3.10 or later, with both CPython and PyPy support.\n\nThe library is a spinoff from the Greynir natural language parser project and is designed as a preprocessing step for NLP tasks like word counting, parsing, spell checking, and corpus analysis. It can be used as a Python module via functions like `split_into_sentences()` and `tokenize()`, or invoked as a command-line tool with options for CSV/JSON output, punctuation normalization, and HTML escape handling.","worth_installing":"Yes. The package is actively maintained, has zero known vulnerabilities, installs with no dependencies, and is production-stable. Install it if you need to tokenize or segment Icelandic text. It is not useful for other languages."},"id":"tokenizer","links":{"html":"https://skillfed.io/packages/tokenizer","md":"https://skillfed.io/packages/tokenizer.md","pypi":"https://pypi.org/project/tokenizer/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2026-07-10","license_spdx":"MIT","license_treatment":"permissive","name":"tokenizer","python_support":"supports_current","summary":"A fast, compact pure-Python tokenizer for Icelandic text with sentence segmentation"},"popularity":{"monthly_downloads":121119,"position":11994,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"3.6.4"}