soynlp

What it is and what it does

soynlp is an unsupervised Korean NLP toolkit designed to extract linguistic structure from text without pre-trained models or labeled training data. It uses statistical patterns—cohesion scores, branching entropy, and accessor variety—to identify word boundaries, extract nouns, and tokenize sentences. The package includes WordExtractor for discovering new words in a corpus, multiple noun extractors (LRNounExtractor versions 1 and 2, NewsNounExtractor), and tokenizers that split sentences into word sequences based on learned word scores.

The toolkit works by analyzing how characters and substrings co-occur and branch within a corpus, learning left-right (L-R) structure patterns typical of Korean morphology. It is intended for processing document collections from a single domain or genre (news articles, movie reviews, social media) where homogeneous vocabulary and writing style yield better statistical signals. Dependencies are standard scientific Python libraries: numpy, scipy, scikit-learn, and psutil for memory monitoring during training.

Use it for:

• Extract new or domain-specific Korean words from a corpus of news articles or social media posts without a pre-trained dictionary.
• Tokenize Korean sentences into word sequences using learned word boundaries from a corpus of similar documents.
• Identify nouns in Korean text by analyzing right-side character patterns (e.g., particles following noun candidates).
• Build a custom Korean tokenizer by combining noun scores with word cohesion scores for domain-specific text.
• Analyze L-R graph structure of Korean words to understand morphological patterns in a specific corpus.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Unsupervised Korean natural language processing toolkit that extracts words and nouns from text, tokenizes sentences, and performs part-of-speech analysis without requiring training data.

Yes, if you are processing Korean text and need unsupervised word/noun extraction or tokenization without external models. The low install friction and active maintenance make it accessible. However, GPLv3 licensing restricts proprietary use, and the package requires a reasonably large, homogeneous corpus to work well—single documents or mixed-domain text will produce poor results. Not suitable for production systems requiring copyleft-incompatible licensing.

Install

soynlp on PyPI

pip install soynlp

uv add soynlp

poetry add soynlp

Installing soynlp

Before you install

Low friction install with four well-established scientific dependencies (numpy, scipy, scikit-learn, psutil). Package is actively maintained with recent commits and no known vulnerabilities.

License in practice

Licensed under GPLv3 (copyleft). Any derivative work or distribution must also be open-source under GPLv3; proprietary use is not permitted.

Quickstart

from soynlp.noun import LRNounExtractor_v2
from soynlp.word import WordExtractor

noun_extractor = LRNounExtractor_v2()
nouns = noun_extractor.train_extract(sentences)

word_extractor = WordExtractor(min_frequency=100)
word_extractor.train(sentences)
words = word_extractor.extract()

Requires a corpus of Korean text documents (homogeneous document collection works best); single sentences or heterogeneous document sets produce poor results.

Verify before relying

• Whether the package is actively maintained beyond the 2019-08-25 latest release date despite recent repository commits.
• Compatibility with Python versions beyond 3.6 given the classifier listing only 3.6 support.

Package facts

Evidence: soynlp-0.0.493-py3-none-any.whl

Keywords: korean-nlp, korean-text-processing, nlp, tokenizer, postagging, word-extraction

Tags

More Linguistic packages