spacy-pkuseg

What it is and what it does

spacy-pkuseg is a spaCy-integrated fork of the pkuseg Chinese word segmentation toolkit. It provides domain-aware tokenization for Chinese text across five pretrained models: a mixed-domain default, plus specialized models for news, web, medicine, and tourism text. The package wraps the underlying segmentation engine (unmodified from the original pkuseg) and simplifies both installation and model serialization for spaCy workflows.

The toolkit supports optional part-of-speech tagging alongside segmentation, batch processing of files with multiprocessing, and user-defined custom dictionaries. It depends on numpy and srsly for numerical and serialization operations. Installation is straightforward via pip with precompiled wheels for modern Python versions and common platforms, though it carries medium friction due to compiled components.

Use it for:

• Segment Chinese news articles or web text where domain-specific accuracy matters more than generic tokenization.
• Build spaCy NLP pipelines for Chinese that need both word boundaries and part-of-speech labels in one step.
• Process medical or tourism domain Chinese documents with models trained on domain-specific corpora.
• Batch-process large Chinese text files with multiprocessing to split into words and optionally tag parts of speech.
• Extend spaCy's Chinese support with a higher-accuracy alternative to generic tokenizers when domain context is known.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Chinese word segmentation for spaCy with domain-specific models (news, web, medicine, tourism) and optional part-of-speech tagging.

Yes, if you need Chinese word segmentation in a spaCy pipeline or want domain-specific accuracy. The package is actively maintained, has no known vulnerabilities, supports modern Python versions, and offers pretrained models for multiple domains. Install friction is moderate but manageable. Not necessary if you only need generic Chinese tokenization or are not using spaCy.

Install

spacy-pkuseg on PyPI

pip install spacy-pkuseg

uv add spacy-pkuseg

poetry add spacy-pkuseg

Installing spacy-pkuseg

Before you install

Medium install friction due to compiled wheels, but well-supported across Python 3.9–3.13 and major platforms (Linux, macOS, Windows). Active maintenance with recent commits and stable release history.

License in practice

MIT license permits commercial and private use with minimal restrictions; no copyleft obligations.

Quickstart

pip install spacy-pkuseg
import spacy_pkuseg
seg = spacy_pkuseg.pkuseg()
text = seg.cut('我爱北京天安门')
print(text)

Requires Python 3.9 or later; precompiled wheels available for Linux, macOS, and Windows x86_64/ARM64.

Verify before relying

• Whether spaCy itself must be installed separately or is pulled in as a transitive dependency.
• Performance benchmarks comparing this fork to the original pkuseg-python on the same hardware.
• Whether domain models (medicine, tourism, etc.) auto-download or require manual setup when used via spaCy.

Package facts

Evidence: spacy_pkuseg-1.0.1-cp310-cp310-macosx_10_9_x86_64.whl; spacy_pkuseg-1.0.1-cp310-cp310-macosx_11_0_arm64.whl; spacy_pkuseg-1.0.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl; spacy_pkuseg-1.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; spacy_pkuseg-1.0.1-cp310-cp310-musllinux_1_2_aarch64.whl; spacy_pkuseg-1.0.1-cp310-cp310-musllinux_1_2_x86_64.whl; spacy_pkuseg-1.0.1-cp310-cp310-win_amd64.whl; spacy_pkuseg-1.0.1-cp311-cp311-macosx_10_9_x86_64.whl; spacy_pkuseg-1.0.1-cp311-cp311-macosx_11_0_arm64.whl; spacy_pkuseg-1.0.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl; spacy_pkuseg-1.0.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; spacy_pkuseg-1.0.1-cp311-cp311-musllinux_1_2_aarch64.whl; spacy_pkuseg-1.0.1-cp311-cp311-musllinux_1_2_x86_64.whl; spacy_pkuseg-1.0.1-cp311-cp311-win_amd64.whl; spacy_pkuseg-1.0.1-cp312-cp312-macosx_10_13_x86_64.whl; spacy_pkuseg-1.0.1-cp312-cp312-macosx_11_0_arm64.whl; spacy_pkuseg-1.0.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl; spacy_pkuseg-1.0.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; spacy_pkuseg-1.0.1-cp312-cp312-musllinux_1_2_aarch64.whl; spacy_pkuseg-1.0.1-cp312-cp312-musllinux_1_2_x86_64.whl

Tags

More Scientific/Engineering packages