spacy-alignments

What it is and what it does

spacy-alignments wraps Yohei Tamura's Rust tokenizations library to solve a core problem in modern NLP: when you use spaCy for linguistic processing and transformer models for semantic understanding, their tokenizers often split text differently, breaking the correspondence between spaCy tokens and transformer subword tokens. This package provides bidirectional alignment mappings—given two tokenization schemes, it computes which tokens in scheme A correspond to which in scheme B and vice versa.

It is designed as a drop-in replacement for the older pytokenizations package, using setuptools-rust to simplify cross-platform binary distribution. The package has no runtime dependencies and compiles to native code, making it fast. It supports Python 3.9–3.12 on macOS, Linux, and Windows, with prebuilt wheels for most platforms; if your platform lacks a wheel, you will need Rust installed to compile from source.

Use it for:

• Align spaCy token spans with BERT or other transformer subword tokens for joint NLP + deep learning pipelines.
• Convert linguistic annotations (POS tags, NER labels) from spaCy tokens to transformer token positions for fine-tuning.
• Map attention weights or hidden states from transformers back to spaCy linguistic units for interpretability.
• Integrate spaCy's rule-based NLP with transformer-based models without losing token-level correspondence.
• Handle multilingual text where spaCy and transformer tokenizers diverge significantly.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Aligns tokenizations between spaCy and transformer models by mapping token boundaries between different tokenization schemes, enabling seamless integration of spaCy NLP pipelines with transformer-based models.

Yes, if you are building a pipeline that combines spaCy and transformer models. The package solves a real integration problem with no runtime dependencies and active maintenance. Medium install friction (potential Rust compilation) is a minor drawback but manageable; prebuilt wheels cover most platforms. No known security issues and permissive MIT license add confidence.

Install

spacy-alignments on PyPI

pip install spacy-alignments

uv add spacy-alignments

poetry add spacy-alignments

Installing spacy-alignments

Before you install

Medium install friction due to Rust compilation requirement when binary wheels are unavailable for your platform. The package maintains active status with recent releases and supports Python 3.9–3.12 across major operating systems, though you may need to install Rust toolchain if a prebuilt wheel is not available.

License in practice

MIT license permits unrestricted use, modification, and distribution with minimal restrictions, making it suitable for both commercial and open-source projects.

Quickstart

pip install spacy-alignments

import spacy_alignments as tokenizations
a2b, b2a = tokenizations.get_alignments(["å", "BC"], ["abc"])
print(a2b, b2a)  # [[0], [0]], [[0, 1]]

If no binary wheel exists for your platform, you must install Rust to build from source.

Verify before relying

• Whether this package is actively maintained by Explosion AI or community-driven given the 437 days since last release.
• Specific transformer model compatibility and tested integration scenarios beyond the general spaCy + transformers use case.
• Performance characteristics and alignment accuracy guarantees for edge cases in multilingual or specialized tokenization schemes.

Package facts

Evidence: spacy_alignments-0.9.2-cp310-cp310-macosx_10_12_x86_64.whl; spacy_alignments-0.9.2-cp310-cp310-macosx_11_0_arm64.whl; spacy_alignments-0.9.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl; spacy_alignments-0.9.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; spacy_alignments-0.9.2-cp310-cp310-musllinux_1_2_aarch64.whl; spacy_alignments-0.9.2-cp310-cp310-musllinux_1_2_x86_64.whl; spacy_alignments-0.9.2-cp310-cp310-win_amd64.whl; spacy_alignments-0.9.2-cp311-cp311-macosx_10_12_x86_64.whl; spacy_alignments-0.9.2-cp311-cp311-macosx_11_0_arm64.whl; spacy_alignments-0.9.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl; spacy_alignments-0.9.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; spacy_alignments-0.9.2-cp311-cp311-musllinux_1_2_aarch64.whl; spacy_alignments-0.9.2-cp311-cp311-musllinux_1_2_x86_64.whl; spacy_alignments-0.9.2-cp311-cp311-win_amd64.whl; spacy_alignments-0.9.2-cp312-cp312-macosx_10_13_x86_64.whl; spacy_alignments-0.9.2-cp312-cp312-macosx_11_0_arm64.whl; spacy_alignments-0.9.2-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl; spacy_alignments-0.9.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; spacy_alignments-0.9.2-cp312-cp312-musllinux_1_2_aarch64.whl; spacy_alignments-0.9.2-cp312-cp312-musllinux_1_2_x86_64.whl

Tags

More Scientific/Engineering packages