spacy-alignments
A spaCy package for the Rust tokenizations library
Decision gist · record as of 2026-08-14
Yes, if you are building a pipeline that combines spaCy and transformer models. The package solves a real integration problem with no runtime dependencies and active maintenance. Medium install friction (potential Rust compilation) is a minor drawback but manageable; prebuilt wheels cover most platforms. No known security issues and permissive MIT license add confidence.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- If no binary wheel exists for your platform, you must install Rust to build from source.
- Medium install friction due to Rust compilation requirement when binary wheels are unavailable for your platform.
- The package maintains active status with recent releases and supports Python 3.9–3.12 across major operating systems, though you may need to install Rust toolchain if a prebuilt wheel is not available.
License · maintenance · safety
MIT (permissive) — MIT license permits unrestricted use, modification, and distribution with minimal restrictions, making it suitable for both commercial and open-source projects.
last release 2025-06-03 (437 days) · last repo commit 2026-03-27 · 35 stars
0 known vulnerabilities (OSV.dev, 2026-08-14) · 213,567 downloads/mo, #9,434 on PyPI
Alternatives
Verify before relying
pip install spacy-alignments
import spacy_alignments as tokenizations
a2b, b2a = tokenizations.get_alignments(["å", "BC"], ["abc"])
print(a2b, b2a) # [[0], [0]], [[0, 1]]- Whether this package is actively maintained by Explosion AI or community-driven given the 437 days since last release.
- Specific transformer model compatibility and tested integration scenarios beyond the general spaCy + transformers use case.
- Performance characteristics and alignment accuracy guarantees for edge cases in multilingual or specialized tokenization schemes.
What it is and what it does
spacy-alignments wraps Yohei Tamura's Rust tokenizations library to solve a core problem in modern NLP: when you use spaCy for linguistic processing and transformer models for semantic understanding, their tokenizers often split text differently, breaking the correspondence between spaCy tokens and transformer subword tokens. This package provides bidirectional alignment mappings—given two tokenization schemes, it computes which tokens in scheme A correspond to which in scheme B and vice versa.
It is designed as a drop-in replacement for the older pytokenizations package, using setuptools-rust to simplify cross-platform binary distribution. The package has no runtime dependencies and compiles to native code, making it fast. It supports Python 3.9–3.12 on macOS, Linux, and Windows, with prebuilt wheels for most platforms; if your platform lacks a wheel, you will need Rust installed to compile from source.
Use it for
- Align spaCy token spans with BERT or other transformer subword tokens for joint NLP + deep learning pipelines.
- Convert linguistic annotations (POS tags, NER labels) from spaCy tokens to transformer token positions for fine-tuning.
- Map attention weights or hidden states from transformers back to spaCy linguistic units for interpretability.
- Integrate spaCy's rule-based NLP with transformer-based models without losing token-level correspondence.
- Handle multilingual text where spaCy and transformer tokenizers diverge significantly.
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
Yes, if you are building a pipeline that combines spaCy and transformer models.
The package solves a real integration problem with no runtime dependencies and active maintenance. Medium install friction (potential Rust compilation) is a minor drawback but manageable; prebuilt wheels cover most platforms. No known security issues and permissive MIT license add confidence.
Install
spacy-alignments on PyPI
Before you install
Medium install friction due to Rust compilation requirement when binary wheels are unavailable for your platform. The package maintains active status with recent releases and supports Python 3.9–3.12 across major operating systems, though you may need to install Rust toolchain if a prebuilt wheel is not available.
If no binary wheel exists for your platform, you must install Rust to build from source.
License in practice
MIT license permits unrestricted use, modification, and distribution with minimal restrictions, making it suitable for both commercial and open-source projects.
Quickstart
pip install spacy-alignments
import spacy_alignments as tokenizations
a2b, b2a = tokenizations.get_alignments(["å", "BC"], ["abc"])
print(a2b, b2a) # [[0], [0]], [[0, 1]]Verify before relying
- Whether this package is actively maintained by Explosion AI or community-driven given the 437 days since last release.
- Specific transformer model compatibility and tested integration scenarios beyond the general spaCy + transformers use case.
- Performance characteristics and alignment accuracy guarantees for edge cases in multilingual or specialized tokenization schemes.
Package facts
| License | MIT permissive |
| Python support | Supports the current Python release <3.14,>=3.9 |
| Install friction | Medium. Platform-specific wheel |
| Runtime dependencies | None |
| Maintenance | Actively maintained 437 days since the last release |
| Last repo commit | |
| First released | |
| Downloads | 213,567 / month, #9,434 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
| Classifiers | Development Status :: 4 - BetaEnvironment :: ConsoleIntended Audience :: DevelopersIntended Audience :: Science/ResearchLicense :: OSI Approved :: MIT LicenseOperating System :: MacOS :: MacOS XOperating System :: Microsoft :: WindowsOperating System :: POSIX :: LinuxProgramming Language :: PythonProgramming Language :: Python :: 3Programming Language :: Python :: 3.10Programming Language :: Python :: 3.11Programming Language :: Python :: 3.12Programming Language :: Python :: 3.9Programming Language :: RustTopic :: Scientific/EngineeringTopic :: Scientific/Engineering :: Artificial Intelligence |
Evidence: spacy_alignments-0.9.2-cp310-cp310-macosx_10_12_x86_64.whl; spacy_alignments-0.9.2-cp310-cp310-macosx_11_0_arm64.whl; spacy_alignments-0.9.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl; spacy_alignments-0.9.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; spacy_alignments-0.9.2-cp310-cp310-musllinux_1_2_aarch64.whl; spacy_alignments-0.9.2-cp310-cp310-musllinux_1_2_x86_64.whl; spacy_alignments-0.9.2-cp310-cp310-win_amd64.whl; spacy_alignments-0.9.2-cp311-cp311-macosx_10_12_x86_64.whl; spacy_alignments-0.9.2-cp311-cp311-macosx_11_0_arm64.whl; spacy_alignments-0.9.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl; spacy_alignments-0.9.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; spacy_alignments-0.9.2-cp311-cp311-musllinux_1_2_aarch64.whl; spacy_alignments-0.9.2-cp311-cp311-musllinux_1_2_x86_64.whl; spacy_alignments-0.9.2-cp311-cp311-win_amd64.whl; spacy_alignments-0.9.2-cp312-cp312-macosx_10_13_x86_64.whl; spacy_alignments-0.9.2-cp312-cp312-macosx_11_0_arm64.whl; spacy_alignments-0.9.2-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl; spacy_alignments-0.9.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl; spacy_alignments-0.9.2-cp312-cp312-musllinux_1_2_aarch64.whl; spacy_alignments-0.9.2-cp312-cp312-musllinux_1_2_x86_64.whl
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “tokenization alignment spacy transformers”
- spacy-alignmentsAligns tokenizations between spaCy and transformer models by mapping…
- spacy-transformersIntegrates pretrained transformer models (BERT, RoBERTa, XLNet,…
- spacyspaCy is an industrial-strength NLP library providing tokenization,…
Give your agent the search over MCP, or paste the wish link into any chat.
More Scientific/Engineering packages
NumPy provides an N-dimensional array object and a comprehensive suite of mathematical, linear algebra, Fourier transform, and random number functions for scientific computing in Python.
pandas provides fast, flexible data structures (Series and DataFrame) for loading, cleaning, transforming, and analyzing labeled or relational data in Python.
scipy provides numerical algorithms for mathematics, science, and engineering—including optimization, integration, linear algebra, Fourier transforms, signal and image processing, and ODE solvers—built on numpy arrays.
scikit-learn provides a comprehensive Python library for supervised and unsupervised machine learning, including classification, regression, clustering, dimensionality reduction, and model evaluation tools built on NumPy and SciPy.
Install it if you need to train, evaluate, or deploy supervised or unsupervised learning models.
dill extends Python's pickle module to serialize and deserialize a much wider range of Python objects, including functions, lambdas, classes, and interpreter sessions, to byte streams for storage or network transmission.
Multiprocess is an enhanced fork of Python's standard multiprocessing library that uses dill for better serialization, allowing you to spawn processes with a threading-like API and share complex objects between them.
Install it if you use multiprocessing and encounter pickle serialization limits with lambdas or complex objects.
See also spacy-transformers · spacy-curated-transformers · spacy · tokenizers · date-spacy · tokie · curated-tokenizers · ja-ginza · textacy · spacy-language-detection