{"categories":[{"label":"Scientific/Engineering","url":"https://skillfed.io/packages/category/scientific-engineering/6"},{"label":"Artificial Intelligence","url":"https://skillfed.io/packages/category/scientific-engineering-artificial-intelligence/7"}],"enrichment":{"capability":"Aligns tokenizations between spaCy and transformer models by mapping token boundaries between different tokenization schemes, enabling seamless integration of spaCy NLP pipelines with transformer-based models.","skillfed_tags":["nlp-integration","tokenization","transformers"],"use_cases":["Align spaCy token spans with BERT or other transformer subword tokens for joint NLP + deep learning pipelines.","Convert linguistic annotations (POS tags, NER labels) from spaCy tokens to transformer token positions for fine-tuning.","Map attention weights or hidden states from transformers back to spaCy linguistic units for interpretability.","Integrate spaCy's rule-based NLP with transformer-based models without losing token-level correspondence.","Handle multilingual text where spaCy and transformer tokenizers diverge significantly."],"what_it_does":"spacy-alignments wraps Yohei Tamura's Rust tokenizations library to solve a core problem in modern NLP: when you use spaCy for linguistic processing and transformer models for semantic understanding, their tokenizers often split text differently, breaking the correspondence between spaCy tokens and transformer subword tokens. This package provides bidirectional alignment mappings\u2014given two tokenization schemes, it computes which tokens in scheme A correspond to which in scheme B and vice versa.\n\nIt is designed as a drop-in replacement for the older pytokenizations package, using setuptools-rust to simplify cross-platform binary distribution. The package has no runtime dependencies and compiles to native code, making it fast. It supports Python 3.9\u20133.12 on macOS, Linux, and Windows, with prebuilt wheels for most platforms; if your platform lacks a wheel, you will need Rust installed to compile from source.","worth_installing":"Yes, if you are building a pipeline that combines spaCy and transformer models. The package solves a real integration problem with no runtime dependencies and active maintenance. Medium install friction (potential Rust compilation) is a minor drawback but manageable; prebuilt wheels cover most platforms. No known security issues and permissive MIT license add confidence."},"id":"spacy-alignments","links":{"html":"https://skillfed.io/packages/spacy-alignments","md":"https://skillfed.io/packages/spacy-alignments.md","pypi":"https://pypi.org/project/spacy-alignments/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2025-06-03","license_spdx":null,"license_treatment":"permissive","name":"spacy-alignments","python_support":"supports_current","summary":"A spaCy package for the Rust tokenizations library"},"popularity":{"monthly_downloads":213567,"position":9434,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"0.9.2"}