{"categories":[{"label":"Artificial Intelligence","url":"https://skillfed.io/packages/category/scientific-engineering-artificial-intelligence/7"}],"enrichment":{"capability":"Reduces transformer model vocabulary to only tokens used during training, cutting embedding memory and compute overhead while preserving the full model after training completes.","skillfed_tags":["transformer-optimization","memory-efficiency","nlp-training"],"use_cases":["Fine-tuning large multilingual models on domain-specific datasets where most of the pretrained vocabulary is unused.","Reducing GPU memory during training of BERT or RoBERTa models on small, specialized text corpora with limited token diversity.","Optimizing embedding layer memory in resource-constrained environments where vocabulary reduction yields measurable savings.","Training on low-resource languages or specialized domains where the full pretrained vocabulary is largely redundant."],"what_it_does":"This package addresses a common inefficiency in transformer training: most tokens in a model's vocabulary are never used on a given dataset, yet their embeddings still consume GPU memory and compute resources during gradient updates. The package provides a context manager that temporarily shrinks the vocabulary to only tokens present in your training data, then restores the full model afterward so you can save the complete version.\n\nIt works by gathering vocabulary usage statistics, reducing embeddings and their gradients to only active tokens, and fitting them back after training. The impact varies by model and dataset; some configurations show substantial vocabulary reductions with corresponding memory savings, though the gradient computation overhead from the full transformer layers typically dominates the overall training time. Support is best with HuggingFace FastTokenizers; slow tokenizers require custom implementations, with only XLMRobertaTokenizer, RobertaTokenizer, and BertTokenizer currently supported.","worth_installing":"Yes, if you are fine-tuning a transformer on a dataset with limited vocabulary diversity and have GPU memory constraints. The savings are real but dataset-dependent. Install friction is low and the codebase is stable. The aging maintenance status (last commit 2025-06-15) is a minor concern but not a blocker for a focused, mature utility."},"id":"transformer-smaller-training-vocab","links":{"html":"https://skillfed.io/packages/transformer-smaller-training-vocab","md":"https://skillfed.io/packages/transformer-smaller-training-vocab.md","pypi":"https://pypi.org/project/transformer-smaller-training-vocab/"},"maintenance":{"status":"aging"},"meta":{"latest_release":"2025-06-15","license_spdx":null,"license_treatment":"permissive","name":"transformer-smaller-training-vocab","python_support":"supports_current","summary":"Temporary remove unused tokens during training to save ram and speed."},"popularity":{"monthly_downloads":228760,"position":9145,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"0.4.2"}