{"categories":[{"label":"Artificial Intelligence","url":"https://skillfed.io/packages/category/scientific-engineering-artificial-intelligence/5"}],"enrichment":{"capability":"Computes cross-entropy loss for large-vocabulary language models with minimal memory footprint by materializing logits selectively rather than globally, using custom kernels and torch.compile fallbacks.","skillfed_tags":["memory-optimization","language-models","gpu-training"],"use_cases":["Training large language models on GPUs with limited memory by reducing classifier head memory footprint from tens of gigabytes to under a gigabyte.","Fine-tuning transformer models (Llama, Phi3, Mistral, Gemma2) using the transformers library without modifying model code via cce_patch.","Computing per-token loss and perplexity efficiently with reduction='none' for detailed loss analysis without materializing full logit matrices.","Enabling training on older GPUs or non-Linux systems via torch.compile fallback when Triton kernels are unavailable.","Reducing overall training-time memory consumption of the classifier head to fit larger models or batch sizes on fixed hardware."],"what_it_does":"Cut Cross-Entropy (CCE) is a memory-efficient implementation of the cross-entropy loss computation for training large-vocabulary language models. During standard training, the cross-entropy layer materializes a full logit matrix covering all token-vocabulary pairs, consuming enormous amounts of GPU memory\u2014often more than the rest of the model combined. CCE avoids this by computing logits on-the-fly in flash memory, computing only the logit for the correct token and evaluating log-sum-exp over all vocabulary items without materializing the full matrix.\n\nThe package provides a drop-in replacement function `linear_cross_entropy` that accepts embeddings, classifier weights, and labels, with optional support for token-shifting in causal language modeling. It includes custom Triton kernels for Ampere and newer GPUs, a torch.compile fallback for older hardware and non-Linux platforms, and direct integration patches for transformers models (Llama, Phi3, Mistral, Gemma2 families). According to the documentation, this reduces memory consumption from 24 GB to 1 MB for the loss computation on Gemma 2 (2B), with negligible impact on training speed.","worth_installing":"Yes, with conditions. Install if you train large-vocabulary language models on memory-constrained GPUs and need dramatic memory reduction in the loss computation layer. The package is well-motivated by published research and offers both optimized Triton kernels and fallback implementations. However, verify the license terms before production use (license treatment is unclear), and be aware that maintenance is dormant\u2014no active development is expected. Requires Python 3.10+, PyTorch 2.4+, and Triton 3.0+ on Ampere+ GPUs."},"id":"cut-cross-entropy","links":{"html":"https://skillfed.io/packages/cut-cross-entropy","md":"https://skillfed.io/packages/cut-cross-entropy.md","pypi":"https://pypi.org/project/cut-cross-entropy/"},"maintenance":{"status":"dormant"},"meta":{"latest_release":"2025-01-07","license_spdx":null,"license_treatment":"unclear","name":"cut-cross-entropy","python_support":"supports_current","summary":"Code for cut cross entropy, a memory efficient implementation of linear-cross-entropy loss."},"popularity":{"monthly_downloads":609745,"position":5770,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"25.1.1"}