{"categories":[{"label":"Artificial Intelligence","url":"https://skillfed.io/packages/category/scientific-engineering-artificial-intelligence/3"}],"enrichment":{"capability":"Provides optimized MLA (Multi-head Latent Attention) kernels for NVIDIA Blackwell GPUs, including prefill and decode operations with FP8 quantization and ragged sequence support.","skillfed_tags":["gpu-kernels","attention-optimization","llm-serving"],"use_cases":["Accelerate token-by-token generation in LLM serving with strict latency budgets, especially for coding agents.","Batch prefill processing of variable-length prompts without padding overhead using ragged attention.","Reduce memory footprint and improve throughput by using FP8 quantization in both prefill and decode phases.","Optimize KV cache utilization in paged attention scenarios with automatic workspace sizing and compile caching.","Improve tile utilization in small-head decode scenarios by folding query tokens into the head dimension."],"what_it_does":"TokenSpeed-MLA provides hand-optimized CUDA kernels for Multi-head Latent Attention on NVIDIA Blackwell GPUs. It splits into two main paths: prefill (processing initial prompt tokens) and decode (generating one token at a time). The prefill path offers both a JIT-compiled CuTe DSL backend and an optional pre-compiled binary backend with NVIDIA-internal optimizations; it handles ragged variable-length sequences without padding and supports FP8 E4M3 quantization. The decode path uses a two-kernel split-KV strategy with runtime auto-sizing and compile caching, supporting FP16, BF16, and FP8 inputs while writing BF16 output for downstream stability.\n\nThe package targets latency-sensitive serving workloads\u2014particularly coding agents and similar use cases with high request concurrency, short decode steps, and strict time-to-first-token requirements. It includes a fused Triton kernel for K/V packing and FP8 quantization that replaces separate cat and cast operations. Key optimizations include query-token folding for small head counts, reduced shared memory usage via 2CTA UTCMMA instructions, and multi-stage epilogue writes. The package is actively maintained and has no known security vulnerabilities.","worth_installing":"Yes, if you are serving models on NVIDIA Blackwell GPUs and need low-latency token generation. The package is actively maintained, has no vulnerabilities, and offers measurable performance gains over TensorRT-LLM's native MLA implementation\u2014particularly the AOT binary backend. Install friction is moderate due to compiled dependencies; ensure torch, apache-tvm-ffi, nvidia-cutlass-dsl, and tokenspeed-triton are available in your environment."},"id":"tokenspeed-mla","links":{"html":"https://skillfed.io/packages/tokenspeed-mla","md":"https://skillfed.io/packages/tokenspeed-mla.md","pypi":"https://pypi.org/project/tokenspeed-mla/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2026-08-09","license_spdx":null,"license_treatment":"permissive","name":"tokenspeed-mla","python_support":"supports_current","summary":"Speed-of-light TokenSpeed MLA kernels for Blackwell SM100 and SM103."},"popularity":{"monthly_downloads":1972060,"position":3395,"tier":"top_5000"},"security":{"n_vulnerabilities":0},"version":"0.2.5"}