transformer-engine

What it is and what it does

Transformer Engine is a library that accelerates Transformer model training and inference on NVIDIA GPUs by providing optimized low-precision computation. It supports FP8 on Hopper, Ada, and Ampere GPUs, and extends to MXFP8 and NVFP4 formats on Blackwell GPUs. The library automatically manages scaling factors and precision conversion, allowing users to enable low-precision training through a simple autocast API.

The package provides Python modules for building Transformer layers with built-in FP8 support and fused kernels for common operations. It integrates with popular frameworks and is designed to work with advanced training techniques like tensor parallelism, sequence parallelism, and mixture-of-experts architectures. Installation requires a compatible NVIDIA GPU, CUDA 12.1 or later, cuDNN 9.3 or later, and a modern C++ compiler with C++17 support.

Use it for:

• Train large language models with reduced memory footprint using low-precision formats on Hopper or Blackwell GPUs.
• Optimize inference latency for deployed Transformer models by leveraging low-precision computation and fused kernels.
• Build mixture-of-experts or other advanced Transformer architectures with automatic mixed-precision support.
• Integrate low-precision Transformer operations into custom deep learning frameworks via the C++ API.
• Experiment with NVFP4 or MXFP8 formats on Blackwell hardware for training efficiency.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Transformer Engine accelerates Transformer model training and inference on NVIDIA GPUs using low-precision formats including FP8, MXFP8, and NVFP4 to reduce memory use and improve performance.

Yes, if you have access to a compatible NVIDIA GPU (Hopper, Ada, Ampere, or Blackwell) and are training or deploying Transformer models where memory and compute efficiency matter. Low install friction, active maintenance, and zero runtime dependencies make adoption straightforward. Verify the license terms before use in proprietary projects, and confirm your GPU and CUDA versions meet the stated requirements.

Install

transformer-engine on PyPI

pip install transformer-engine

uv add transformer-engine

poetry add transformer-engine

Installing transformer-engine

Before you install

Low install friction with a pure Python wheel distribution. Active maintenance with a release 3 days old as of the fact sheet date. No runtime dependencies to manage.

License in practice

License treatment is unclear—the package description references a LICENSE file but the fact sheet provides no SPDX identifier or raw license text. Verify the actual license terms before use in proprietary or copyleft-sensitive contexts.

Quickstart

import transformer_engine.pytorch as te
from transformer_engine.common import recipe

model = te.Linear(768, 3072, bias=True)
fp8_recipe = recipe.DelayedScaling(margin=0, fp8_format=recipe.Format.E4M3)

with te.autocast(enabled=True, recipe=fp8_recipe):
    out = model(inp)

Requires NVIDIA GPU (Hopper, Ada, Ampere, or Blackwell) with CUDA 12.1+ (12.8+ for Blackwell), cuDNN 9.3+, and GCC 9+ or Clang 10+ with C++17 support.

Verify before relying

• Whether the license is open-source or proprietary—the fact sheet does not specify.
• Exact performance gains and memory savings for specific model sizes and GPU architectures.
• Compatibility with frameworks beyond PyTorch and JAX (e.g., TensorFlow).
• Whether the package works with Python versions below 3.10 despite requiring 3.10+.

Package facts

Evidence: transformer_engine-2.18.0-py3-none-any.whl

Tags

More Artificial Intelligence packages

Further reading