tensorrt-llm
TensorRT-LLM accelerates large language model inference on NVIDIA GPUs through advanced optimization techniques including quantization, in-flight batching, and multi-GPU parallelism. Achieve production-grade throughput and latency for real-time applications with support for 100+ models.
TensorRT-LLM delivers 100× faster LLM inference on NVIDIA GPUs compared to PyTorch through compiled optimization and quantization.
AI-generated summary based on this skill's SKILL.md
Install
Orchestra-Research/AI-Research-SKILLs/tensorrt-llm · repository language: TeX
git clone https://github.com/Orchestra-Research/AI-Research-SKILLs
cp -r AI-Research-SKILLs/12-inference-serving/tensorrt-llm ~/.claude/skills/tensorrt-llmnpx skillfed install Orchestra-Research/AI-Research-SKILLs/tensorrt-llmFrequently asked questions
AI-generated answers based on this skill's SKILL.md and metadata
What is TensorRT-LLM and how does it optimize LLM inference?
TensorRT-LLM accelerates large language model inference on NVIDIA GPUs through advanced optimization techniques including quantization, in-flight batching, and multi-GPU parallelism. The framework delivers production-grade throughput and latency for real-time applications with support for 100+ models, enabling 10-100x faster inference compared to standard PyTorch implementations.
How can TensorRT-LLM quantization like FP8 improve performance?
TensorRT-LLM supports multiple quantization strategies including FP8 and INT4 to reduce model size and memory bandwidth requirements. Quantization enables faster computation on NVIDIA GPUs while maintaining model accuracy, allowing you to serve larger models or increase batch sizes within the same hardware constraints.
What is tensorrt llm inference optimization for production deployment?
TensorRT-LLM provides production-grade LLM inference with advanced batching, memory efficiency, and multi-GPU scaling capabilities. Features like in-flight batching, tensor parallelism, and speculative decoding enable high throughput and low latency serving. The framework supports deployment on A100, H100, and other NVIDIA GPUs with Docker containerization for easy production rollout.
Does TensorRT-LLM support multi-GPU scaling and tensor parallelism?
Yes, TensorRT-LLM enables multi-GPU scaling through tensor parallelism, allowing you to distribute large models across multiple NVIDIA GPUs. This approach optimizes inference performance for models that exceed single-GPU memory capacity while maintaining low latency through efficient communication patterns.
How does TensorRT-LLM compare to alternative frameworks like vLLM?
TensorRT-LLM is NVIDIA's native inference optimization framework specifically designed for maximum throughput and minimal latency on NVIDIA GPUs. It provides specialized support for NVIDIA hardware features, advanced quantization options, and production-grade batching strategies tailored for enterprise LLM serving workloads.
What licensing terms apply to TensorRT-LLM?
TensorRT-LLM is released under the MIT license, allowing free use, modification, and distribution in both open-source and commercial projects with minimal restrictions.
SKILL.md
rendered from the published skill — quoted content, verbatim
TensorRT-LLM
NVIDIA's open-source library for optimizing LLM inference with state-of-the-art performance on NVIDIA GPUs.
When to use TensorRT-LLM
Use TensorRT-LLM when: - Deploying on NVIDIA GPUs (A100, H100, GB200) - Need maximum throughput (24,000+ tokens/sec on Llama 3) - Require low latency for real-time applications - Working with quantized models (FP8, INT4, FP4) - Scaling across multiple GPUs or nodes
Use vLLM instead when: - Need simpler setup and Python-first API - Want PagedAttention without TensorRT compilation - Working with AMD GPUs or non-NVIDIA hardware
**Use llama.cpp instead
(truncated - see the full file via the links below)
Read as markdown · JSON record · Browse the source repository
File tree — 4 files
12-inference-serving/tensorrt-llm/SKILL.md
12-inference-serving/tensorrt-llm/references/multi-gpu.md
12-inference-serving/tensorrt-llm/references/optimization.md
12-inference-serving/tensorrt-llm/references/serving.md