tensorrt-llm
TensorRT-LLM accelerates large language model inference on NVIDIA GPUs through advanced optimization techniques including quantization, in-flight batching, and multi-GPU parallelism. Achieve production-grade throughput and latency for real-time applications with support for 100+ models.
TensorRT-LLM delivers 100× faster LLM inference on NVIDIA GPUs compared to PyTorch through compiled optimization and quantization.
AI-generated summary based on this skill's SKILL.md
Decision gist · record as of 2026-06-16
TensorRT-LLM delivers 100× faster LLM inference on NVIDIA GPUs compared to PyTorch through compiled optimization and quantization. TensorRT-LLM accelerates large language model inference on NVIDIA GPUs through advanced optimization techniques including quantization, in-flight batching, and multi-GPU parallelism. Achieve production-grade throughput and latency for real-time applications with support for 100+ models.
Use it when
- TensorRT-LLM supports multiple quantization strategies including FP8 and INT4 to reduce model size and memory bandwidth requirements.
- TensorRT-LLM provides production-grade LLM inference with advanced batching, memory efficiency, and multi-GPU scaling capabilities.
Verify before relying
Read SKILL.md below before installing (4 files). Open directory: indexed for reading, not audited.
Install
Orchestra-Research/AI-Research-SKILLs/tensorrt-llm · repository language: TeX
Open directory. Skills are indexed for reading, not audited. Review a skill's body before installing it.
Frequently asked questions
AI-generated answers based on this skill's SKILL.md and metadata
What is TensorRT-LLM and how does it optimize LLM inference?
TensorRT-LLM accelerates large language model inference on NVIDIA GPUs through advanced optimization techniques including quantization, in-flight batching, and multi-GPU parallelism. The framework delivers production-grade throughput and latency for real-time applications with support for 100+ models, enabling 10-100x faster inference compared to standard PyTorch implementations.
How can TensorRT-LLM quantization like FP8 improve performance?
TensorRT-LLM supports multiple quantization strategies including FP8 and INT4 to reduce model size and memory bandwidth requirements. Quantization enables faster computation on NVIDIA GPUs while maintaining model accuracy, allowing you to serve larger models or increase batch sizes within the same hardware constraints.
What is tensorrt llm inference optimization for production deployment?
TensorRT-LLM provides production-grade LLM inference with advanced batching, memory efficiency, and multi-GPU scaling capabilities. Features like in-flight batching, tensor parallelism, and speculative decoding enable high throughput and low latency serving. The framework supports deployment on A100, H100, and other NVIDIA GPUs with Docker containerization for easy production rollout.
Does TensorRT-LLM support multi-GPU scaling and tensor parallelism?
Yes, TensorRT-LLM enables multi-GPU scaling through tensor parallelism, allowing you to distribute large models across multiple NVIDIA GPUs. This approach optimizes inference performance for models that exceed single-GPU memory capacity while maintaining low latency through efficient communication patterns.
How does TensorRT-LLM compare to alternative frameworks like vLLM?
TensorRT-LLM is NVIDIA's native inference optimization framework specifically designed for maximum throughput and minimal latency on NVIDIA GPUs. It provides specialized support for NVIDIA hardware features, advanced quantization options, and production-grade batching strategies tailored for enterprise LLM serving workloads.
What licensing terms apply to TensorRT-LLM?
TensorRT-LLM is released under the MIT license, allowing free use, modification, and distribution in both open-source and commercial projects with minimal restrictions.
SKILL.md
Rendered from the published skill. Quoted content, verbatim.
TensorRT-LLM
NVIDIA's open-source library for optimizing LLM inference with state-of-the-art performance on NVIDIA GPUs.
When to use TensorRT-LLM
Use TensorRT-LLM when: - Deploying on NVIDIA GPUs (A100, H100, GB200) - Need maximum throughput (24,000+ tokens/sec on Llama 3) - Require low latency for real-time applications - Working with quantized models (FP8, INT4, FP4) - Scaling across multiple GPUs or nodes
Use vLLM instead when: - Need simpler setup and Python-first API - Want PagedAttention without TensorRT compilation - Working with AMD GPUs or non-NVIDIA hardware
**Use llama.cpp instead
(truncated - see the full file via the links below)
File tree — 4 files
12-inference-serving/tensorrt-llm/SKILL.md
12-inference-serving/tensorrt-llm/references/multi-gpu.md
12-inference-serving/tensorrt-llm/references/optimization.md
12-inference-serving/tensorrt-llm/references/serving.md
Let your AI agent find skills like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 56,283 agent skills by what they can do, searchable in plain language.
wish › “Deploy and serve LLMs with maximum throughput and minimal latency on NVIDIA GPUs”
Give your agent the search over MCP, or paste the wish link into any chat. No install? Search from any chat →
Related skills
This skill delivers operational patterns for tuning LLM inference performance, cost, and reliability in production. It covers latency budgeting, continuous batching, KV-cache optimization, speculative decoding, quantization strategies, and parallelism choices—with decision trees and checklists to guide infrastructure and serving stack tuning.
A structured decision guide for deploying and tuning vLLM in production environments. Covers the PagedAttention memory model, continuous batching, quantization tradeoffs, tensor/pipeline parallelism choices, and step-by-step operational workflows for diagnosing throughput, latency, and out-of-memory issues. Includes comparisons to alternative inference engines and guidance on when vLLM is the right fit.
vLLM accelerates LLM inference for production deployments using PagedAttention and continuous batching to achieve significantly higher throughput than standard approaches. It provides OpenAI-compatible endpoints, supports quantization methods like AWQ and GPTQ for memory-constrained setups, and enables tensor parallelism across multiple GPUs. Deploy via Docker, monitor with Prometheus metrics, and handle batch processing or real-time serving workflows.
serving-llms-vllm accelerates LLM inference for production environments through PagedAttention-based memory optimization and continuous batching. It supports OpenAI-compatible endpoints, quantization methods like GPTQ and AWQ, and tensor parallelism across multiple GPUs. Use this skill when deploying scalable LLM services that demand both low latency and high request throughput.
Model Serving equips you to run LLMs and traditional ML models in production using optimized inference engines like vLLM and BentoML. It covers GPU memory optimization, continuous batching, streaming response patterns, and integration with frontend applications for real-world AI deployments.
PyTorch Lightning eliminates boilerplate by structuring PyTorch projects into reusable LightningModules, automated Trainers, and data pipelines. Scale training across multiple GPUs or TPUs using strategies like DDP, FSDP, and DeepSpeed, with built-in support for callbacks, experiment logging, and best practices.
More skills agentsop-llm-engine-selection (MIT) · llama-cpp (MIT) · Llama Cpp (Apache-2.0)