torchrl

What it is and what it does

TorchRL is a modular reinforcement learning library built on PyTorch and TensorDict, designed to keep research code close to the PyTorch programming model while scaling from local prototypes to distributed, multi-agent, and model-based workflows. It provides reusable components—environments, policies, collectors, replay buffers, transforms, and loss functions—that communicate through a common TensorDict data model, eliminating the need to rewrite training loops when switching between single-process, vectorized, multiprocess, or distributed execution.

The library emphasizes composability and explicit structure: data carries names, batch dimensions, and device information throughout the training loop, and each component (environment, policy, replay buffer, loss) can be swapped independently. It includes native PyTorch environments, wrappers for popular libraries (Gymnasium, DM Control, Brax, PettingZoo, VMAS, OpenSpiel, Isaac Lab), vectorized containers for local and multiprocess execution, and a rich set of transforms for observation normalization, action scaling, reward shaping, and state reconstruction.

Use it for:

• Prototyping single-agent RL algorithms locally, then scaling to distributed training without changing the data model or core training code.
• Building multi-agent RL systems with explicit agent grouping, value normalization, and algorithms like MAPPO and IPPO.
• Training recurrent policies with optimized GRU/LSTM reset handling and scan-based forward passes.
• Integrating custom MuJoCo environments or wrappers for third-party simulators (Gymnasium, Isaac Lab, Brax) with standardized observation and action specs.
• Managing large replay buffers with prioritized sampling, async writes, and optional CUDA-accelerated kernels.
• Implementing offline RL or model-based workflows where the same TensorDict interface handles both real and synthetic trajectories.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

TorchRL is a PyTorch-native toolkit for building reinforcement learning systems with composable components for environments, policies, collectors, replay buffers, and loss functions.

Yes, with conditions. TorchRL is actively maintained, has no known vulnerabilities, and is well-suited for research and production RL systems that need to scale from prototypes to distributed training. However, verify the license status before committing to proprietary use, and ensure your target platform has compatible wheels (Python 3.10–3.14, macOS/Linux/Windows). Medium install friction due to torch and tensordict dependencies is typical for PyTorch-based ML libraries.

Install

torchrl on PyPI

pip install torchrl

uv add torchrl

poetry add torchrl

Installing torchrl

Before you install

Medium install friction due to compiled dependencies (torch, tensordict). Wheels are available for Python 3.10–3.14 across macOS (ARM64), Linux (x86_64, aarch64), and Windows. Active maintenance with a release 31 days ago and 3519 repository stars.

License in practice

License treatment is unclear—no SPDX identifier or raw license string is present in the metadata. Verify the actual license before adopting in proprietary or restricted-distribution projects.

Quickstart

pip install torchrl torch tensordict

from torchrl.envs import PendulumEnv, TransformedEnv
from tensordict.nn import TensorDictModule
from torch import nn

env = TransformedEnv(PendulumEnv())
policy = TensorDictModule(
    nn.Sequential(nn.LazyLinear(64), nn.Tanh(), nn.Linear(64, 1)),
    in_keys=["observation"],
    out_keys=["action"],
)
rollout = env.rollout(max_steps=32, policy=policy)

Requires torch and tensordict as runtime dependencies; torch installation may require system-level build tools or pre-built wheels for your platform.

Verify before relying

• Whether the unclear license permits commercial or proprietary use without restrictions.
• Performance characteristics and scalability limits for distributed multi-agent training at scale.
• Compatibility with specific environment libraries (Gymnasium, DM Control, Isaac Lab) beyond the documented wrappers.

Package facts

Evidence: torchrl-0.13.3-cp310-cp310-macosx_11_0_arm64.whl; torchrl-0.13.3-cp310-cp310-manylinux_2_28_aarch64.whl; torchrl-0.13.3-cp310-cp310-manylinux_2_28_x86_64.whl; torchrl-0.13.3-cp310-cp310-win_amd64.whl; torchrl-0.13.3-cp311-cp311-macosx_11_0_arm64.whl; torchrl-0.13.3-cp311-cp311-manylinux_2_28_aarch64.whl; torchrl-0.13.3-cp311-cp311-manylinux_2_28_x86_64.whl; torchrl-0.13.3-cp311-cp311-win_amd64.whl; torchrl-0.13.3-cp312-cp312-macosx_11_0_arm64.whl; torchrl-0.13.3-cp312-cp312-manylinux_2_28_aarch64.whl; torchrl-0.13.3-cp312-cp312-manylinux_2_28_x86_64.whl; torchrl-0.13.3-cp312-cp312-win_amd64.whl; torchrl-0.13.3-cp313-cp313-macosx_12_0_arm64.whl; torchrl-0.13.3-cp313-cp313-manylinux_2_28_aarch64.whl; torchrl-0.13.3-cp313-cp313-manylinux_2_28_x86_64.whl; torchrl-0.13.3-cp313-cp313-win_amd64.whl; torchrl-0.13.3-cp314-cp314-macosx_12_0_arm64.whl; torchrl-0.13.3-cp314-cp314-manylinux_2_28_aarch64.whl; torchrl-0.13.3-cp314-cp314-manylinux_2_28_x86_64.whl; torchrl-0.13.3-cp314-cp314-win_amd64.whl

Keywords: reinforcement-learning, pytorch, rl, machine-learning

Tags

More Artificial Intelligence packages

Further reading