---
id: hyper-connections
version: "0.4.11"
license: MIT License Copyright (c) 2024 Phil Wang Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in… (full text in the JSON record)
license_treatment: permissive
maintenance: active
---

# hyper-connections — Hyper-Connections

License: permissive · Maintenance: active · Downloads: 321.8K/mo

## What it is and what it does

Hyper-Connections is a PyTorch library that implements the multiple residual streams architecture from the Hyper-Connections paper (arXiv:2409.19606). It provides utilities to split a single residual pathway into multiple independent streams, apply branch operations to each stream separately, and then recombine them—a technique proposed to improve deep network training. The library wraps around standard PyTorch layers and offers both a high-level API (expand_stream, reduce_stream, init_hyper_conn) and lower-level control for manual stream management.

The package depends on torch, einops, and torch-einops-utils for tensor manipulation. It supports disabling the multi-stream behavior for ablation studies and includes experimental support for fractionated feature dimensions from follow-up research. The library is actively maintained, has no known security vulnerabilities, and is designed to integrate into existing model architectures with minimal code changes.

Use it for:

- Experimenting with multi-stream residual architectures in transformer or CNN models to improve training stability.
- Comparing standard residual connections against hyper-connections by toggling disable=True without refactoring code.
- Implementing fractionated feature dimension splits across residual pathways as proposed in follow-up papers.
- Prototyping deep networks (1000+ layers) where multiple residual streams may reduce gradient flow issues.
- Researching variants of residual connections for improved convergence in large-scale deep learning models.

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

Implements multiple residual streams in neural networks as proposed in the Hyper-Connections paper, allowing you to split, process, and recombine residual pathways within PyTorch models.

Yes, if you are actively researching or experimenting with advanced residual connection architectures in PyTorch. The library is low-friction to install, has no security issues, and offers a clean API for integrating multi-stream residuals into existing models. Not necessary for standard production models using conventional residual connections.

## Install

pip install hyper-connections
uv add hyper-connections
poetry add hyper-connections

## Installing hyper-connections

Before you install: Low install friction with a pure Python wheel and only three runtime dependencies (torch, einops, torch-einops-utils). Repository is active with recent commits and no known vulnerabilities.

License in practice: MIT License permits unrestricted use, modification, and distribution in commercial and private projects with minimal obligations—only attribution and license inclusion required.

Quickstart:

pip install hyper-connections

import torch
from hyper_connections import get_init_and_expand_reduce_stream_functions

init_hyper_conn, expand_stream, reduce_stream = get_init_and_expand_reduce_stream_functions(4)
residual = torch.randn(2, 1024, 512)
residual = expand_stream(residual)
residual = reduce_stream(residual)

Requires PyTorch and Python 3.9 or later; intended for use within existing neural network training pipelines.

Verify before relying:

- Whether the library supports automatic differentiation through all stream operations without gradient issues.
- Performance overhead of expand/reduce operations compared to standard residual connections at scale.
- Compatibility with specific PyTorch versions or distributed training frameworks.

## Package facts

- License: MIT License Copyright (c) 2024 Phil Wang Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in… (full text in the JSON record) (permissive)
- Python support: supports_current
- Install friction: low
- Maintenance: active
- Downloads: 321.8K/month (top 15,000 on PyPI)
- Known vulnerabilities: none known

## Tags

multiple residual streams pytorch, hyper connections neural networks, residual pathway splitting, deep learning residual architecture, torch residual connections, branching residual streams, neural network residual design, residual-networks, transformer-research, pytorch-extension

[View on SkillFed](https://skillfed.io/packages/hyper-connections) · [View on PyPI](https://pypi.org/project/hyper-connections/)