---
id: local-attention
version: "1.11.2"
license: MIT
license_treatment: permissive
maintenance: aging
---

# local-attention — Local attention, window with lookback, for language modeling

License: permissive · Maintenance: aging · Downloads: 311.6K/mo

## What it is and what it does

Local-attention is a PyTorch module that implements local windowed attention—a restricted form of transformer attention where each query attends only to keys and values within a fixed window, rather than the entire sequence. This is useful for language modeling because it allows transformer models to use computationally cheaper local attention in lower layers while reserving global attention for higher layers to integrate information across the full context.

The package provides configurable windowed attention with support for causal (auto-regressive) masking, lookback/lookahead windows, shared query-key spaces (Reformer-style), and automatic padding. It depends on torch for tensor operations and einops for tensor reshaping, plus hyper-connections as a utility dependency. The implementation is described as battle-tested across multiple repositories.

Use it for:

- Building efficient language models where full-sequence attention is computationally prohibitive for long contexts.
- Implementing Reformer-style architectures with shared query-key spaces and local attention windows.
- Replacing standard attention in transformer layers to reduce memory and compute while maintaining reasonable model capacity.
- Prototyping sparse attention patterns in transformer research without writing custom CUDA kernels.
- Training models on long sequences (e.g., 8192 tokens) where quadratic attention complexity becomes a bottleneck.

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

Implements local windowed attention for transformer models, allowing efficient attention computation over fixed-size windows rather than full sequences.

Yes, if you are building or experimenting with transformer models and need efficient local attention. The package has low install friction, permissive licensing, and no known vulnerabilities. However, the 394-day gap since the last release and aging maintenance status suggest it is stable but not actively developed—suitable for production use in established architectures but not for cutting-edge research requiring frequent updates.

## Install

pip install local-attention
uv add local-attention
poetry add local-attention

## Installing local-attention

Before you install: Low install friction with a pure Python wheel. Maintenance status is aging—last release was 394 days ago—but the repository remains active and unarchived with recent commits, suggesting the package is stable rather than abandoned.

License in practice: MIT license is permissive; you can use this package freely in commercial and private projects with minimal restrictions.

Quickstart:

pip install local-attention

import torch
from local_attention import LocalAttention

q = torch.randn(2, 8, 2048, 64)
k = torch.randn(2, 8, 2048, 64)
v = torch.randn(2, 8, 2048, 64)

attn = LocalAttention(dim=64, window_size=512, causal=True)
out = attn(q, k, v)

Requires torch and einops as runtime dependencies; intended for use within transformer architectures.

Verify before relying:

- Performance characteristics compared to standard attention or other sparse attention implementations
- Compatibility with recent PyTorch versions and whether Python 3.6 support is still maintained
- Whether the package is actively maintained or in maintenance-only mode given the 394-day release gap

## Package facts

- License: MIT (permissive)
- Python support: unspecified
- Install friction: low
- Maintenance: aging
- Downloads: 311.6K/month (top 15,000 on PyPI)
- Known vulnerabilities: none known

## Tags

local windowed attention transformer, efficient attention mechanism, sparse attention pytorch, local attention language model, window-based transformer attention, causal local attention, attention with lookback, transformer-attention, sparse-attention, language-modeling

[View on SkillFed](https://skillfed.io/packages/local-attention) · [View on PyPI](https://pypi.org/project/local-attention/)