weicj/vLLM-2080Ti-Definitive

Running 27B and 35B parameter models on six-year-old consumer GPUs sounds like a compromise. This fork argues it is not, and it backs that argument with numbers.

The core bet is hardware arbitrage. Two RTX 2080 Ti cards with 22GB VRAM mods and NVLink cost roughly half what a single RTX 3090 Ti commands on the secondary market. The README's comparison table shows the dual setup delivers more than 1.6 times the CUDA core count, more than three times the physical Tensor Core count, and 44GB of total VRAM against the 3090 Ti's 24GB. The memory bandwidth advantage is more modest — about 22 percent — but for LLM inference, where decode throughput is often memory-bandwidth-bound, that still matters. The fork's job is to close the gap between what the Turing silicon can theoretically do and what vLLM will actually use.

The SM75 architecture (Turing's compute capability designation) is not officially a first-class vLLM target, which is why a fork exists at all. The patches cover Marlin quantized weight kernels, FlashQLA and FlashInfer attention paths adapted for SM75, TurboQuant INT8 KV cache compression, CUDAGraph integration, and Multi-Token Prediction decoding. The headline throughput figure — over 100 tokens per second on a single-request Qwen3.6 27B FP8 decode — is validated against official Qwen checkpoints, not synthetic benchmarks.

The project is explicit about what it is not. This is a single-concurrency personal-agent stack, not a multi-tenant serving platform. Long prefill is capacity-safe when tuned but effectively serialized by the scheduler on a TP=2 profile. Anyone expecting horizontal scaling or parallel long-prefill throughput is looking at the wrong tool.

The profile system is the practical core of the fork. Profiles are organized by model family, mode (safe / normal / fast / aggressive), weight format, and KV precision, and the launcher validates mode choices against profile declarations rather than silently accepting bad combinations. The README's hardware Q&A section is unusually candid: it notes that a Core i3-9100T with 16GB RAM outperformed a dual Xeon X5675 host by roughly 35 tokens per second on the same model route, and it flags thermal throttling as a failure mode that can masquerade as a software regression.

Gemma4 31B is listed as experimental, with INT8 and TurboQuant KV routes showing known initialization and capacity failures. That honesty about the support matrix is more useful than a blanket compatibility claim.

The fork tracks base vLLM 0.21.0 and does not promise to follow upstream closely — patches from upstream will be re-validated within the SM75 scope rather than merged automatically. For anyone who already owns this hardware and wants to run serious models locally without buying newer silicon, that is a reasonable trade.