s3tokenizer

What it is and what it does

S3Tokenizer is a PyTorch reimplementation of the Supervised Semantic Speech Tokenizer originally introduced in CosyVoice. It converts raw audio into discrete semantic tokens that preserve textual and paralinguistic information, designed for use in speech language models. The original authors released only an ONNX version and did not open-source the PyTorch implementation, so this package reverse-engineers that functionality.

The package provides three main workflows: offline batch inference (with CPU and multi-GPU support via torchrun), online token extraction during SpeechLLM training, and automatic long-audio processing for files longer than 30 seconds. It supports multiple model versions (V1 at 50Hz and 25Hz, V2 25Hz, V3 25Hz) and includes command-line tools for distributed processing. The key advantage over the original pipeline is throughput—batch inference with proper distribution can achieve significant speedups, and online extraction eliminates the need for separate preprocessing passes.

Use it for:

• Batch preprocessing of large audio datasets for speech model training with multi-GPU parallelization.
• Online speech code extraction during SpeechLLM forward passes to avoid offline preprocessing bottlenecks.
• Fine-tuning CosyVoice models when batch-size-1 extraction in the original pipeline is too slow.
• Building speech-to-text or speech-to-speech systems that require semantic token representations.
• Processing long-form audio (>30 seconds) with automatic windowing and segment batching.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Converts audio into discrete semantic speech tokens using a PyTorch implementation of the S3Tokenizer model, supporting batch inference and online extraction during model training.

Yes, if you are training or fine-tuning speech language models (especially CosyVoice-based systems) and need efficient tokenization. The reverse-engineered PyTorch implementation removes a significant bottleneck in the original pipeline, and the package is actively maintained with no known vulnerabilities. The permissive Apache 2.0 license poses no restrictions. Install friction is low. Caveat: the aging maintenance status and relatively small user base (527 stars) mean fewer eyes on edge cases—verify performance on your specific hardware before committing to production.

Install

s3tokenizer on PyPI

pip install s3tokenizer

uv add s3tokenizer

poetry add s3tokenizer

Installing s3tokenizer

Before you install

Low install friction with a pure Python wheel. Runtime dependencies include torch, torchaudio, numpy, and einops—all standard ML stack components. Maintenance status is aging (235 days since release), but the repository remains active with recent commits and no archived status.

License in practice

Apache 2.0 permissive license allows commercial use, modification, and redistribution with minimal restrictions, making it suitable for both research and production deployments.

Quickstart

import s3tokenizer

tokenizer = s3tokenizer.load_model("speech_tokenizer_v1").cuda()
audio = s3tokenizer.load_audio("audio.wav")
mels = s3tokenizer.log_mel_spectrogram(audio)
mels, mels_lens = s3tokenizer.padding([mels])
codes, codes_lens = tokenizer.quantize(mels.cuda(), mels_lens.cuda())

Requires PyTorch and torchaudio installed; GPU recommended for practical throughput (CPU batch inference supported but slower).

Verify before relying

• Whether the ~790x speedup claim versus the original CosyVoice pipeline is reproducible on typical hardware configurations.
• Exact memory requirements for distributed inference across multiple GPUs.
• Whether online extraction during training introduces measurable latency or gradient computation overhead.

Package facts

Evidence: s3tokenizer-0.3.0-py3-none-any.whl

Tags

More Scientific/Engineering packages

Further reading