gen-worker

What it is and what it does

gen-worker is a Python SDK for building serverless endpoints that run on Cozy's managed worker pool. You write a single decorated function or class, and the SDK handles the infrastructure: model discovery and download from HuggingFace, Civitai, or other sources; GPU device placement and low-VRAM offload; request scheduling and cancellation; file I/O and streaming; and reporting back to the control plane. It abstracts away device management entirely—endpoint code never touches `.to("cuda")` or offload configuration.

The package supports both simple stateless endpoints (a decorated function) and stateful ones (a class with setup() and handler methods). It includes bindings for popular model sources (HF, Hub, Civitai, ModelScope), optional support for PyTorch inference and training, media I/O for images, audio, and video, and streaming via async generators. Local development is supported through a CLI (gen-worker run, serve, invoke, prefetch) for testing before deployment. The SDK is actively maintained, requires Python 3.12+, and has no known vulnerabilities.

Use it for:

• Deploy a text-to-image model (e.g., Stable Diffusion XL) as a managed endpoint with automatic GPU placement and VRAM optimization.
• Build a lightweight API proxy endpoint that routes requests to external services without needing GPU resources.
• Create a model conversion pipeline that ingests checkpoints, quantizes them to fp8, and publishes them to the hub.
• Develop a streaming inference endpoint (e.g., token-by-token LLM generation) using async generators.
• Run multi-model endpoints where users select a checkpoint at request time via a ModelChoice enum.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

A Python SDK for writing serverless endpoints that run on Cozy's worker pool, handling model discovery, scheduling, download, device placement, cancellation, and streaming.

Yes, if you are building endpoints for Cozy's platform or need a framework that abstracts GPU orchestration and model management. The SDK is actively maintained, has low install friction, permissive licensing, and zero known vulnerabilities. It is production-ready for inference and training workloads. Not suitable if you need to deploy to other serverless platforms or prefer manual device management.

Install

gen-worker on PyPI

pip install gen-worker

uv add gen-worker

poetry add gen-worker

Installing gen-worker

Before you install

Low friction install with a pure-Python wheel. Active maintenance with a release on 2026-08-14. Requires Python 3.12 or later. Twelve runtime dependencies including grpcio, huggingface-hub, and protobuf suggest a mature, production-oriented codebase.

License in practice

MIT license (permissive) means you can use, modify, and distribute this package freely in commercial and private projects with minimal restrictions.

Quickstart

pip install gen-worker

from gen_worker import RequestContext, endpoint
import msgspec

class Input(msgspec.Struct):
    prompt: str

@endpoint
def echo(ctx: RequestContext, payload: Input) -> str:
    return f"got: {payload.prompt}"

# Run locally: gen-worker run --payload '{"prompt": "hello"}'

Requires Python 3.12 or later. For model inference with PyTorch, install gen-worker[torch]. Local testing requires uv; never pip install globally as it shadows the working tree.

Verify before relying

• Whether the platform's profiling gate for VRAM measurement works reliably across different GPU types and model sizes.
• How well the per-request state isolation (ctx.for_request) scales under high concurrency.
• Whether optional extras ([images], [audio], [video], [vision]) are fully documented and stable.

Package facts

Evidence: gen_worker-0.116.0-py3-none-any.whl

Keywords: ai, cozy, inference, ml, serverless

Tags

More Artificial Intelligence packages