model-hosting-container-standards

What it is and what it does

This package standardizes how ML models are deployed to Amazon SageMaker by providing a framework-agnostic Python toolkit built on FastAPI. It handles the boilerplate of setting up `/ping` and `/invocations` endpoints that SageMaker expects, with support for framework integration (vLLM, TensorRT-LLM) via decorators and customer customization via environment variables or custom handler functions. The package also supports LoRA adapter injection, automatic dependency installation from model artifacts, and comprehensive logging and error handling.

Typical use: framework developers register default handlers using `@register_ping_handler` and `@register_invocation_handler` decorators; customers then override behavior by placing a `model.py` script in their model artifact folder with `@custom_ping_handler` or `@custom_invocation_handler` decorators, or by setting environment variables. Handler resolution follows a priority chain (environment variables → customer decorators → function discovery → framework decorators), so customization is straightforward without forking the framework.

Use it for

• Deploy vLLM or TensorRT-LLM models to SageMaker endpoints with standardized health-check and invocation routes.
• Add LoRA adapter support to model inference by injecting adapter IDs from request headers into model calls.
• Customize model behavior per deployment (e.g., different preprocessing logic) by writing a simple `model.py` without modifying framework code.
• Automatically install and manage model-specific dependencies from a `requirements.txt` in model artifacts before server startup.
• Integrate multiple ML frameworks under a single SageMaker hosting standard without writing repetitive endpoint boilerplate.

Worth the install?

AI-flagged interpretation of the facts on this page. Verify before relying on it.

Install

model-hosting-container-standards on PyPI

Before you install

Low friction: pure Python wheel with no compiled dependencies. Active maintenance (released 60 days ago), supports current Python versions (3.10–3.14). Seven runtime dependencies (fastapi, httpx, jmespath, pydantic, setuptools, starlette, supervisor) are all widely used and stable.

Requires Python >= 3.10 and FastAPI >= 0.117.1. Intended for use within a SageMaker container or FastAPI application; not a standalone server.

License in practice

Apache-2.0 (permissive): you may use, modify, and distribute this package freely in commercial and open-source projects, provided you include a copy of the license and state any material changes.

Quickstart

# Install
pip install model-hosting-container-standards

# In your FastAPI/vLLM server code
import model_hosting_container_standards.sagemaker as sagemaker_standards
from fastapi import Request, Response
import json

@sagemaker_standards.register_ping_handler
async def ping(raw_request: Request) -> Response:
    return Response(content='{"status": "healthy"}', media_type="application/json")

@sagemaker_standards.register_invocation_handler
async def invocations(raw_request: Request) -> Response:
    body = json.loads(await raw_request.body())
    return Response(content=json.dumps({"predictions": ["result"]}), media_type="application/json")

Package facts

Evidence: model_hosting_container_standards-0.1.16-py3-none-any.whl

Tags

More Artificial Intelligence packages

See also sagemaker-serve · sagemaker-inference · sagemaker-training · opentelemetry-instrumentation-sagemaker · sagemaker-schema-inference-artifacts · sagemaker-containers · sagemaker-train · tensorflow-serving-api · azure-ai-agentserver-invocations · sagemaker-data-insights