foundry-local-sdk

What it is and what it does

Foundry Local SDK is a Python control plane for running large language models, embeddings, and speech-to-text inference entirely on your local machine. It wraps a native Foundry Local Core library (via ctypes FFI) and provides a model catalog, download/cache management, and OpenAI-compatible APIs for chat completions (streaming and non-streaming), tool calling, embeddings, and Whisper-based audio transcription. The package selects the appropriate ONNX Runtime variant per platform (CUDA on Linux x86_64, CPU/WebGPU elsewhere) and publishes a separate WinML variant for Windows.

You initialize a singleton manager with a configuration, discover models from the catalog, download and load them into memory, then interact via chat or embedding clients. The SDK handles model lifecycle (load, unload, cache), execution provider discovery and registration, and optional HTTP endpoints for multi-process scenarios. It is actively maintained, recently released, and has no known security vulnerabilities.

Use it for:

• Run private LLM inference on-device without sending data to cloud APIs or paying per-token fees
• Build chat applications with streaming responses using OpenAI-compatible client code
• Generate text embeddings locally for semantic search or RAG workflows
• Transcribe audio files or streams using Whisper models without external services
• Prototype and test LLM-powered features before deploying to production infrastructure
• Call functions/tools from LLM responses in a local environment with full control

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

A Python SDK for discovering, downloading, loading, and running local AI models with OpenAI-compatible chat, embeddings, and audio transcription APIs — no cloud required.

Yes, if you need local LLM inference with OpenAI-compatible APIs and don't require production-grade stability. The package is actively maintained, has low install friction, permissive licensing, and zero known vulnerabilities. It is in Alpha status (Development Status :: 3), so expect API changes and incomplete documentation; verify that the model catalog and hardware support match your target deployment before committing to it.

Install

foundry-local-sdk on PyPI

pip install foundry-local-sdk

uv add foundry-local-sdk

poetry add foundry-local-sdk

Installing foundry-local-sdk

Before you install

Low friction wheel install with platform-specific native dependencies (onnxruntime variants, foundry-local-core). Active maintenance with recent release (15 days old); requires Python 3.11+.

License in practice

MIT license permits unrestricted use, modification, and distribution with minimal restrictions.

Quickstart

pip install foundry-local-sdk

from foundry_local_sdk import Configuration, FoundryLocalManager

config = Configuration(app_name="MyApp")
FoundryLocalManager.initialize(config)
manager = FoundryLocalManager.instance
model = manager.catalog.get_model("phi-3.5-mini")
model.load()
client = model.get_chat_client()
response = client.complete_chat([{"role": "user", "content": "Why is the sky blue?"}])
print(response.choices[0].message.content)

Requires Python 3.11 or later. Platform-specific native binaries (CUDA-enabled on Linux x86_64, CPU/WebGPU on other platforms) are installed as wheel dependencies; WinML variant available for Windows only.

Verify before relying

• Whether model catalog is pre-populated or requires external download/configuration
• Typical memory footprint and hardware requirements for common models
• Latency characteristics for inference on different hardware (CPU vs GPU)
• Whether streaming chat and embeddings APIs are fully OpenAI-compatible or have known differences

Package facts

Evidence: foundry_local_sdk-1.2.4-py3-none-any.whl

Tags

More Software Development packages