Semantic Kernel Production Guide (Python)
A production-shaped FastAPI service for Semantic Kernel in Python: shared AI service with a kernel per request, OpenTelemetry and token accounting, retries and timeouts, a caching filter and offline tests.
Targets
semantic-kernel1.45.0 on Python 3.10+ (checked 2026-11-19). The modules below were run together: the tests pass against a stub chat service, and the token counters were read back from an in-memory OpenTelemetry reader.
This page is the Python implementation of the production guide, which covers the reasoning (deployment shape, security, scaling, cost) and the release checklist. Here: a small FastAPI service you can copy, with each Semantic Kernel-specific decision explained.
pip install "semantic-kernel==1.45.0" fastapi "uvicorn[standard]" azure-monitor-opentelemetrypip install pytest httpx # for the testsSemantic Kernel already depends on openai, pydantic, azure-identity and the OpenTelemetry API and SDK; let it choose their versions and lock the result (uv lock, pip-compile).
app.py FastAPI app: shared service, kernel per requestsettings.py configuration from the environmenttelemetry.py exporter set-up and a token-usage filtercache.py optional exact-match cache filtertests/test_app.py1. Settings
Section titled “1. Settings”import osfrom dataclasses import dataclass
@dataclass(frozen=True)class Settings: azure_openai_endpoint: str azure_openai_deployment: str appinsights_connection_string: str | None request_timeout_s: float = 60.0
@classmethod def from_env(cls) -> "Settings": return cls( azure_openai_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"], azure_openai_deployment=os.environ["AZURE_OPENAI_CHAT_DEPLOYMENT_NAME"], appinsights_connection_string=os.getenv("APPLICATIONINSIGHTS_CONNECTION_STRING"), )Secrets do not appear here: the service authenticates to Azure OpenAI with managed identity, and the Application Insights connection string is injected by the platform (Container Apps, App Service or a Kubernetes secret).
2. The app: one service, one kernel per request
Section titled “2. The app: one service, one kernel per request”import asyncioimport loggingfrom contextlib import asynccontextmanagerfrom typing import Annotated
from azure.identity.aio import DefaultAzureCredential, get_bearer_token_providerfrom fastapi import Depends, FastAPI, HTTPException, Requestfrom openai import AsyncAzureOpenAIfrom pydantic import BaseModel
from semantic_kernel import Kernelfrom semantic_kernel.connectors.ai.open_ai import AzureChatCompletionfrom semantic_kernel.exceptions import KernelInvokeExceptionfrom semantic_kernel.functions import KernelArguments, KernelFunctionFromPrompt
from settings import Settingsfrom telemetry import register_filters, setup_telemetry
logger = logging.getLogger("app")
# Prompt functions are immutable and can be shared; plugins and filters are added per kernel.REPLY = KernelFunctionFromPrompt( function_name="reply", plugin_name="chat", prompt="You are a concise support assistant.\n\nCustomer: {{$input}}",)
def create_chat_service(settings: Settings) -> AzureChatCompletion: client = AsyncAzureOpenAI( azure_endpoint=settings.azure_openai_endpoint, azure_ad_token_provider=get_bearer_token_provider( # managed identity in Azure, `az login` locally DefaultAzureCredential(), "https://cognitiveservices.azure.com/.default"), api_version="2024-10-21", max_retries=2, # the OpenAI SDK retries 429/5xx with back-off; don't add a second layer timeout=30.0, # per attempt ) return AzureChatCompletion(deployment_name=settings.azure_openai_deployment, async_client=client)
@asynccontextmanagerasync def lifespan(app: FastAPI): settings = Settings.from_env() setup_telemetry(settings.appinsights_connection_string) app.state.settings = settings # One AI service per process: it holds the HTTP connection pool and is safe to share. app.state.chat_service = create_chat_service(settings) yield
app = FastAPI(lifespan=lifespan)
def get_kernel(request: Request) -> Kernel: """A fresh kernel per request, built around the shared service.""" kernel = Kernel() kernel.add_service(request.app.state.chat_service) register_filters(kernel) return kernel
class ChatRequest(BaseModel): message: str tenant: str = "default"
class ChatResponse(BaseModel): reply: str
@app.get("/health")async def health() -> dict: return {"status": "ok"} # liveness: never calls the model
@app.post("/chat", response_model=ChatResponse)async def chat(body: ChatRequest, request: Request, kernel: Annotated[Kernel, Depends(get_kernel)]): try: result = await asyncio.wait_for( kernel.invoke(REPLY, KernelArguments(input=body.message, tenant=body.tenant)), timeout=request.app.state.settings.request_timeout_s, ) except (KernelInvokeException, asyncio.TimeoutError): logger.exception("chat failed") raise HTTPException(status_code=503, detail="The assistant is unavailable.") # no exception text return ChatResponse(reply=str(result))What matters here:
- The AI service is created once, in the lifespan handler, and kept on
app.state. It owns the HTTP connection pool and the token provider, and is safe to share. - The kernel is built per request in
get_kernel. AKernelis a cheap container; building one per request means plugins or filters that a request adds (a tenant’s tools, a per-user approval filter) cannot leak into another request. This is the same rule as registeringKernelas transient in .NET. - Retries happen once, in the OpenAI SDK.
max_retriesandtimeouton theAsyncAzureOpenAIclient control them; the request-levelasyncio.wait_forbounds the whole call, including any tool-calling round trips. - Errors are logged, not returned.
KernelInvokeExceptionwraps the connector’sServiceResponseException; the client sees a generic 503.
For agents, build the ChatCompletionAgent inside the handler in the same way (ChatCompletionAgent(kernel=kernel, ...)): construction is cheap, and the conversation state belongs in a thread you load from a store, not in the agent object. The streaming server shows that shape with SSE.
3. Telemetry and token usage
Section titled “3. Telemetry and token usage”from opentelemetry import metrics
from semantic_kernel.filters import FilterTypes, FunctionInvocationContext
meter = metrics.get_meter("app.sk")prompt_tokens = meter.create_counter("app.llm.tokens.prompt", unit="{token}")completion_tokens = meter.create_counter("app.llm.tokens.completion", unit="{token}")
def setup_telemetry(connection_string: str | None) -> None: """Export SK's spans, metrics and logs. Call once, before the first request.""" if connection_string: from azure.monitor.opentelemetry import configure_azure_monitor
configure_azure_monitor(connection_string=connection_string, logger_name="semantic_kernel")
async def record_token_usage(context: FunctionInvocationContext, next) -> None: """Function-invocation filter: count tokens per function and tenant.
Python SK has no token-usage metric of its own, so read the usage the connector attaches to each completion's metadata. """ await next(context) if context.result is None: return tenant = str((context.arguments or {}).get("tenant", "unknown")) attrs = {"function": context.function.fully_qualified_name, "tenant": tenant} for completion in context.result.metadata.get("metadata", []): usage = completion.get("usage") if usage is not None: prompt_tokens.add(usage.prompt_tokens or 0, attrs) completion_tokens.add(usage.completion_tokens or 0, attrs)
def register_filters(kernel) -> None: kernel.add_filter(FilterTypes.FUNCTION_INVOCATION, record_token_usage)Semantic Kernel instruments itself with OpenTelemetry under the semantic_kernel namespace. In Python that means:
- Spans for each function invocation and, with
SEMANTICKERNEL_EXPERIMENTAL_GENAI_ENABLE_OTEL_DIAGNOSTICS=trueset in the environment, GenAI semantic-convention spans for model calls, including thegen_ai.usage.input_tokens/gen_ai.usage.output_tokensattributes. The..._SENSITIVE=truevariant also records prompts and completions; leave it off in production. - Metrics: only
semantic_kernel.function.invocation.durationandsemantic_kernel.function.streaming.durationhistograms. The token-usage metrics (semantic_kernel.function.invocation.token_usage.*) exist only in the .NET SDK.
So token accounting in Python is yours to record. The filter above reads the usage (a CompletionUsage) that the OpenAI connectors attach to every completion’s metadata and adds it to counters with function and tenant attributes. Agents call the chat service directly, not through a kernel function, so for agent turns read response.message.metadata["usage"] instead (the resilient agent recipe does).
4. Caching deterministic functions
Section titled “4. Caching deterministic functions”import hashlibimport json
from semantic_kernel.filters import FunctionInvocationContextfrom semantic_kernel.functions import FunctionResult
CACHEABLE = {"text-classify"} # deterministic functions only_cache: dict[str, str] = {} # use Redis or similar when you run several replicas
async def cache_filter(context: FunctionInvocationContext, next) -> None: name = context.function.fully_qualified_name if name not in CACHEABLE: await next(context) return args = {k: str(v) for k, v in (context.arguments or {}).items()} key = hashlib.sha256(json.dumps([name, args], sort_keys=True).encode()).hexdigest() if key in _cache: context.result = FunctionResult(function=context.function.metadata, value=_cache[key]) return # skip the model call await next(context) if context.result is not None: _cache[key] = str(context.result)Register it in get_kernel with kernel.add_filter(FilterTypes.FUNCTION_INVOCATION, cache_filter). Setting context.result and returning without calling next skips the model call. Cache only functions whose output should not vary (classification, extraction) and include everything that changes the answer in the key; an exact-match cache misses near-duplicate questions by design.
5. Tests
Section titled “5. Tests”from types import SimpleNamespace
import pytestfrom fastapi.testclient import TestClient
from semantic_kernel.connectors.ai.chat_completion_client_base import ChatCompletionClientBasefrom semantic_kernel.connectors.ai.completion_usage import CompletionUsagefrom semantic_kernel.contents import AuthorRole, ChatMessageContent
import app as app_module
class StubChat(ChatCompletionClientBase): """Offline chat service: canned reply plus usage metadata, like a real connector."""
async def _inner_get_chat_message_contents(self, chat_history, settings): return [ChatMessageContent( role=AuthorRole.ASSISTANT, content="Have you tried turning it off and on again?", metadata={"usage": CompletionUsage(prompt_tokens=12, completion_tokens=9)}, )]
@pytest.fixturedef client(monkeypatch): monkeypatch.setenv("AZURE_OPENAI_ENDPOINT", "https://example.openai.azure.com/") monkeypatch.setenv("AZURE_OPENAI_CHAT_DEPLOYMENT_NAME", "test") monkeypatch.setattr(app_module, "create_chat_service", lambda settings: StubChat(ai_model_id="stub")) with TestClient(app_module.app) as c: yield c
def test_chat(client): r = client.post("/chat", json={"message": "My router is broken", "tenant": "acme"}) assert r.status_code == 200 assert "turning it off" in r.json()["reply"]
def test_kernel_per_request(client): request = SimpleNamespace(app=client.app) k1, k2 = app_module.get_kernel(request), app_module.get_kernel(request) assert k1 is not k2 assert k1.get_service() is k2.get_service()Subclassing ChatCompletionClientBase gives a service the kernel accepts anywhere a real one goes, including behind agents; unittest.mock.AsyncMock does not. Replacing create_chat_service keeps the test offline and avoids acquiring an Azure token. Add a tests/conftest.py that puts the project root on sys.path if you keep tests in a subfolder.
6. Container
Section titled “6. Container”FROM python:3.12-slimWORKDIR /appCOPY requirements.txt .RUN pip install --no-cache-dir -r requirements.txtCOPY . .RUN useradd -m -u 1000 appuserUSER appuserEXPOSE 8000CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]The service needs AZURE_OPENAI_ENDPOINT, AZURE_OPENAI_CHAT_DEPLOYMENT_NAME, optionally APPLICATIONINSIGHTS_CONNECTION_STRING and SEMANTICKERNEL_EXPERIMENTAL_GENAI_ENABLE_OTEL_DIAGNOSTICS, and an identity with the Cognitive Services OpenAI User role on the Azure OpenAI resource (on AKS, workload identity; AZURE_CLIENT_ID selects a user-assigned identity). Point liveness probes at /health, which never calls the model. Scale on concurrency and watch the provider’s tokens-per-minute quota, which usually runs out before CPU does.
Revision history
Section titled “Revision history”| Date | Version | Changes |
|---|---|---|
| 2026-10-21 | 1.45.0 | Rewritten. The previous modules did not import (Config, ChatRequest, logger and a health module were undefined), built one kernel for the whole app despite the kernel-per-request advice, and claimed Python SK emits token-usage metrics (only .NET does). Generic Kubernetes manifests and checklists replaced by a link to the language-neutral production guide. |