Skip to content
Sk

Semantic Kernel: Production Deployment and Operations Guide

Language-neutral production guidance for Semantic Kernel — deployment shape, configuration and secrets, resilience, observability, security, scaling, cost and testing.

Applies to semantic-kernel 1.45.0 (Python) and Microsoft.SemanticKernel 1.81.0 (.NET), checked 2026-11-19. This page holds the reasoning and the checklist; the runnable code is in the Python production guide (FastAPI service, telemetry, tests) and the .NET production guide (resilience, telemetry, tests).

Semantic Kernel is in long-term support while its successor, Microsoft Agent Framework, receives most new features. Everything below applies equally to both, which keeps a later migration cheap.


  1. Deployment shape
  2. Configuration and secrets
  3. Resilience
  4. Observability
  5. Security
  6. Scaling and state
  7. Performance and caching
  8. Cost management
  9. Testing
  10. Release checklist

  • Stateless service, stateful stores. Run SK inside an ordinary web or worker process (FastAPI, ASP.NET Core, Azure Functions, a queue consumer). Keep conversation history, vector data and caches in external stores so any replica can serve any request.
  • Build the kernel per request (or per tenant). AI service clients are expensive and thread-safe — create them once. Kernels are cheap; creating one per request stops plugins and filters added for one user leaking to another. In .NET register Kernel as transient; in Python build it in a dependency or factory.
  • Long-running agent work goes to a queue. Multi-agent orchestrations can take minutes. Accept the request, enqueue it, process it in a worker and let the client poll or subscribe (SSE/WebSocket) for the result.
  • Containers. Use slim official base images (python:3.12-slim, mcr.microsoft.com/dotnet/aspnet:10.0), run as a non-root user, and expose a health endpoint that does not call the model (a readiness check may verify configuration and store connectivity).

ConcernRecommendation
Azure OpenAI authManaged identity / Entra ID (DefaultAzureCredential) rather than API keys
Other provider keysA secret store (Azure Key Vault, AWS Secrets Manager, Kubernetes secrets) injected as environment variables
Model and deployment namesConfiguration, not code — they change more often than the code does
Prompt templatesVersioned files (YAML/skprompt.txt) reviewed like code
Per-environment differencesSettings objects (OpenAISettings, AzureOpenAISettings in Python; IConfiguration in .NET)

Never log API keys, and never return raw exception text to clients — connector errors can include request details.


  • Know what already retries. The OpenAI and Azure OpenAI connectors use the official OpenAI SDKs, which retry 429 and 5xx responses with back-off. Adding a second retry layer multiplies attempts; if you add one, keep it small and limited to service errors (ServiceResponseException in Python, HttpOperationException with a 429/5xx StatusCode in .NET).
  • Timeouts everywhere. Model calls, tool calls and orchestration results (result.get(timeout=...) / GetValueAsync(timeout)) all need explicit timeouts.
  • Circuit breakers around flaky tools and secondary providers, so one failing dependency does not tie up every worker.
  • Fallback models. Register a second chat service and fall back to it on repeated failure; record which model answered.
  • Idempotent tools. The model may call a tool twice, and retries replay turns. Tools with side effects need idempotency keys.
  • Loop limits. Cap automatic function-calling iterations and orchestration rounds, and use an auto-function-invocation filter to terminate runaway tool loops.

Semantic Kernel emits OpenTelemetry traces, metrics and logs:

  • Traces for function invocations and, with GenAI diagnostics enabled, model calls using the GenAI semantic conventions (model, token counts, finish reason).
  • Metrics: semantic_kernel.function.invocation.duration and .streaming.duration in both SDKs. Token usage per function (semantic_kernel.function.invocation.token_usage.prompt / .completion) is recorded only by .NET. In Python, read usage from each completion’s metadata (or the gen_ai.usage.* span attributes) and record it yourself; the Python production guide does this in a filter.
  • Logs through the standard logging frameworks (logging / ILogger).

Enable GenAI diagnostics with SEMANTICKERNEL_EXPERIMENTAL_GENAI_ENABLE_OTEL_DIAGNOSTICS=true (Python) or the Microsoft.SemanticKernel.Experimental.GenAI.EnableOTelDiagnostics AppContext switch (.NET). The sensitive variants also capture prompts and completions; use them only in development.

Export with OTLP to any back end, or with the Azure Monitor distro to Application Insights. Alert on: error rate per function, p95 latency per model, token usage per tenant (sudden growth usually means a prompt or loop regression), and tool-call failure rate.


  • Keep template encoding on. Prompt templates HTML-encode inserted variables and function results, so user text cannot open a new <message role="system">. Do not set allow_dangerously_set_content / AllowDangerouslySetContent on anything that carries user input, retrieved documents or tool output.
  • Prompt injection. Encoding protects message structure, not meaning: treat retrieved documents, tool output and user input as untrusted. Use a prompt-render filter to apply checks, keep system instructions separate from data, and never let model output choose arbitrary URLs, file paths or SQL. For OpenAPI plugins, Python checks request URLs against server_url_validation_allowed_base_urls and blocks private network addresses by default; keep it that way.
  • Least-privilege tools. Give each agent only the plugins it needs, and narrow what is advertised per call with FunctionChoiceBehavior.Auto(filters={"included_plugins": [...]}) (Python) or FunctionChoiceBehavior.Auto(functions: [...]) (.NET). Use read-only credentials for read-only tools, and gate destructive tools with an auto-function-invocation filter.
  • Data handling. Redact personal data before it reaches prompts or logs (filters again), and check the data-residency settings of each model provider and vector store.
  • MCP servers run code with your credentials — pin their versions, run local ones in containers, and review the tools they expose.
  • Dependencies. Pin SK exactly, update deliberately, and watch both SDKs’ release notes; several .NET packages are still preview or alpha.

StateWhere to keep it
Chat history / agent threadsA database or cache keyed by session; reload into a ChatHistory / thread per request. Service-backed agents (Azure AI, OpenAI Responses) keep threads server-side.
Long-term knowledgeA vector store collection
Orchestration progressPersist inputs and outputs per step if a run must survive a restart (SK orchestrations are in-process); for durable multi-step processes prefer Agent Framework workflows
CachesRedis or equivalent shared cache

Scale horizontally on CPU and request concurrency; model latency, not CPU, is usually the bottleneck, so watch queue depth and provider rate limits (tokens per minute) before adding replicas. Spread load across several deployments or regions when a single quota is the limit.


  • Stream responses to users; perceived latency matters more than total time.
  • Parallelise independent calls (asyncio.gather / Task.WhenAll), and let models emit parallel tool calls.
  • Cache deterministic work — classification, extraction, embeddings of unchanged content — keyed by a hash of prompt template, arguments and model. Exact-match caches are simple and safe; semantic caches need a similarity threshold and an eviction policy.
  • Trim context. Use chat-history reducers (truncation or summarisation), retrieve fewer but better chunks, and keep plugin descriptions short — every advertised function costs prompt tokens on every call.
  • Use the smallest model that works per task, routing by service_id / ServiceId.

  • Track token usage per function, tenant and model (from SK’s token metrics in .NET; from completion metadata in Python), and convert to cost using your provider’s current price list held in configuration (prices change too often to hard-code).
  • Set budgets per tenant and refuse or degrade (smaller model, shorter answers) when exceeded.
  • Watch for silent multipliers: retries, multi-agent rounds, large tool schemas, and verbose system prompts repeated on every turn.

LevelApproach
UnitTest native plugin functions as plain code. Test prompt functions and agents against a stub chat service (Python: subclass ChatCompletionClientBase; .NET: mock IChatCompletionService).
Filters and routingCall filters directly with a constructed context; routers and group-chat managers are deterministic and easy to unit test.
IntegrationRun against a real model in CI on a small, fixed set of cases; assert on structure (valid JSON, required fields, tool called) rather than exact wording.
EvaluationKeep a golden set of questions with reference answers; score relevance, groundedness and safety on each prompt or model change.
LoadExercise the real provider quota in a staging subscription; rate limits are the usual first failure.

  • Semantic Kernel pinned exactly; lock file committed
  • Secrets in a secret store; managed identity for Azure resources
  • Kernel built per request; services created once
  • Timeouts on model calls, tools and orchestrations; loop limits set
  • Filters for logging, redaction and approval of sensitive tools
  • OpenTelemetry exporting traces and metrics; sensitive diagnostics off
  • Conversation state in a shared store
  • Token budgets and alerts per tenant
  • Golden-set evaluation runs on prompt and model changes
  • Migration path to Microsoft Agent Framework noted for new features

DateVersionChanges
2026-10-21Py 1.45.0 / .NET 1.81.0Token-usage metrics corrected (.NET only; Python records usage from metadata). Security: template encoding, OpenAPI URL validation and the actual function-restriction APIs named.
2026-10-21Py 1.45.0 / .NET 1.81.0Rewritten as language-neutral guidance. The previous edition’s code used pre-1.0 Python APIs (invoke_async, create_function_from_prompt), wrong .NET signatures and hard-coded 2024 model prices; runnable code now lives in the language-specific production guides.