Advanced framework guide

Advanced Haystack

Scale Haystack from static tool demos to governed agent workloads with semantic selection, explicit LLM-call policy, tracing, and deployment boundaries.

← Basic Haystack usage

Advanced page: dynamic tool selection

Before each Haystack run, ask Gantry for a context-aware subset of tools. Combine request metadata, tenant policy, semantic similarity, and reranker results so the agent sees a concise, relevant tool surface.

candidate_tools = gantry.select_tools(
    query=user_message,
    tags=["support"],
    limit=8,
)
framework_tools = [HaystackAdapter.convert(spec) for spec in candidate_tools]

LLM calls and model policy

Centralize LLM calls when Haystack supports custom model clients; otherwise add middleware around the framework call. The policy should select the provider, enforce budget ceilings, redact sensitive fields, emit token metrics, and decide whether tool results should be summarized before returning to the model.

response = llm_policy.complete(
    framework="Haystack",
    model="primary-or-fallback-model",
    messages=messages,
    tools=framework_tools,
    trace_id=trace_id,
)

Production patterns

• Observability: attach trace IDs to every Haystack run, LLM call, and tool execution.
• Safety: keep high-risk tools behind explicit allowlists, human approval, or dry-run executors.
• Resilience: add provider fallback for LLM calls and executor fallback for remote MCP/A2A tools.
• Evaluation: replay representative prompts with fixed tool snapshots before broadening dynamic retrieval.