Field notes from working through example 16 of Ardan Labs’ Ultimate AI course by Bill Kennedy and Florin Pățan (Apache 2.0). My fork: PratikDhanave/ai-training. Thank you Bill and Florin for teaching this material — the patterns in this post are derived from the course; the production reflections at the end are mine.

What the example teaches

The naive tool execution:

result := tool.Run(args)

The hardened version:

func runToolSafely(ctx context.Context, tool Tool, args any) (result any, err error) {
    defer func() {
        if r := recover(); r != nil {
            err = fmt.Errorf("tool %s panicked: %v\n%s", tool.Name, r, debug.Stack())
        }
    }()

    ctx, cancel := context.WithTimeout(ctx, tool.MaxDuration)
    defer cancel()

    return tool.Run(ctx, args)
}

Plus structured error wrapping so the agent can decide whether to retry, escalate, or give up.

What it looks like

The full pattern wraps recover + timeout + error classification. The agent loop sees (result, err); whether the err is ErrTimeout, ErrPanic, or ErrToolRefused decides what happens next.

What I learned

The recover() is non-negotiable. One bad tool implementation panics; without recover the entire agent loop crashes, taking the conversation with it. Per-tool wrapping isolates the blast radius.

Per-tool timeout, not per-loop timeout. A loop-level timeout means a slow tool eats the budget for every other tool in the loop. Per-tool means the slow one fails, the others continue.

Production connection

Genie’s pkg/orchestration/orchestrator.go runs every agent in a similar wrapper. We added a circuit-breaker layer on top: a tool that fails 5× in 60 seconds gets short-circuited automatically. Pattern from this example; circuit breaker from the LLM wrapper work (also a separate post in this series later in the roadmap).

Credit & reference. This post is field notes on example 16 from Ardan Labs’ Ultimate AI by Bill Kennedy + Florin Pățan, licensed Apache 2.0. The original example: cmd/examples/example16-tool-hardening/. My fork with notes: PratikDhanave/ai-training. Highly recommend the course for anyone building AI applications in Go — the material is rigorous and the Kronk + yzma + llama.cpp pipeline gives you hardware-accelerated local inference end-to-end. Thank you, Bill and Florin.