Building a Good AI Agent Harness in Rust

How to use this book

Build while reading. The fastest way to understand agents is to implement the loop yourself, then add one reliability layer at a time. Avoid starting with a large framework: frameworks make more sense after you understand the runtime they are hiding.

The most useful mental model is:

A stronger harness can make the same model dramatically more useful. Most of the engineering work in a dependable coding agent is ordinary software engineering: APIs, state machines, process control, validation, access control, event streams, file handling, persistence, and tests.

Observation and action spaces

The agent sees observations and chooses actions. Tool interfaces define those spaces. A coding agent does not directly “see your computer.” It sees structured slices exposed by tools.

A capable agent can begin with an almost embarrassingly small loop:

pub async fn run_agent(
    model: &dyn ModelProvider,
    tools: &ToolRegistry,
    mut state: AgentState,
) -> anyhow::Result<AgentOutcome> {
    for step in 0..state.max_steps {
        let request = state.context.build_request(&state)?;
        let response = model.complete(request).await?;

        state.record_model_response(&response);

        if response.tool_calls.is_empty() {
            return Ok(AgentOutcome::Finished {
                text: response.text.unwrap_or_default(),
                steps: step + 1,
            });
        }

        for call in response.tool_calls {
            let result = tools.execute(call, &state.runtime).await?;
            state.record_tool_result(result);
        }
    }

    Ok(AgentOutcome::Stopped {
        reason: StopReason::StepLimit,
    })
}

The loop needs explicit stopping rules

• final answer with no tool calls
• maximum model turns
• maximum wall-clock duration
• maximum token/cost budget
• fatal tool failure
• user cancellation
• permission denial

State machine view

Thinking in states makes cancellation, persistence, UI rendering, and recovery much easier than burying all logic in one function.

Starter dependencies

[dependencies]
tokio = { version = "1", features = ["full"] }
reqwest = { version = "0.12", features = ["json", "stream"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
async-trait = "0.1"
anyhow = "1"
thiserror = "2"
futures = "0.3"
tokio-stream = "0.1"
uuid = { version = "1", features = ["v4", "serde"] }
chrono = { version = "0.4", features = ["serde"] }
tracing = "0.1"
tracing-subscriber = "0.3"

Core domain types

#[derive(Clone, Debug, Serialize, Deserialize)]
pub enum Role {
    System,
    User,
    Assistant,
    Tool,
}

#[derive(Clone, Debug, Serialize, Deserialize)]
pub struct Message {
    pub role: Role,
    pub content: String,
    pub tool_call_id: Option<String>,
}

#[derive(Clone, Debug, Serialize, Deserialize)]
pub struct ToolCall {
    pub id: String,
    pub name: String,
    pub arguments: serde_json::Value,
}

Start with Ollama because it cleanly separates agent engineering from inference engineering. Your harness talks HTTP; Ollama handles model loading and GPU/runtime details.

#[async_trait::async_trait]
pub trait ModelProvider: Send + Sync {
    async fn complete(
        &self,
        request: ModelRequest,
    ) -> Result<ModelResponse, ModelError>;

    async fn stream(
        &self,
        request: ModelRequest,
    ) -> Result<ModelStream, ModelError>;
}

pub struct ModelRequest {
    pub messages: Vec<Message>,
    pub tools: Vec<ToolSpec>,
    pub temperature: Option<f32>,
    pub max_tokens: Option<u32>,
}

Why own the abstraction?

• Switch Ollama → llama.cpp → hosted API without rewriting the harness.
• Normalize tool calls and finish reasons.
• Centralize timeouts, retries, token accounting, and tracing.
• Test the agent with a fake deterministic model.

Do not over-generalize too early

Different model APIs expose different features. Keep a small common interface, and allow provider-specific capabilities through a capability struct instead of inventing an enormous universal trait.

pub struct ModelCapabilities {
    pub tool_calls: bool,
    pub parallel_tool_calls: bool,
    pub structured_output: bool,
    pub streaming: bool,
    pub max_context_tokens: usize,
}

Tools are contracts between a probabilistic caller and deterministic code. Design them more carefully than ordinary internal functions.

#[async_trait::async_trait]
pub trait Tool: Send + Sync {
    fn name(&self) -> &'static str;
    fn description(&self) -> &'static str;
    fn schema(&self) -> serde_json::Value;
    fn permission(&self) -> PermissionClass;

    async fn execute(
        &self,
        args: serde_json::Value,
        ctx: &ToolContext,
    ) -> Result<ToolOutput, ToolError>;
}

pub struct ToolRegistry {
    tools: HashMap<String, Arc<dyn Tool>>,
}

impl ToolRegistry {
    pub async fn execute(
        &self,
        call: ToolCall,
        ctx: &ToolContext,
    ) -> Result<ToolOutput, ToolError> {
        let tool = self.tools.get(&call.name)
            .ok_or_else(|| ToolError::UnknownTool(call.name.clone()))?;

        ctx.permissions.authorize(tool.permission(), &call).await?;

        tool.execute(call.arguments, ctx).await
    }
}

Validate arguments before execution

Never trust generated JSON simply because the model produced it. Deserialize into a typed Rust struct, enforce bounds, canonicalize paths, and reject unknown or contradictory fields.

#[derive(Deserialize)]
#[serde(deny_unknown_fields)]
struct ReadFileArgs {
    path: PathBuf,
    #[serde(default = "default_start")]
    start_line: usize,
    #[serde(default = "default_limit")]
    max_lines: usize,
}

Principles

• Distinct purpose: minimize ambiguous overlap.
• High-signal output: avoid dumping giant payloads into context.
• Pagination/ranges: every potentially large tool needs bounds.
• Useful errors: error messages should teach the agent how to recover.
• Stable naming: consistent verbs and namespaces.
• Descriptions are prompts: tell the model when to use the tool and when not to.

Design errors for recovery

{
  "ok": false,
  "error": {
    "kind": "ambiguous_match",
    "message": "old_text matched 3 locations",
    "recovery": "Read the surrounding lines and retry with a larger unique old_text block."
  }
}

Do not make the CLI the agent runtime. Make the runtime emit typed events; the CLI/TUI/web UI is merely one consumer.

#[derive(Clone, Debug, Serialize)]
pub enum AgentEvent {
    RunStarted { run_id: Uuid },
    TurnStarted { turn: usize },
    TextDelta { text: String },
    ToolRequested { call: ToolCall },
    PermissionRequested { request_id: Uuid, summary: String },
    ToolStarted { call_id: String },
    ToolOutputDelta { call_id: String, chunk: String },
    ToolFinished { call_id: String, result: ToolOutput },
    ContextCompacted { before: usize, after: usize },
    Warning { message: String },
    RunFinished { outcome: AgentOutcome },
}

Use a bounded Tokio channel so slow consumers create intentional backpressure rather than unbounded memory growth.

let (event_tx, event_rx) = tokio::sync::mpsc::channel::<AgentEvent>(256);

Benefits

• CLI today, web/SSE tomorrow.
• Tool stdout can stream while a command runs.
• Events become persistence and tracing inputs.
• Cancellation and approvals become clean asynchronous interactions.

The model does not reason over your entire application state. It reasons over the tokens you select for the next inference. Context engineering is therefore the process of constructing the smallest high-signal view that lets the model take the next good action.

Context layers

Token budget algorithm

pub struct ContextBudget {
    pub max_input_tokens: usize,
    pub reserve_for_output: usize,
    pub reserve_for_tools: usize,
}

impl ContextManager {
    pub fn build(&self, state: &AgentState) -> Result<Vec<Message>> {
        // 1. Add invariant instructions.
        // 2. Add durable task state.
        // 3. Add compacted history.
        // 4. Add recent turns from newest → oldest.
        // 5. Add only retrieved artifacts needed now.
        // 6. Stop before the budget is exceeded.
        todo!()
    }
}

Compaction converts verbose old history into a small state representation. It is not ordinary summarization: it must preserve details needed to continue work.

A useful compaction schema

## Objective
Implement reconnectable command execution.

## Constraints
- Rust
- Windows + macOS
- Existing public API must remain compatible

## Completed
- Added task registry
- Added command IDs
- Unit tests for lookup pass

## Important discoveries
- Process ownership currently lives in client.rs
- stdout reader exits when transport closes

## Files changed
- src/task.rs
- src/client.rs

## Open problems
- Decouple process lifetime from connection lifetime
- Add integration test for reconnect

## Last verified state
`cargo test` passes 41/43 tests.

Three different kinds of memory

The safest useful rule is: the model proposes; deterministic policy decides whether execution is permitted.

pub enum PermissionClass {
    ReadOnly,
    WorkspaceWrite,
    ProcessExecution,
    Network,
    Destructive,
    Privileged,
}

pub enum Decision {
    Allow,
    AskUser { reason: String },
    Deny { reason: String },
}

Policy dimensions

• tool name
• canonical path / workspace boundary
• command executable and arguments
• network host
• whether operation can destroy data
• whether secrets may be exposed
• user-approved session rules

Path containment

fn ensure_inside_workspace(
    workspace: &Path,
    requested: &Path,
) -> Result<PathBuf, ToolError> {
    let root = workspace.canonicalize()?;
    let candidate = root.join(requested).canonicalize()?;

    if !candidate.starts_with(&root) {
        return Err(ToolError::OutsideWorkspace(candidate));
    }

    Ok(candidate)
}

Sandboxing levels

Start with five excellent tools instead of twenty mediocre ones.

Safe exact-match edit

#[derive(Deserialize)]
struct EditArgs {
    path: PathBuf,
    old_text: String,
    new_text: String,
}

fn apply_exact_edit(source: &str, args: &EditArgs) -> Result<String, EditError> {
    let matches = source.match_indices(&args.old_text).count();

    match matches {
        0 => Err(EditError::NotFound),
        1 => Ok(source.replacen(&args.old_text, &args.new_text, 1)),
        n => Err(EditError::Ambiguous { matches: n }),
    }
}

Shell output must be bounded

Stream live output to the UI, but do not automatically feed unlimited stdout back to the model. Keep a ring buffer, truncate intelligently, and expose a way to request more.

pub struct CommandResult {
    pub exit_code: Option<i32>,
    pub stdout_tail: String,
    pub stderr_tail: String,
    pub truncated: bool,
    pub duration_ms: u64,
}

Separate agent state from environment state. Files on disk already persist; your harness should persist the metadata required to understand what happened and continue.

#[derive(Serialize, Deserialize)]
pub struct SessionState {
    pub session_id: Uuid,
    pub created_at: DateTime<Utc>,
    pub workspace: PathBuf,
    pub user_goal: String,
    pub messages: Vec<Message>,
    pub compacted_summary: Option<String>,
    pub plan: Option<Plan>,
    pub permission_grants: Vec<SessionGrant>,
}

Persistence progression

1. JSON file per session.
2. Append-only event log.
3. SQLite when you need indexing/search/concurrent metadata.

Checkpoint intentionally

Persist after model turns, after side-effecting tools, after compaction, and before waiting for human approval.

Planning is state, not magic

A plan can be an ordinary tool the model calls:

{
  "name": "update_plan",
  "arguments": {
    "items": [
      {"text": "Inspect failing tests", "status": "done"},
      {"text": "Find parser bug", "status": "in_progress"},
      {"text": "Implement fix", "status": "pending"},
      {"text": "Run full suite", "status": "pending"}
    ]
  }
}

Workflow vs agent

Subagents come later

A subagent is most useful when it gets a genuinely separable objective and its own small context. Good examples: repository exploration, test diagnosis, documentation research, or review.

Your own Tool trait should come first. MCP is then an adapter that can populate your registry from external tool servers.

When MCP helps

• you want interoperable external tools
• you do not own the integration
• you want servers to expose tools/resources dynamically

When RAG helps

Use retrieval when the information space is larger than the agent can inspect efficiently via direct tools. For a code repository, start with file search + structural discovery before jumping to embeddings.

Just-in-time retrieval

Capture a trace for every run and spans for every meaningful operation.

Record at least

• run/session/turn IDs
• model name and provider
• input/output token estimates
• latency
• tool call names and sanitized arguments
• tool durations and exit status
• context size before/after compaction
• permission decisions
• stop reason

Rust tracing

#[tracing::instrument(
    skip(model, request),
    fields(model = %model.name(), turn = turn)
)]
async fn run_model_turn(
    model: &dyn ModelProvider,
    request: ModelRequest,
    turn: usize,
) -> Result<ModelResponse> {
    let started = std::time::Instant::now();
    let response = model.complete(request).await?;
    tracing::info!(elapsed_ms = started.elapsed().as_millis(), "model turn completed");
    Ok(response)
}

Without evals, harness changes become vibes. A new prompt/tool may look smarter on one demo while making average performance worse.

Evaluate multiple layers

A tiny coding-agent eval case

name: fix_off_by_one
fixture: fixtures/off_by_one_repo
task: "Fix the failing range test without changing the test."
checks:
  - command: "cargo test"
    exit_code: 0
  - file_not_modified: "tests/range_test.rs"
budgets:
  max_turns: 12
  max_shell_calls: 5

Use deterministic checks first

Unit tests, exit codes, file diffs, schema validation, and exact expected artifacts are stronger than “LLM-as-judge.” Use model judging for subjective qualities that deterministic assertions cannot capture.

Classify failure before retrying

Idempotency matters

If a model request times out after a side effect was triggered, blindly retrying can duplicate writes or actions. Give side-effecting operations stable IDs when possible and persist execution status.

Cancellation

Propagate cancellation through model streaming, shell children, tool tasks, and pending approval waits.

tokio::select! {
    result = execute_tool(call) => result,
    _ = cancel.cancelled() => {
        Err(ToolError::Cancelled)
    }
    _ = tokio::time::sleep(timeout) => {
        Err(ToolError::Timeout)
    }
}

Parallelism

Parallelize only independent, side-effect-free work by default. Two read-only searches are easy; two file edits may race. Preserve model tool-call order when semantics are unclear.

Build vertically. Every milestone should leave you with a program you can run.

V1 definition of done

• Runs with a local Ollama model.
• Can inspect a Rust repository.
• Can edit files safely.
• Can run cargo test and react to failures.
• Streams model and command output.
• Requires approval for writes and risky commands.
• Stops on budgets and supports cancellation.
• Persists and resumes a session.
• Compacts long conversations.
• Produces a trace you can inspect afterward.
• Has at least 10 repeatable eval tasks.

System prompt starter

You are a software engineering agent operating inside a tool-based environment.

## Goal
Solve the user's task completely and verify the result.

## Working style
- Inspect before changing.
- Use tools to obtain facts instead of guessing.
- Prefer small, reversible edits.
- After modifying code, run the narrowest useful verification.
- Treat tool errors as observations and recover when possible.
- Do not claim success without evidence.

## Tool discipline
- Read only the file ranges you need.
- Search before reading large directories.
- Avoid repeated calls that return the same information.
- Never bypass permission or workspace restrictions.

## Completion
Finish when the task is satisfied and verified, or explain the blocking condition clearly.

What to postpone

Architecture Invariants Worth Keeping

1. The UI never directly executes model-requested side effects.
2. All tools go through one authorization/validation dispatch path.
3. Provider types do not leak through the core harness.
4. Every run has explicit budgets and cancellation.
5. Potentially large outputs are bounded.
6. Important transitions emit events.
7. Side effects are auditable.
8. Context construction is centralized and testable.
9. Expected tool failures are structured observations.
10. Harness changes are evaluated against repeatable tasks.

Further reading

• Anthropic — Building Effective Agents
• Anthropic — Effective Context Engineering for AI Agents
• Anthropic — Writing Effective Tools for Agents
• Anthropic — Effective Harnesses for Long-Running Agents
• OpenAI Agents SDK docs — useful as a reference for runtime primitives
• Ollama — local model runtime
• AI Agents in Depth — book you found

This guide deliberately teaches the runtime from first principles rather than prescribing an agent framework. The architecture is provider-neutral and can evolve from a local personal tool into a more isolated production-style harness.