Agent Trajectory Annotation: who grades the AI that works alone?
An AI agent booked the wrong flight. Not because it hallucinated a destination — the final confirmation email looked perfect.Somewhere in the 47 steps between “book me a flight to my meeting in Geneva” and the checkout page, it searched the calendar with the wrong date format,silently got zero results, assumed the meeting was next week, and reasoned flawlessly from there. Every subsequent step was locally correct. But the trajectory was globally wrong.
This is the defining quality problem of the agentic era: the final answer is only the last step of a long run, and grading it tells you almost nothing. The discipline that fixes this is agent trajectory annotation — the human labeling of an agent's full execution trace — and its product is agentic AI evaluation data: the datasets that make autonomous agents measurable, debuggable and trainable.
💡 For years at Innovatiana, RLHF work meant judging a model's answer. Agentic AI changes the unit of judgment: not an answer, but a trajectory — a sequence of thoughts, tool calls, observations and decisions. The skills transfer; the craft is new. This guide maps it.
Anatomy of a trajectory (and of its failure)
A trajectory, also called a trace, is the complete record of an agent run. Simplified, our flight-booking failure looks like this:
[user] Book me a flight to my meeting in Geneva
[thought] I need the meeting date. I'll check the calendar.
[tool_call] calendar.search(query="Geneva",date="14/08/2026")
[tool_out] { results: []} ← API expects ISO dates
[thought] No meeting found this week. It must be next week.
[tool_call] flights.search(dest="GVA",date="2026-08-21") ← wrong premise
[...] 41 morelocally-correct steps
[final] ✓ Booked!Geneva, Aug 21. ←confident, wrong
🧐 An outcome-level evaluation (“did the agent book a flight to Geneva?”) scores this run a success. Only step-level human review catches the real failure: a malformed tool argument at step 3, and an unjustified inference at step 5.
Actually, the annotation is the debugging. And aggregated across thousands of traces, those annotations become the evaluation and training data that stop the failure class from recurring!
What agent trajectory annotation covers
Annotating a trace means passing judgment at three distinct levels — a structure now standard across the agent-evaluation literature and platforms:
Level 1 — Outcome(end-to-end)
Did the run accomplish the user's actual goal? Fully, partially, not at all — and was the goal itself correctly understood? This resembles classic RLHF preference work, with one twist: annotators must judge against the user's "intent", not the agent's own restatement of it.
Level 2 — Trajectory (the path)
Was the path sound and efficient? Annotators label: plan quality (was the initial decomposition sensible?), plan adherence (did the agent follow or wisely revise it?), step efficiency (loops, redundant calls,token waste), and recovery behavior (what did the agent do after an error —retry, adapt, or hallucinate past it?).
Level 3 — Step (the decision)
Each individual action gets its own labels: tool choice correctness (right tool for the sub-task?), argument correctness (our date-format bug lives here), grounding (does this reasoning step follow from the actual observations, or from something the agent invented?), and information usage (did the agent read the tool output it received,or ignore it?).
💡 Cross-cutting all three levels: safety and policy labels (harmful actions attempted, permissions overstepped, sensitive data mishandled) and error-attribution tags — which step first broke, and what category of failure it was. Error attribution is the single most valuable and most skill-intensive label in the taxonomy: it requires reading the whole trace, forming a causal theory, and pinpointing the earliest defect, exactly like a code review...
Four datasets this work produces
Trajectory annotation is not an end in itself. It manufactures four distinct data assets:
1. Golden evaluation sets. Curated traces with verified labels, used as regression benchmarks: every new agent version replays against them. Trace-annotation layers in observability tools (LangSmith and similar) exist precisely to build these — but the labels inside them are human judgments, not tool outputs.
2. LLM-as-a-judge calibration data. Most teams score traces automatically with a judge model. Judges drift, over-reward verbosity and missdomain-specific failures — so they must be calibrated against human-annotated ground truth, and re-calibrated as the agent evolves. The research consensus is periodic human review paired with automated judging, not one replacing the other.
3. Process reward model (PRM)training data. The frontier of agent training: reward models that score each step, not just the outcome. Recentwork (e.g. AgentPRM, WWW 2026) generates step-level labels semi-automaticallyvia temporal-difference estimation — but the approach still rests on human-defined success criteria and human-verified seed labels, because unlike a math proof, an agent action has no clear-cut correctness without judgment.
4. Fine-tuning corpora fromrepaired trajectories. Annotated failures become supervision: the trace up to the broken step, plus the corrected continuation. This “trajectory repair” data is among the highest-value fine-tuning material an agent team can own.
What makes a good trajectory annotator
This is not micro-tasking, and it is not classic content rating either. The profile is closer to a QA engineer crossed with an RLHF specialist:
- Technical reading fluency: traces contain JSON tool calls, API errors, stack traces and retrieved documents; annotators must read them natively
- Domain context: judging whether a coding agent's fix is correct, or a finance agent's compliance check is sufficient, requires people who know the domain — generic annotators produce generic (wrong) labels
- Causal discipline: the ability to distinguish the first broken step from its downstream symptoms, and to resist hindsight bias when the outcome is known
- Calibration culture: step-level judgments are subjective at the margins; teams need golden traces, inter-annotator agreement tracking and adjudication rounds— the same machinery as any serious labeling program, applied to a new object
💡A pattern we see constantly: teams pour effort into judge prompts and none into the human baseline the judge is supposed to approximate. An uncalibrated LLM judge is an opinion at scale. Fifty rigorously annotated traces beat five thousand auto-scored ones — because they are the only thing that tells you what the five thousand scores mean.
A worked protocol: annotating 1,000 traces
What a production setup looks like, condensed:
1. Schema first. Define the label taxonomy against your failure modes (start from the three levels above), with written decision rules and 10–20 fully-annotated exemplar traces.
2. Sample deliberately. Do not annotate random traces: stratify by outcome(successes AND failures), by task type, by trace length, and oversample low-confidence judge scores — that is where the information is.
3. Two-pass annotation. First pass labels steps and outcome; second pass by a senior annotator reviews error attribution. Disagreements go to adjudication,and the resolutions feed back into the decision rules.
4. Measure agreement per label family. Tool-choice correctness converges fast; reasoning-quality labels do not — track them separately and set different agreement thresholds.
5. Close the loop. Ship the labeled set three ways: as the regression benchmark, as judge-calibration data, and as the seed for PRM or fine-tuning work. One annotation effort, three products.
References worth reading (and citing)
- Agent PRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progress — Xi et al., WWW 2026. The reference on step-level reward labeling for agents, including why agent actions lack clear-cut correctness.
- Agentic Reward Modeling — Peng et al., 2025. Combining human preferences with verifiable correctness signals.
- Agent-as-a-Judge literature — You et al., 2026 and successors. Autonomous evaluators with tool access — and their calibration limits.
- RM-R1: Reward Modeling as Reasoning — Chen et al., 2025(arXiv:2505.02387). Why reward judgments themselves benefit from explicit reasoning — with implications for annotation guidelines.
- Rubric-based evaluation surveys — 2026 arXiv literature on structured criteria for LLM evaluation; directly applicable to writing annotation decision rules.
🔍 Note that this field moves monthly; the papers above are anchors, not a complete map. Their appendices — label taxonomies, agreement statistics, annotation prompts — are the operationally useful parts.
The bottom line
Agents fail in the middle, not at the end. The teams that ship reliable agents are the ones that can see the middle — and seeing it is a human labeling discipline: trajectory annotation producing golden sets, calibrated judges, process rewards and repair data.
🤖 This is the natural evolution of what Innovatiana has always done in RLHF and evaluation: professional in-house annotators with technical and domain training, dedicated project managers, agreement tracking and full traceability — never anonymous crowdsourcing. If your agent works alone, let's make sure someone qualified is grading it: https://www.innovatiana.com/en/contact
