Agent builders do not need another blank config. They need to run the agent they already have, see what it actually did, and turn that behavior into a reviewable eval policy.
Agent evaluation usually starts with a blank page. Define a dataset. Write an expected trajectory. Choose a judge. Create a rubric. Map your tools. Decide what counts as success. Then run the agent.
That order makes sense if the team already understands the agent's behavior. Most teams do not. They know the agent returns an answer. They may not know which tools it called, which arguments it passed, which artifacts it used, which searches it skipped, or whether the final answer depended on evidence that was never fetched.
Observe first flips the order. The user does not begin by declaring the perfect evaluation. They begin by running the agent they already have. GroundEval watches the run, records the trajectory, and drafts the starting policy from what actually happened.
The first product question is not "Did this score pass?" It is "What did this agent actually do?"
The public feedback around agent frameworks points in the same direction. Builders are not only asking whether an answer was good. They are asking whether the agent really used the tool, whether the tool call had the right arguments, whether the path changed after a prompt update, and whether the observed trace explains the failure.
A CrewAI issue describes an agent generating a fake Observation instead of executing the tool, with no trace of tool usage in logs or Phoenix.
CrewAI issue →LangSmith's trajectory evals compare the exact sequence of messages and tool calls using trajectory matching or an LLM judge.
LangSmith docs →Google ADK evals focus on golden datasets, evaluation criteria, automated checks, and catching regressions early.
ADK codelab →The OpenAI community has discussed pytest-style behavioral tests for tool usage, tool order, step limits, unsupported success claims, and regression baselines.
OpenAI community →The pattern is clear. The market already knows final answers are too thin. The next question is what to do with the behavior once it is visible.
Traces are useful, but a trace is still raw material. A trace can tell you that the agent called fetch_customer, then search_tickets, then fetch_policy. It does not automatically tell you which of those calls were required, which artifact defined ground truth, which role was allowed to see which subsystem, or whether the final decision depended on a missing precondition.
Golden datasets and expected tool trajectories help, but they create another setup burden. The user still has to decide what the golden path is before seeing how the agent behaves in practice.
GroundEval's observe first model uses the run itself to remove the blank page.
| Current eval surface | What it gives you | What GroundEval adds |
|---|---|---|
| Tracing | Events, spans, tool calls, model turns | Draft config and review checklist from the observed run |
| Trajectory matching | Expected path compared to actual path | Evidence path validity against preconditions and artifacts |
| Golden datasets | Known examples for regression testing | A starter policy generated from real behavior, then reviewed |
| Tool call accuracy | Tool name, order, and parameter checks | Whether the returned artifact actually supports the decision |
| Output scoring | Whether the final response meets a metric | Whether the decision was supported, complete, and allowed |
The new GroundEval workflow is deliberately simple. The user runs the agent once. GroundEval observes by default and drafts the eval assets during that same run.
$ groundeval observe --framework crewai --crew-class my_project.crew.MyCrew ✓ ran agent ✓ recorded trajectory ✓ drafted config ✓ wrote review checklist ✓ wrote observe report
Drafting is on by default because the runtime already has the richest view of the run in memory: tool names, arguments, returned fields, artifacts, task context, final answer, and framework metadata. Users who only want the observation report can turn drafting off.
$ groundeval observe --framework crewai --crew-class my_project.crew.MyCrew --no-draft
The default output is a working starting point, not a certified truth file.
Observe first only works if the product is honest about the boundary. Observed behavior is not ground truth. The agent might have called the wrong tool. It might have retrieved the wrong artifact. It might have skipped a required search. It might have used a field that should not define correctness.
So GroundEval marks semantic inferences as review required. It can draft a likely task policy, but the user validates the policy before deterministic scoring begins.
Framework, observed tools, tool names, argument names, output path, raw trajectory, and observed artifacts.
Preconditions, ground truth fields, role permissions, action validity, artifact authority, and temporal cutoffs.
groundeval: config_status: draft generated_from_observation: true reviewed: false task_policys: - name: verify_escalation_readiness review_required: true preconditions: - check: customer_is_enterprise required_facts: [plan_tier] ground_truth_field: crm_account.plan_tier inferred_from: [fetch_customer, final_answer]
The user can validate the draft, edit it, and then run the deterministic scorer. That keeps onboarding easy without turning inferred behavior into unquestioned truth.
The observe report is useful even before formal scoring. It tells the user what the agent did in a language closer to evaluation than raw logs.
Which tools were called, what arguments were passed, and whether each tool behaved like search, fetch, or action.
Which artifacts were retrieved, which fields appeared decision relevant, and which artifacts were mentioned in the final answer.
Which checks appear to control the decision, with provenance showing which events caused each inference.
What the user must confirm before the config can be treated as a deterministic state policy.
This is the adoption win: GroundEval gives the user a useful artifact before asking them to become an eval expert.
The old path made the user author the eval before they could see the agent's behavior. The new path lets the user review a concrete draft based on a real run.
This also makes framework support easier. A plugin does not need to implement the entire scoring model on day one. It only needs to observe enough behavior to produce a useful report and a useful draft. Full gating and scoring can follow.
The current space helps teams observe, compare, and regression test agent behavior. GroundEval uses those same signals as the starting point for a stricter workflow.
The claim is narrow:
GroundEval helps you turn real agent behavior into a reviewed deterministic policy, then scores whether future decisions are evidence supported, complete, and allowed.
That means the product does not need to cover every evaluation use case. It owns the slice where agent correctness depends on tools, artifacts, preconditions, permissions, and trace evidence.
Agent builders already have traces, expected tool trajectories, golden datasets, and scorers. What they still need is a path from raw behavior to a state policy.
Observe first is that path. GroundEval watches the run, drafts the config by default, marks what needs review, and only scores once the user has validated what correctness means.
That is how evaluation becomes approachable without becoming loose. The first run shows the behavior. The draft removes the blank page. The review protects the truth boundary. The scorer measures the thing that matters: whether the agent's decision was justified by valid evidence.