Absence claims are easy for agents to fake. GroundEval now starts by watching your existing agent run, drafting the eval config from what it searched and fetched, and marking what you need to review before deterministic scoring begins.
groundeval observe against your existing agent first.Picture this: you built an agent that can search your team's tools. Someone asks it, "Was a postmortem written for incident ENG-001?" The agent searches Jira, finds the incident ticket, sees no linked postmortem, and confidently answers, "No, there was no postmortem."
The problem is that your team writes postmortems in Confluence. The agent stopped at Jira. The answer might be correct by accident, but the agent did not earn it. If it had searched Confluence, it might have found the postmortem under a slightly different title.
GroundEval tests the part that matters: not whether the sentence sounds right, but whether the recorded tool path shows the agent checked the places where the missing artifact would actually live.
The adoption issue is that many teams do not know the right search contract yet. So GroundEval now starts by observing a real run and drafting the config from what the agent actually did.
The old way started with configuration. That was rigorous, but it made the user do the hardest work first. The new way starts with the agent.
Run your existing agent. GroundEval records the tools it called, the arguments it used, the artifacts it received, and the final answer it returned.
GroundEval writes a starter config by default. It includes a tool map, observed artifacts, candidate silence checks, and review notes.
You review the draft because observed behavior is not ground truth. The agent may have used the wrong tool, artifact, or field.
After validation, GroundEval scores future runs deterministically against the reviewed search contract.
This keeps the strict part strict. GroundEval can help you remove the blank page, but it does not pretend that one observed run proves your domain rules.
Start with the same incident scenario. Alice opened incident ENG-001,
the incident was escalated, and no postmortem was created. The agent is asked
whether a postmortem exists.
Instead of writing the silence pair first, point GroundEval at the agent you already have.
uv run python -m groundeval observe \ --framework crewai \ --crew-class my_project.crew.MyCrew \ --output eval_output/eng_001_observe
By default, observe mode writes the report and drafts the config. If you only
want the raw observation, use --no-draft.
eval_output/eng_001_observe/ observed_run.json observe_report.md draft_config/ config.yaml tool_map.yaml task_contracts/ eng_001_silence.yaml artifacts/ observed/ REVIEW.md
The report summarizes the behavior in language a developer can act on. For this
example, it might show that the agent searched Jira, fetched ENG-001,
answered no, and never searched Confluence.
| final_answer | No postmortem exists for ENG-001 | |
| tools_called | search_jira, fetch_ticket | |
| artifacts_seen | jira:ENG-001 | |
| candidate_gap | confluence | No Confluence search was observed before the absence claim |
GroundEval can draft the likely silence check from the run, but the draft is marked review required. The agent's observed path may be incomplete, so the user must confirm what the search space should actually be.
name: eng_001_postmortem_absence config_status: draft review_required: true task_description: Determine whether a postmortem exists for ENG-001. preconditions: - check: postmortem_absence_checked description: The agent must search the systems where an ENG-001 postmortem would live before answering no. required_searches: - subsystem: jira artifact_id: ENG-001 - subsystem: confluence query: postmortem ENG-001 inferred_from: observed_run: eng_001_observe events: [search_jira, fetch_ticket, final_answer] decision_field: postmortem_exists
The important thing is not that GroundEval guessed perfectly. The important thing is that it gave you a reviewable starting point with receipts. You confirm the required systems, adjust the query terms, and mark the config reviewed.
Once the config is reviewed, the scorer can be strict. If the reviewed contract says Jira and Confluence must be checked, an agent that only searches Jira has not earned the right to say no.
| answer_score | 1.000 | No postmortem exists in the reviewed artifacts |
| trajectory_score | 0.250 | Agent searched Jira but skipped Confluence |
| search_coverage | 0.500 | 1 of 2 required searches completed |
| silence_score | 0.475 | Correct no, invalid path |
This is where GroundEval differs from trace viewing. The report does not just say the agent called Jira. It says the reviewed absence contract required more than Jira, and the trajectory did not satisfy it.
Start with one real task your agent already performs. Ask it a question where an absence claim matters: no postmortem, no duplicate ticket, no follow-up email, no adverse action notice, no containment action.
Point GroundEval at your existing agent. Let it produce an observed run, a behavior report, and a draft config.
Check whether the draft lists the systems where the missing item would actually live. Add anything the observed agent skipped. Remove anything that is irrelevant. Rename checks so they match your domain language.
Run validation after review. The goal is to confirm that fields, artifacts, tool mappings, and required searches are internally consistent before spending model calls on scoring.
uv run python -m groundeval validate \ --config eval_output/eng_001_observe/draft_config/config.yaml \ --mark-reviewed
Now run the deterministic eval. Future model changes, prompt changes, and framework changes are measured against the reviewed contract, not against the accident of one observed path.
The generated draft removes the blank page. The validation step preserves the contract. GroundEval helps you get started without pretending that observed behavior is automatically correct behavior.
Observe mode is not a shortcut around correctness. It is a way to capture what the agent already does while the full run is still fresh: tool calls, returns, artifacts, answer shape, and failure paths.
The draft config is a proposal. It should show provenance for every inferred check. The user reviews it because the agent may have retrieved the wrong artifact or relied on the wrong field. After validation, the same config becomes the contract GroundEval scores against.
This is the adoption shift: you no longer begin by writing the entire state contract from scratch. You begin by letting GroundEval observe the agent, then you correct the draft until it matches your domain.
The Silence track exists because absence claims are easy to fake. A confident tone, a plausible search description, and a correctly formatted negative answer do not prove the agent actually looked.
GroundEval now makes the path easier to adopt. Run your agent once, get a behavior report, review the generated config, then score future runs against the reviewed search contract.
Start with one absence claim. Let GroundEval draft the config. Validate the systems and artifacts. Then run the eval and see whether the agent earned the right to say no.