The agent says done. What does that actually mean?
A browser test repair shows why agent completion needs a precise finish line, evidence tied to the tested version, and an explicit handoff to human QA and release approval.
An isolated browser test suite at appointmed had started returning to the login screen instead of reaching the scenarios it was meant to exercise. Its test setup had fallen out of step with the application.
The repair reached the end of its automated run with quite a lot behind it. All 56 tests in the isolated suite passed. Nine scenarios passed against a deployed development environment. Independent reviews had approved the change. The staging deployment had succeeded, and its running version had been checked. The pull request for the production branch had green checks.
Human QA on staging was still pending. So was approval and the production merge.
The agent's assignment was complete at that point. The release still had work left.
That distinction is easy to lose in a final message. An agent can put a reassuring word at the top, list some passing checks, and leave me to reconstruct where the work actually stopped. Every individual statement might be accurate while the overall impression goes too far.
The useful artifact from this run was the handoff. It separated the local results from the deployed checks, identified the version they applied to, and named the human decisions still outstanding. That made the stopping point inspectable.
Give the assignment a finish line
In the first issue I wrote about moving implementation to agents while keeping engineering responsibility with humans. The awkward part is defining where one piece of that responsibility passes back.
Our delivery workflow gives the agent a substantial assignment. Implement the change, get it reviewed, validate it in development, prepare the production pull request, and deliver the staging build for human QA. The routine human gates are staging QA and the production merge.
That means stopping can be the correct behavior even when the larger job remains unfinished. The agent has reached an agreed boundary and prepared the next step.
It also means stopping after writing code would be too early for that assignment. A diff and a suggested test command would leave much of the authorized work undone.
I want that boundary defined before the run starts. If the finish line is a change ready for human QA, put that in the assignment along with the evidence expected there. Leaving it until the final message makes completion depend on the agent's interpretation of how much work feels sufficient.
Each check answers a particular question
The test repair makes this concrete because it involved two different kinds of evidence.
The isolated browser suite used controlled test fixtures. Passing it showed that those scenarios worked with that setup. Other scenarios ran against the deployed development environment. Those results supported a different claim about the integrated system.
Both were useful. The handoff kept them separate.
If I compress that into "end to end tests passed," I lose information I need for review. Which environment? Which scenarios? Did the test exercise a real integration, or did the fixture provide the response? A passing result becomes much easier to overread once its conditions disappear.
The same applies further along the path. A successful deployment tells me the deployment completed. Checking the running version establishes what arrived. A response from the site gives me another small piece of evidence. Human QA still has its own job.
I do not need every check to prove everything. I need the final claim to stay within what each check actually established.
That includes missing evidence. Our delivery policy explicitly distinguishes a check that could not run from one that passed. An unavailable environment or unusable fixture leaves a question unanswered. The report needs to preserve that gap so the next person can decide what it means.
Evidence belongs to a version
There is another condition attached to all those results: the code they describe.
A review approves a particular implementation. Tests exercise particular code. A deployed check observes the build that was running during the check. The connection between those things matters as much as the green status beside them.
Our workflow binds evidence to commit identifiers and deployment versions. It has explicit rules for deciding whether later changes require fresh verification. The final handoff needs to make that relationship understandable without asking me to replay the whole run.
Suppose an agent fixes something after a successful test run. The earlier result still describes the earlier code. Before carrying that result forward, we need to know whether the fix changed the behavior under test. A confident summary cannot answer that by itself.
This is ordinary release discipline, but agents make it particularly valuable. A run can move through implementation, review findings, repairs, integration, and another repair before I return to it. The conversation contains many true statements about versions that are no longer the final version.
The handoff should point me to the evidence that supports the change I am being asked to approve now.
Make the next decision cheap to understand
In the attention budget issue, I wrote about the cost of supervising several pieces of agent work. The final report is one of the places where that cost becomes visible.
If I have to read the entire conversation to find the deployment, work out which tests ran afterwards, and discover whether QA is still waiting, the agent has handed me a reconstruction task.
A useful handoff lets me start with the decision in front of me. For the test repair, that meant being able to see:
- What changed: the test setup and coverage, with no product behavior change.
- What supported it: the local checks, independent reviews, and deployed development results.
- What reached staging: a successful deployment with its running version verified.
- What remained at that handoff: human staging QA, then production approval and merge.
The underlying report had more detail, including links to the evidence. I want that detail available when something needs investigation. I also want the current state readable before I open those links.
There is a practical reason to include cleanup too. If a test creates temporary data or a run starts a local server, its ending state belongs in the handoff. Otherwise the agent can finish its feature assignment while leaving operational work behind for someone else to discover.
The amount of reporting should fit the task. A small copy edit needs much less evidence than a change crossing several environments. What matters is that the next person can understand the remaining work and act on it.
Write the handoff into the task
The starting point I would use is to define the final report alongside the acceptance criteria.
Ask the agent to explain the changed behavior, identify the version being handed over, link the checks that support it, and state the next action. Require it to name anything it could not verify. If there is a human boundary, make that boundary explicit in both the assignment and the report.
That also gives review something concrete to challenge. The reviewer can trace a claim in the summary back to the tested version and its supporting result. Any remaining action should be clear enough for someone who did not watch the run.
That is what I want an agent to mean when it says done: I can inspect what it delivered and make the next decision without guessing what happened.
How agent-ready is your team? Take the readiness check.
Get the build log