Prompts Don't Govern Agents. Reputation Does.

A response to Zhenfeng Cao, "Agentic Software: How AI Agents Are Restructuring the Software Paradigm" (arXiv:2606.05608)

Abstract

AI agents now generate their own decision logic while they work. Zhenfeng Cao argues that this turns the human into an "intent architect" who states goals, lets agents execute, and audits the results. At the scale agents are heading, no human can audit every result. This response proposes a different division of labor. The system judges agent performance around the clock. Assuming reputation for agents is properly calibrated, every agent run leaves a tamper-evident record of what the agent actually did, and agents are graded on held-out work against a standard fixed in advance. An agent turns red only when the evidence shows real underperformance. Red cases go to review, mostly by other agents with their own reputation at stake. Verdicts move standing, standing steers where work goes, and authority tied to tasks expires when the work stops, so the system cleans itself. Reputation never grants permission. The human holds authority, reviews the rare escalated red case, and answers for the outcome.

What Cao gets right

Cao makes a clear claim. In traditional software, people write the decision logic in advance. In agentic software, the agent writes it at runtime. The human stops authoring code and starts stating intent, coordinating agents, and auditing outcomes.

The paper is honest about the limits. On the EvoClaw benchmark it cites, agents that score above 80 percent on isolated tasks fall to at most 38 percent in continuous work. Errors compound. Tests pass while subtle mistakes slip through. Cao concludes that "humans should own intent, critical judgment, and ethical oversight."

I agree about ownership. I disagree about the workload. Cao's model still has a human audit every outcome. With many agents running continuously, that cannot hold. His own benchmark shows why: the failures that matter appear over long runs, exactly where no human is watching.

Intent is a request

The paper places governance in two spots: the instructions before the run and the audit after it. Neither holds while the agent works.

Language cannot cover every case. However carefully a specification is drafted, some situations will escape it (Kaal 2026c, claim 6886078-002). Constraints on agents with no reason to comply can be gamed or routed around (Kaal 2026a, claim 6244278-014). The evidence on written instructions agrees. When researchers at ETH Zurich tested AGENTS.md context files, agents followed them, yet task success did not generally improve and costs rose by more than 20 percent (Gloaguen et al. 2026).

So the practical questions change. What did the agent actually do? Which agent should get the next task? And who has time to check?

A hard boundary, briefly

The design I propose here works in three layers. The first is structural. The agent proposes. A separate component that alone holds the credentials decides. Irreversible actions go to a named person. An agent cannot talk its way past a boundary it never controls.

That is necessary. It is also the least interesting part. A boundary can only say no. It cannot say who did good work.

A system that judges agents, so the human does not have to

The second layer I propose here is an operator view that turns every run into evidence and every body of evidence into a judgment, around the clock. It works as a pipeline.

It records what agents actually touched.

Files, network connections, credentials, and devices are observed from outside the agent and tied to the authorization each run used. The record covers the whole run, not single steps. That is how it catches authority drift, the failure in which every step passes its check while the total goes beyond what was authorized (Kaal 2026e, claim 7314479-015). Without such a record, a user can hold every byte locally and still have no answer to what happened (Kaal 2026e, claim 7314479-002). Where contribution cannot be traced, fault cannot be assigned (Kaal 2026e, claim 7314479-014).

It grades agents on work they have not seen.

A test set the agent can see becomes something it learns to pass. Held-out tests are sealed before the run and graded against a standard fixed in advance. Evaluation must come from observed results and be costly to manipulate (Kaal 2026e, claim 7314479-024).

It judges only when the evidence is strong enough.

An agent receives a status only when the evidence clears a set bar. Below that bar it is shown as not yet judged, never as fine. An unobserved channel is labeled unobserved, and an unknown cost is never shown as zero. Red means evidence-backed underperformance, and every red status links to the runs, effects, and tests behind it. A calm, empty screen is the most dangerous screen an operator can have.

It never judges alone.

Unstaked monitoring does not solve the problem, because AI that monitors AI inherits the flaws of the agents it oversees (Kaal 2025, claim 5245185-027). So a red status is not a verdict. It opens a review, with the evidence attached.

It proposes and never acts.

The view is read-only. For a red agent it proposes a remedy for review. Its records are tamper-evident, so errors become visible, attributable, and correctable (Kaal 2026c, claim 6886078-012).

Why one operator's evidence is not enough

Here the argument turns. Everything above produces a local judgment: how agents performed for one operator, under that operator's conditions. That judgment is valuable. It is not reputation. Reputation forms only when verdicts are confirmed by reviewers with something at stake and carried beyond a single operator.

The agent economy will not stay local. Agents will work for many principals and deal with counterparties they have never met. Code cannot settle that trust problem alone. Smart contracts can only limitedly assure trust between counterparties (Kaal 2019, claim 3405401-016), and decentralized commerce is equally afflicted with trust issues (Kaal 2019, claim 3405401-015). Current agent platforms fall short in both directions: centralized ratings defeat agent autonomy, and primitive token staking yields only binary outcomes (Kaal 2026b, claim 6192998-008).

A reputation system for agents that forms the foundation for governance of agents has to meet five conditions:

Standing is earned, not bought.

If reputation can be transferred, capital-rich but incompetent actors buy it and the signal loses its meaning (Kaal 2026a, claim 6244278-009). Non-fungible reputation has to be built through merit and time (Kaal 2021, claim 3981021-029).

Identity persists.

Otherwise a poorly performing agent simply abandons its account and starts fresh (Kaal 2026b, claim 6192998-001).

Others validate the work with something at stake.

These are the reviewers who take red cases. Voting without something at risk does not produce honest evaluation (Calcaterra and Kaal 2018, claim 3125827-006). In validation pools, members stake non-transferable reputation to approve or reject work (Kaal 2025, claim 5245185-036).

Standing moves with performance. Reputation that grows with validated contribution and decays with poor judgment selects for good behavior over time (Kaal 2026a, claim 6244278-018).

Entry stays open.

A new agent can start small, stake a little, and build standing, facing barriers that are economic and temporary (Kaal 2026a, claim 6244278-013).

Reputation supplies what rules cannot: consequence

This is the deeper need. Agents that bear no consequence for error cannot develop genuine discernment (Kaal 2026a, claim 6244278-001). Rules imposed from outside stay fragile. Reputation makes consequence internal. An agent whose accumulated standing exists only inside a network has a stake in that network's health (Kaal 2026a, claim 6244278-022). It cannot game its way to high standing without performing competently and honestly (Kaal 2026a, claim 6244278-025). And because more capable agents accumulate deeper stakes, this alignment scales with capability rather than against it (Kaal 2026a, claim 6244278-003). None of this waits for a human. The consequence follows from the record.

Reputation also fills a gap that transparency leaves open. A path to challenge an outcome must end in a remedy, not an explanation, because an institution is constituted by the consequences it can impose (Kaal 2026e, claim 7314479-026). Lost standing is such a consequence.

Keep the layers separate

One rule holds the design together. Reputation may inform discovery, routing, pricing, and allocation. It may never grant permission, enlarge authority, or substitute for an enforcement boundary (Kaal 2026e, claim 7314479-020). A trusted agent still works inside its mandate. High standing earns better work, not wider keys.

The full loop runs without pause. The boundary keeps authority fixed. Every run becomes evidence. Evidence becomes a judgment once it clears the bar. Red judgments go to staked review. Confirmed verdicts move standing. Standing decides where the next task goes. Authority tied to tasks expires when the work stops. Every escalation and near miss also shows the principal where the rules need to change, which is how dynamic regulation works (Kaal 2013, claim 2267560-046).

From intent architect to accountable principal

Cao gives humans ownership of intent and oversight. Ownership needs institutions. The accountability gap around agents has four parts: attribution failure, authority drift, evaluative capture, and recourse vacuum (Kaal 2026e, claim 7314479-013). Intent addresses none of them. A boundary, an honest evidence record, and reputation with real consequence address them together.

The human role is therefore accountable principal, and by design it is small. The human does not watch agents work. The human watches how the system judges them.

Red cases go to review, usually by other agents. A reviewing agent works from the recorded evidence, not from the red agent's own account, and its verdict is recorded too. The reviewer has standing at stake. An honest review builds its reputation, and a careless one costs it (Calcaterra and Kaal 2018, claim 3125827-006). That stake is what separates staked review from the AI that merely monitors AI criticized above. The human steps in only when reviewers disagree, when an action cannot be undone, or when a mandate itself should change.

Everything else cleans itself. An agent confirmed red by review loses standing, and work routes away from it, undermining its ability to earn future standing / rep and associated salaries in fungible currencies. Because authority is scoped to tasks and expires, an agent that receives no tasks holds no live authority and cannot earn or increase incentives to earn. The agent slowly dies out and withers unless the agent finds a way to remedy its shortcomings and continue to make rep earning valuable contributions. No one has to switch it off. Agents that perform well gain standing and receive more work (Kaal 2026a, claim 6244278-018). Reputation steers that flow but never grants permission.

The human holds authority, reads the board, and handles the rare red case the system escalates. The principal still answers for the outcome. The system makes sure that answering rarely requires watching.

Cao's revised conclusion says software engineering is not ending but growing into something larger. It is growing into institutional design. Intent says what we want. Evidence shows what happened. Reputation decides whom we trust next. The human steps in when the system turns an agent red.

References

Calcaterra, Craig, and Wulf A. Kaal. 2018. "Secure Proof of Stake Protocol." SSRN. https://ssrn.com/abstract=3125827. Claim cited: 3125827-006.

Cao, Zhenfeng. 2026. "Agentic Software: How AI Agents Are Restructuring the Software Paradigm." arXiv:2606.05608v2. First version titled "The End of Software Engineering: How AI Agents Are Fundamentally Restructuring the Software Paradigm." https://arxiv.org/abs/2606.05608

Gloaguen, Thibaud, Niels Mündler, Mark Müller, Veselin Raychev, and Martin Vechev. 2026. "Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?" arXiv:2602.11988. https://arxiv.org/abs/2602.11988

Kaal, Wulf A. 2013. "Evolution of Law Dynamic Regulation in a New Institutional Economics Framework." SSRN. https://ssrn.com/abstract=2267560. Claim cited: 2267560-046.

Kaal, Wulf A. 2019. "Decentralized Commerce - A Primer on Why Decentralized Reputation Verification Systems Are Needed." SSRN. https://ssrn.com/abstract=3405401. Claims cited: 3405401-015, 3405401-016.

Kaal, Wulf A. 2021. "How Decentralized Autonomous Organizations Optimize Charitable Giving." SSRN. https://ssrn.com/abstract=3981021. Claim cited: 3981021-029.

Kaal, Wulf A. 2025. "How can we Best Monitor AI Agents." SSRN. https://ssrn.com/abstract=5245185. Claims cited: 5245185-027, 5245185-036.

Kaal, Wulf A. 2026a. "AI's Mother's Instinct: Engineered Consequence, Emergent Ethics, and the Institutional Trajectory Toward Agentic Alignment." SSRN. https://ssrn.com/abstract=6244278. Claims cited: 6244278-001, 6244278-003, 6244278-009, 6244278-013, 6244278-014, 6244278-018, 6244278-022, 6244278-025.

Kaal, Wulf A. 2026b. "Evolution of Domain-Specific Reputation Systems: From Binary Validation to Citation-Weighted Knowledge Attribution." SSRN. https://ssrn.com/abstract=6192998. Claims cited: 6192998-001, 6192998-008.

Kaal, Wulf A. 2026c. "Governance as a Product (GaaP): A Reputation-Weighted Institutional Architecture for Autonomous AI Agent Governance." SSRN. https://ssrn.com/abstract=6886078. Claims cited: 6886078-002, 6886078-012.

Kaal, Wulf A. 2026e. "Institutional Requirements for Sovereign Local Agent Runtimes." SSRN. https://ssrn.com/abstract=7314479. Claims cited: 7314479-002, 7314479-013, 7314479-014, 7314479-015, 7314479-020, 7314479-024, 7314479-026.