Agent Harness Evaluation Scorecard: How to Compare Open-Source Runners

The seven evaluation dimensions

Compare harnesses across installation, tool support, workspace files, memory, permissions, observability, and handoff. A harness can be exciting and still fail a business workflow if it lacks logs or approval gates.

Weight the scorecard by risk. A research-only workflow can tolerate more rough edges than a checkout, email, ad budget, or customer support workflow.

Questions to ask before installing

Can the harness run locally or self-hosted? Can you inspect tool calls? Can you disable risky tools? Does it support MCP or equivalent tool adapters? Can it preserve task state without leaking secrets into memory?

Does the project have active commits, issue responses, clear docs, and a security model? Open source does not help if nobody can understand or maintain the implementation.

How to test safely

Run a dry workflow with fake accounts, sample data, and read-only tools. Ask the agent to produce a handoff report that explains what it did, what it did not do, and what it needs from the buyer.

Only after the dry run passes should you connect real tools, and even then start with draft-only or read-only permissions.

FAQ