@ruah-dev/eval
Eval
Same task — which executor wins, and what did it cost?
A fast micro-benchmark harness for AI coding agents. Run one task specification across multiple executors in fresh, isolated sandboxes with deterministic acceptance criteria and cost scorecards.
See it
Quick start
Up and running in seconds
@ruah-dev/eval
$ ruah eval run spec.json --runs 5
# mock-pass 1.0 mock-fail 0.0Features
01Ephemeral sandbox isolation per run (/tmp/ruah-sandbox-*)
02Multi-executor orchestration (Claude Code, Codex, OpenCode, Scripts)
03Deterministic criteria referee (powered by Ruah Verify)
04Multi-run variance analysis (--runs N with median & spread)
05Exportable scorecards in JSON, Markdown, and self-contained HTML
06Real cost-per-successful-delivery calculations
N
Executors
tmp
Sandbox
md|html
Scorecard
Command Reference
Commands
ruah eval run <spec.json>Execute benchmark spec across all configured executorsruah eval run <spec.json> --runs 5Run multiple iterations for statistical varianceruah eval compare a.json b.jsonDiff two benchmark scorecards side-by-sideruah eval report <scorecard.json>Render scorecard as markdown or shareable HTMLEcosystem
Part of the ruah toolchain
CLI Router
@ruah-dev/cli
One command. Every agent tool.
Guard
@ruah-dev/guard
Can I trust the agent not to do damage — in any harness, with proof?
Verify
@ruah-dev/verify
Did the agent actually finish?
Optimizer
@ruah-dev/opt
Where did my tokens go?
Watch (Lite)
@ruah-dev/watch
Can I show someone what this session did?
Converter
@ruah-dev/conv
How do I make this API agent-sized?
Orchestrator
@ruah-dev/orch
Parallel agents. Zero collisions.