Files
opencode-power-pack/evals/behavioral/README.md
T
Wayner Barrios 81a7048aa6 Address whole-branch review findings
- write reports and snapshots atomically, cleaning temp files on failure
- isolate CLI run tests from repository artifacts via env overrides
- cover version-detection failures, unsupported modes, manifest bounds,
  redaction and grading edges, and malformed mutations with tests
- document path overrides and completed plan steps
2026-07-31 17:28:13 -04:00

1.4 KiB

Behavioral Evaluations

These evaluations are contributor infrastructure, not a guarantee that every model follows every workflow.

Offline Replay

Run npm test. Replay requires no model, network, or credentials and rejects stale skill or case hashes.

Live Run

Run OPENCODE_EVAL_MODEL=provider/model npm run eval:behavioral from a source checkout. The runner uses an empty temporary project, denies model-facing tools, and writes only a redacted report to .artifacts/behavioral-evals/latest.json.

Review And Acceptance

Review the report for complete case coverage, passing oracles, and redaction. Then run npm run eval:behavioral:accept. Acceptance is explicit, deterministic, and unavailable for failed, incomplete, stale, duplicate, or partial reports.

Path Overrides

Tests and scripted runs isolate evaluation output with BEHAVIORAL_EVAL_LATEST_PATH (default .artifacts/behavioral-evals/latest.json) and BEHAVIORAL_EVAL_SNAPSHOTS_PATH (default evals/behavioral/snapshots.json). Both must be absolute paths and override only the live report and snapshot file respectively; cases, mutations, and the runner itself are always read from the repository.

Safety

Fixtures contain only clearly fictitious sk-eval-* values. Do not add real credentials, private repository content, raw OpenCode events, or user-project paths to cases, reports, or snapshots.