Document behavioral evaluation workflow

This commit is contained in:
Wayner Barrios
2026-07-31 16:50:05 -04:00
parent b0a4780acc
commit d2e9b4ee88
3 changed files with 34 additions and 0 deletions
+8
View File
@@ -193,6 +193,14 @@ npm run smoke:opencode
npm pack --dry-run
```
Live behavioral evaluation is opt-in, needs `OPENCODE_EVAL_MODEL`, and runs only from a source checkout:
```bash
OPENCODE_EVAL_MODEL=provider/model npm run eval:behavioral
```
See [`evals/behavioral/README.md`](evals/behavioral/README.md) for the replay, review, and acceptance workflow. Snapshots are content-addressed evidence from one reviewed model execution, not universal guarantees.
Every skill must use a lowercase hyphenated directory/name, provide a trigger-specific description, remain under 500 lines where practical, and record its exact upstream in `UPSTREAMS.json`.
## Acknowledgments
+19
View File
@@ -0,0 +1,19 @@
# Behavioral Evaluations
These evaluations are contributor infrastructure, not a guarantee that every model follows every workflow.
## Offline Replay
Run `npm test`. Replay requires no model, network, or credentials and rejects stale skill or case hashes.
## Live Run
Run `OPENCODE_EVAL_MODEL=provider/model npm run eval:behavioral` from a source checkout. The runner uses an empty temporary project, denies model-facing tools, and writes only a redacted report to `.artifacts/behavioral-evals/latest.json`.
## Review And Acceptance
Review the report for complete case coverage, passing oracles, and redaction. Then run `npm run eval:behavioral:accept`. Acceptance is explicit, deterministic, and unavailable for failed, incomplete, stale, duplicate, or partial reports.
## Safety
Fixtures contain only clearly fictitious `sk-eval-*` values. Do not add real credentials, private repository content, raw OpenCode events, or user-project paths to cases, reports, or snapshots.
+7
View File
@@ -28,6 +28,13 @@ test("published package relies on native commands and ships every skill", () =>
packaged.has("skills/agents-md-improver/references/project-rule-resolution.md"),
"project-rule resolution matrix is published",
);
for (const prefix of ["evals/", "scripts/", "tests/", "docs/superpowers/"]) {
assert.equal(
files.some((file) => file.path.startsWith(prefix)),
false,
`${prefix} is contributor-only`,
);
}
});
test("plugin loads as an ES module without runtime warnings", () => {