diff --git a/README.md b/README.md index 0d8f5e6..6ec4e04 100644 --- a/README.md +++ b/README.md @@ -193,6 +193,14 @@ npm run smoke:opencode npm pack --dry-run ``` +Live behavioral evaluation is opt-in, needs `OPENCODE_EVAL_MODEL`, and runs only from a source checkout: + +```bash +OPENCODE_EVAL_MODEL=provider/model npm run eval:behavioral +``` + +See [`evals/behavioral/README.md`](evals/behavioral/README.md) for the replay, review, and acceptance workflow. Snapshots are content-addressed evidence from one reviewed model execution, not universal guarantees. + Every skill must use a lowercase hyphenated directory/name, provide a trigger-specific description, remain under 500 lines where practical, and record its exact upstream in `UPSTREAMS.json`. ## Acknowledgments diff --git a/evals/behavioral/README.md b/evals/behavioral/README.md new file mode 100644 index 0000000..78c30fd --- /dev/null +++ b/evals/behavioral/README.md @@ -0,0 +1,19 @@ +# Behavioral Evaluations + +These evaluations are contributor infrastructure, not a guarantee that every model follows every workflow. + +## Offline Replay + +Run `npm test`. Replay requires no model, network, or credentials and rejects stale skill or case hashes. + +## Live Run + +Run `OPENCODE_EVAL_MODEL=provider/model npm run eval:behavioral` from a source checkout. The runner uses an empty temporary project, denies model-facing tools, and writes only a redacted report to `.artifacts/behavioral-evals/latest.json`. + +## Review And Acceptance + +Review the report for complete case coverage, passing oracles, and redaction. Then run `npm run eval:behavioral:accept`. Acceptance is explicit, deterministic, and unavailable for failed, incomplete, stale, duplicate, or partial reports. + +## Safety + +Fixtures contain only clearly fictitious `sk-eval-*` values. Do not add real credentials, private repository content, raw OpenCode events, or user-project paths to cases, reports, or snapshots. diff --git a/tests/package-surface.test.mjs b/tests/package-surface.test.mjs index 30b09a1..52ef1eb 100644 --- a/tests/package-surface.test.mjs +++ b/tests/package-surface.test.mjs @@ -28,6 +28,13 @@ test("published package relies on native commands and ships every skill", () => packaged.has("skills/agents-md-improver/references/project-rule-resolution.md"), "project-rule resolution matrix is published", ); + for (const prefix of ["evals/", "scripts/", "tests/", "docs/superpowers/"]) { + assert.equal( + files.some((file) => file.path.startsWith(prefix)), + false, + `${prefix} is contributor-only`, + ); + } }); test("plugin loads as an ES module without runtime warnings", () => {