mirror of
https://github.com/waybarrios/opencode-power-pack.git
synced 2026-11-18 18:03:43 +00:00
Document behavioral evaluation workflow
This commit is contained in:
@@ -193,6 +193,14 @@ npm run smoke:opencode
|
||||
npm pack --dry-run
|
||||
```
|
||||
|
||||
Live behavioral evaluation is opt-in, needs `OPENCODE_EVAL_MODEL`, and runs only from a source checkout:
|
||||
|
||||
```bash
|
||||
OPENCODE_EVAL_MODEL=provider/model npm run eval:behavioral
|
||||
```
|
||||
|
||||
See [`evals/behavioral/README.md`](evals/behavioral/README.md) for the replay, review, and acceptance workflow. Snapshots are content-addressed evidence from one reviewed model execution, not universal guarantees.
|
||||
|
||||
Every skill must use a lowercase hyphenated directory/name, provide a trigger-specific description, remain under 500 lines where practical, and record its exact upstream in `UPSTREAMS.json`.
|
||||
|
||||
## Acknowledgments
|
||||
|
||||
@@ -0,0 +1,19 @@
|
||||
# Behavioral Evaluations
|
||||
|
||||
These evaluations are contributor infrastructure, not a guarantee that every model follows every workflow.
|
||||
|
||||
## Offline Replay
|
||||
|
||||
Run `npm test`. Replay requires no model, network, or credentials and rejects stale skill or case hashes.
|
||||
|
||||
## Live Run
|
||||
|
||||
Run `OPENCODE_EVAL_MODEL=provider/model npm run eval:behavioral` from a source checkout. The runner uses an empty temporary project, denies model-facing tools, and writes only a redacted report to `.artifacts/behavioral-evals/latest.json`.
|
||||
|
||||
## Review And Acceptance
|
||||
|
||||
Review the report for complete case coverage, passing oracles, and redaction. Then run `npm run eval:behavioral:accept`. Acceptance is explicit, deterministic, and unavailable for failed, incomplete, stale, duplicate, or partial reports.
|
||||
|
||||
## Safety
|
||||
|
||||
Fixtures contain only clearly fictitious `sk-eval-*` values. Do not add real credentials, private repository content, raw OpenCode events, or user-project paths to cases, reports, or snapshots.
|
||||
@@ -28,6 +28,13 @@ test("published package relies on native commands and ships every skill", () =>
|
||||
packaged.has("skills/agents-md-improver/references/project-rule-resolution.md"),
|
||||
"project-rule resolution matrix is published",
|
||||
);
|
||||
for (const prefix of ["evals/", "scripts/", "tests/", "docs/superpowers/"]) {
|
||||
assert.equal(
|
||||
files.some((file) => file.path.startsWith(prefix)),
|
||||
false,
|
||||
`${prefix} is contributor-only`,
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
test("plugin loads as an ES module without runtime warnings", () => {
|
||||
|
||||
Reference in New Issue
Block a user