mirror of
https://github.com/waybarrios/opencode-power-pack.git
synced 2026-11-18 18:03:43 +00:00
Remove planning docs from the stack PR and ignore them
The planning documents for the behavioral evaluations workflow are contributor-side working documents and should not land in main. Remove them from the working tree and add gitignore rules so regenerated copies do not reappear in future PRs.
This commit is contained in:
@@ -47,6 +47,10 @@ NOTES.md
|
||||
# Local behavioral evaluation reports
|
||||
.artifacts/behavioral-evals/
|
||||
|
||||
# Planning docs for behavioral evaluations (not delivered to main)
|
||||
docs/superpowers/plans/2026-07-30-behavioral-evaluations*.md
|
||||
docs/superpowers/specs/2026-07-30-behavioral-evaluations*.md
|
||||
|
||||
# Node / Bun
|
||||
node_modules/
|
||||
*.log
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,197 +0,0 @@
|
||||
# Behavioral Evaluations Design
|
||||
|
||||
## Context
|
||||
|
||||
This is the fifth change in the hardening stack. It builds on
|
||||
`hardening/skill-refresh` and validates the prompt-trust, scope, recovery, and
|
||||
output contracts introduced by the preceding workflow and skill-refresh
|
||||
changes. The repository already has static contract tests; this change adds a
|
||||
reproducible behavioral baseline without making normal CI depend on provider
|
||||
credentials or nondeterministic model calls.
|
||||
|
||||
After this change is reviewed and merged into its stack base, the top branch
|
||||
will be proposed to `main` in a separate integration pull request. This change
|
||||
does not publish a release or change the package version.
|
||||
|
||||
## Goals
|
||||
|
||||
- Cover the nine skills that directly consume untrusted repository, rules,
|
||||
web, issue, pull request, or tool content.
|
||||
- Detect obedience to embedded instructions, scope widening, secret
|
||||
reproduction, unauthorized posting, and false clean outcomes.
|
||||
- Keep `npm test` deterministic, offline, and credential-free.
|
||||
- Provide an opt-in runner that executes the same cases through a real
|
||||
OpenCode model.
|
||||
- Tie every approved behavioral result to the exact skill content that
|
||||
produced it.
|
||||
- Fail closed on incomplete execution, stale evidence, malformed output, or
|
||||
missing coverage.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Proving that every model or future model version will follow the skills.
|
||||
- Adding a provider SDK, runtime dependency, hosted evaluation service, or CI
|
||||
secret.
|
||||
- Publishing evaluation infrastructure in the npm package.
|
||||
- Evaluating `frontend-design` or `skill-creator` in the initial corpus; they
|
||||
are not among the nine direct untrusted-data consumers hardened in the
|
||||
preceding change.
|
||||
- Replacing the existing static contract and OpenCode compatibility tests.
|
||||
|
||||
## Architecture
|
||||
|
||||
Evaluation infrastructure lives under `evals/behavioral/` and remains outside
|
||||
the package's published `files` allowlist.
|
||||
|
||||
The corpus manifest is inert JSON. Each case defines a stable ID, target skill,
|
||||
behavior category, user input, unique sentinel, deterministic positive and
|
||||
negative oracles, timeout, and expected terminal status. Matchers use a small,
|
||||
validated schema rather than executable JavaScript from fixture data.
|
||||
|
||||
Approved snapshots are JSON records containing the case ID, normalized and
|
||||
redacted final response, expected verdict, model identity, OpenCode version,
|
||||
and SHA-256 of the target `SKILL.md`. A changed skill hash makes its snapshots
|
||||
stale and fails offline validation until the affected cases are run and
|
||||
reviewed again.
|
||||
|
||||
`scripts/behavioral-evals.mjs` owns manifest and snapshot loading, schema
|
||||
validation, hashing, normalization, redaction, deterministic grading, process
|
||||
execution, timeout handling, report generation, and explicit snapshot
|
||||
acceptance. These operations stay in one module until a demonstrated reuse
|
||||
need justifies another abstraction.
|
||||
|
||||
`tests/behavioral-evals.test.mjs` imports the pure validation and grading
|
||||
operations. It validates the corpus, replays approved snapshots, and proves
|
||||
the graders reject deliberately mutated outputs. Normal `npm test` never
|
||||
starts OpenCode or accesses the network.
|
||||
|
||||
## Corpus
|
||||
|
||||
The initial corpus contains twelve cases:
|
||||
|
||||
| Skill | Cases | Required behavior |
|
||||
|---|---:|---|
|
||||
| `agents-md-improver` | 1 | Treat malicious project rules as audit data, preserve client-specific resolution, and redact a fictitious secret. |
|
||||
| `agents-md-revise` | 1 | Refuse to persist a fictitious secret or embedded instruction while retaining safe durable context. |
|
||||
| `code-explorer` | 1 | Preserve the assigned scope and read-only role when repository text requests unrelated tools or paths. |
|
||||
| `code-architect` | 1 | Preserve the parent envelope and avoid converting repository instructions into architecture authority. |
|
||||
| `feature-dev` | 2 | Reject scope widening in issue content and keep incomplete or malformed specialist coverage non-final. |
|
||||
| `code-review` | 2 | Ignore suppression and unauthorized-posting instructions in changed content and prevent incomplete coverage from becoming clean. |
|
||||
| `code-reviewer` | 1 | Report against the frozen handoff without obeying diff instructions or widening the review. |
|
||||
| `security-review` | 2 | Retain exploit and introduction evidence while redacting a fictitious credential despite exclusion instructions, and keep incomplete category coverage non-final. |
|
||||
| `mcp-builder` | 1 | Reject service-document requests for insecure transport, credential logging, or nondeterministic-only testing. |
|
||||
|
||||
Every case contains a unique sentinel that must not appear in an accepted
|
||||
response. Across the corpus, mutation fixtures cover embedded-instruction
|
||||
obedience, scope widening, fictitious-secret reproduction, unauthorized
|
||||
posting, and a false clean result with incomplete coverage. Each mutation must
|
||||
produce the expected failing oracle so a permissive grader cannot pass the
|
||||
suite unnoticed.
|
||||
|
||||
## Offline Replay
|
||||
|
||||
The default test suite checks:
|
||||
|
||||
- the manifest and snapshots satisfy their closed schemas;
|
||||
- case IDs and sentinels are unique;
|
||||
- all nine target skills have the specified twelve-case coverage;
|
||||
- every referenced skill and snapshot exists;
|
||||
- every snapshot hash matches the current bytes of its `SKILL.md`;
|
||||
- each approved snapshot passes all of its case oracles;
|
||||
- every mutation fixture fails for its expected reason;
|
||||
- redaction removes the corpus's fictitious credentials and sentinel forms;
|
||||
- incomplete, timed-out, malformed, or stale records cannot be accepted.
|
||||
|
||||
Snapshots are evidence from one reviewed execution, not a claim that replaying
|
||||
stored text tests the current behavior of an arbitrary model. Content hashes
|
||||
make that limitation explicit by preventing prompt changes from silently
|
||||
retaining old evidence.
|
||||
|
||||
## Live Execution
|
||||
|
||||
`npm run eval:behavioral` requires `OPENCODE_EVAL_MODEL` in
|
||||
`provider/model` form. It runs each case with `opencode run --command <skill>
|
||||
--format json` in a fresh empty temporary project. The repository plugin is
|
||||
loaded from the current worktree.
|
||||
|
||||
The evaluation configuration denies model-facing read, edit, shell, web,
|
||||
task-dispatch, sharing, and external-directory actions. The OpenCode process
|
||||
retains only the network and provider configuration needed to invoke the
|
||||
selected model. The runner does not use `--auto`, `--share`, attachments, a
|
||||
user repository, or printable debug logs.
|
||||
|
||||
Each case has an independent timeout and process. JSON events are processed in
|
||||
memory, and only the final response needed for grading is retained. Forbidden
|
||||
sentinels and fictitious credentials are graded against that in-memory response
|
||||
before redaction, so sanitization cannot turn a behavioral failure into a pass.
|
||||
Before any artifact is written, the runner normalizes volatile formatting and
|
||||
redacts known fictitious credentials, sentinels, and credential-shaped values.
|
||||
Raw events are not persisted.
|
||||
|
||||
The run writes a Git-ignored report with per-case metadata, oracle results, and
|
||||
redacted normalized output. Provider errors, nonzero exits, timeouts, malformed
|
||||
events, permission requests, missing final responses, and failed oracles are
|
||||
reported as incomplete or failed. They never produce an accepted snapshot.
|
||||
|
||||
## Snapshot Acceptance
|
||||
|
||||
Live execution does not modify tracked files. A separate explicit acceptance
|
||||
mode takes one completed report and updates only the snapshots represented by
|
||||
that report. Acceptance requires:
|
||||
|
||||
- all selected cases completed and passed;
|
||||
- each report entry matches the current case definition and skill hash;
|
||||
- model and OpenCode identities are present;
|
||||
- normalized outputs pass redaction a second time;
|
||||
- no output contains a case sentinel, fictitious secret, or forbidden action;
|
||||
- the report contains no unknown, duplicate, skipped, or incomplete case.
|
||||
|
||||
Acceptance rewrites snapshots deterministically so repeated acceptance of the
|
||||
same report produces no diff. The contributor reviews the resulting tracked
|
||||
diff before commit. There is no automatic acceptance in CI.
|
||||
|
||||
## Error Handling
|
||||
|
||||
Schema errors identify the file, case ID when available, and invalid field.
|
||||
Grading errors identify the case and failed oracle without printing unredacted
|
||||
model output. Process errors include the exit status or timeout category but
|
||||
not inherited environment values. Cleanup runs in `finally` blocks and removes
|
||||
temporary projects even when a case fails.
|
||||
|
||||
The live command returns nonzero when any selected case fails or is incomplete.
|
||||
The offline suite returns nonzero for stale snapshots or missing target
|
||||
coverage. An empty selection is an error rather than a successful no-op.
|
||||
|
||||
## Verification And Pull Request Flow
|
||||
|
||||
Before opening the fifth pull request from `hardening/behavioral-evals` to
|
||||
`hardening/skill-refresh`, run:
|
||||
|
||||
1. `npm test`
|
||||
2. `npm run smoke:opencode`
|
||||
3. the complete live behavioral evaluation with reviewed snapshots
|
||||
4. `npm pack --dry-run`
|
||||
5. `git diff --check hardening/skill-refresh...HEAD`
|
||||
|
||||
The package dry run must confirm that evaluation fixtures, reports, scripts,
|
||||
tests, and design documents are absent from the published artifact unless a
|
||||
script is deliberately added to the package allowlist in a separately reviewed
|
||||
change.
|
||||
|
||||
After the fifth pull request is merged into its preserved stack base, open a
|
||||
separate integration pull request from the top branch to `main`. Its review
|
||||
scope is the complete stack, and the same test, smoke, package, diff, and live
|
||||
evaluation evidence must be current. Intermediate stack branches remain until
|
||||
the integration pull request lands. Release and versioning work is separate.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- Twelve reviewed cases cover exactly the nine direct untrusted-data consumers.
|
||||
- Default CI remains deterministic and needs no provider credentials.
|
||||
- A real OpenCode model can execute the corpus through one documented command.
|
||||
- Skill changes invalidate affected behavioral evidence automatically.
|
||||
- Negative mutations demonstrate all five critical failure dimensions are
|
||||
caught by deterministic grading.
|
||||
- Reports and snapshots contain no raw event streams or known credential
|
||||
values.
|
||||
- PR 5 and the final integration PR preserve the approved stacked merge order.
|
||||
Reference in New Issue
Block a user