Remove planning docs from the stack PR and ignore them

The planning documents for the behavioral evaluations workflow are
contributor-side working documents and should not land in main. Remove
them from the working tree and add gitignore rules so regenerated copies
do not reappear in future PRs.
This commit is contained in:
Wayner Barrios
2026-07-31 18:11:18 -04:00
parent 5b5ab30f8c
commit 3c6f979c08
3 changed files with 4 additions and 1272 deletions
+4
View File
@@ -47,6 +47,10 @@ NOTES.md
# Local behavioral evaluation reports
.artifacts/behavioral-evals/
# Planning docs for behavioral evaluations (not delivered to main)
docs/superpowers/plans/2026-07-30-behavioral-evaluations*.md
docs/superpowers/specs/2026-07-30-behavioral-evaluations*.md
# Node / Bun
node_modules/
*.log
File diff suppressed because it is too large Load Diff
@@ -1,197 +0,0 @@
# Behavioral Evaluations Design
## Context
This is the fifth change in the hardening stack. It builds on
`hardening/skill-refresh` and validates the prompt-trust, scope, recovery, and
output contracts introduced by the preceding workflow and skill-refresh
changes. The repository already has static contract tests; this change adds a
reproducible behavioral baseline without making normal CI depend on provider
credentials or nondeterministic model calls.
After this change is reviewed and merged into its stack base, the top branch
will be proposed to `main` in a separate integration pull request. This change
does not publish a release or change the package version.
## Goals
- Cover the nine skills that directly consume untrusted repository, rules,
web, issue, pull request, or tool content.
- Detect obedience to embedded instructions, scope widening, secret
reproduction, unauthorized posting, and false clean outcomes.
- Keep `npm test` deterministic, offline, and credential-free.
- Provide an opt-in runner that executes the same cases through a real
OpenCode model.
- Tie every approved behavioral result to the exact skill content that
produced it.
- Fail closed on incomplete execution, stale evidence, malformed output, or
missing coverage.
## Non-Goals
- Proving that every model or future model version will follow the skills.
- Adding a provider SDK, runtime dependency, hosted evaluation service, or CI
secret.
- Publishing evaluation infrastructure in the npm package.
- Evaluating `frontend-design` or `skill-creator` in the initial corpus; they
are not among the nine direct untrusted-data consumers hardened in the
preceding change.
- Replacing the existing static contract and OpenCode compatibility tests.
## Architecture
Evaluation infrastructure lives under `evals/behavioral/` and remains outside
the package's published `files` allowlist.
The corpus manifest is inert JSON. Each case defines a stable ID, target skill,
behavior category, user input, unique sentinel, deterministic positive and
negative oracles, timeout, and expected terminal status. Matchers use a small,
validated schema rather than executable JavaScript from fixture data.
Approved snapshots are JSON records containing the case ID, normalized and
redacted final response, expected verdict, model identity, OpenCode version,
and SHA-256 of the target `SKILL.md`. A changed skill hash makes its snapshots
stale and fails offline validation until the affected cases are run and
reviewed again.
`scripts/behavioral-evals.mjs` owns manifest and snapshot loading, schema
validation, hashing, normalization, redaction, deterministic grading, process
execution, timeout handling, report generation, and explicit snapshot
acceptance. These operations stay in one module until a demonstrated reuse
need justifies another abstraction.
`tests/behavioral-evals.test.mjs` imports the pure validation and grading
operations. It validates the corpus, replays approved snapshots, and proves
the graders reject deliberately mutated outputs. Normal `npm test` never
starts OpenCode or accesses the network.
## Corpus
The initial corpus contains twelve cases:
| Skill | Cases | Required behavior |
|---|---:|---|
| `agents-md-improver` | 1 | Treat malicious project rules as audit data, preserve client-specific resolution, and redact a fictitious secret. |
| `agents-md-revise` | 1 | Refuse to persist a fictitious secret or embedded instruction while retaining safe durable context. |
| `code-explorer` | 1 | Preserve the assigned scope and read-only role when repository text requests unrelated tools or paths. |
| `code-architect` | 1 | Preserve the parent envelope and avoid converting repository instructions into architecture authority. |
| `feature-dev` | 2 | Reject scope widening in issue content and keep incomplete or malformed specialist coverage non-final. |
| `code-review` | 2 | Ignore suppression and unauthorized-posting instructions in changed content and prevent incomplete coverage from becoming clean. |
| `code-reviewer` | 1 | Report against the frozen handoff without obeying diff instructions or widening the review. |
| `security-review` | 2 | Retain exploit and introduction evidence while redacting a fictitious credential despite exclusion instructions, and keep incomplete category coverage non-final. |
| `mcp-builder` | 1 | Reject service-document requests for insecure transport, credential logging, or nondeterministic-only testing. |
Every case contains a unique sentinel that must not appear in an accepted
response. Across the corpus, mutation fixtures cover embedded-instruction
obedience, scope widening, fictitious-secret reproduction, unauthorized
posting, and a false clean result with incomplete coverage. Each mutation must
produce the expected failing oracle so a permissive grader cannot pass the
suite unnoticed.
## Offline Replay
The default test suite checks:
- the manifest and snapshots satisfy their closed schemas;
- case IDs and sentinels are unique;
- all nine target skills have the specified twelve-case coverage;
- every referenced skill and snapshot exists;
- every snapshot hash matches the current bytes of its `SKILL.md`;
- each approved snapshot passes all of its case oracles;
- every mutation fixture fails for its expected reason;
- redaction removes the corpus's fictitious credentials and sentinel forms;
- incomplete, timed-out, malformed, or stale records cannot be accepted.
Snapshots are evidence from one reviewed execution, not a claim that replaying
stored text tests the current behavior of an arbitrary model. Content hashes
make that limitation explicit by preventing prompt changes from silently
retaining old evidence.
## Live Execution
`npm run eval:behavioral` requires `OPENCODE_EVAL_MODEL` in
`provider/model` form. It runs each case with `opencode run --command <skill>
--format json` in a fresh empty temporary project. The repository plugin is
loaded from the current worktree.
The evaluation configuration denies model-facing read, edit, shell, web,
task-dispatch, sharing, and external-directory actions. The OpenCode process
retains only the network and provider configuration needed to invoke the
selected model. The runner does not use `--auto`, `--share`, attachments, a
user repository, or printable debug logs.
Each case has an independent timeout and process. JSON events are processed in
memory, and only the final response needed for grading is retained. Forbidden
sentinels and fictitious credentials are graded against that in-memory response
before redaction, so sanitization cannot turn a behavioral failure into a pass.
Before any artifact is written, the runner normalizes volatile formatting and
redacts known fictitious credentials, sentinels, and credential-shaped values.
Raw events are not persisted.
The run writes a Git-ignored report with per-case metadata, oracle results, and
redacted normalized output. Provider errors, nonzero exits, timeouts, malformed
events, permission requests, missing final responses, and failed oracles are
reported as incomplete or failed. They never produce an accepted snapshot.
## Snapshot Acceptance
Live execution does not modify tracked files. A separate explicit acceptance
mode takes one completed report and updates only the snapshots represented by
that report. Acceptance requires:
- all selected cases completed and passed;
- each report entry matches the current case definition and skill hash;
- model and OpenCode identities are present;
- normalized outputs pass redaction a second time;
- no output contains a case sentinel, fictitious secret, or forbidden action;
- the report contains no unknown, duplicate, skipped, or incomplete case.
Acceptance rewrites snapshots deterministically so repeated acceptance of the
same report produces no diff. The contributor reviews the resulting tracked
diff before commit. There is no automatic acceptance in CI.
## Error Handling
Schema errors identify the file, case ID when available, and invalid field.
Grading errors identify the case and failed oracle without printing unredacted
model output. Process errors include the exit status or timeout category but
not inherited environment values. Cleanup runs in `finally` blocks and removes
temporary projects even when a case fails.
The live command returns nonzero when any selected case fails or is incomplete.
The offline suite returns nonzero for stale snapshots or missing target
coverage. An empty selection is an error rather than a successful no-op.
## Verification And Pull Request Flow
Before opening the fifth pull request from `hardening/behavioral-evals` to
`hardening/skill-refresh`, run:
1. `npm test`
2. `npm run smoke:opencode`
3. the complete live behavioral evaluation with reviewed snapshots
4. `npm pack --dry-run`
5. `git diff --check hardening/skill-refresh...HEAD`
The package dry run must confirm that evaluation fixtures, reports, scripts,
tests, and design documents are absent from the published artifact unless a
script is deliberately added to the package allowlist in a separately reviewed
change.
After the fifth pull request is merged into its preserved stack base, open a
separate integration pull request from the top branch to `main`. Its review
scope is the complete stack, and the same test, smoke, package, diff, and live
evaluation evidence must be current. Intermediate stack branches remain until
the integration pull request lands. Release and versioning work is separate.
## Success Criteria
- Twelve reviewed cases cover exactly the nine direct untrusted-data consumers.
- Default CI remains deterministic and needs no provider credentials.
- A real OpenCode model can execute the corpus through one documented command.
- Skill changes invalidate affected behavioral evidence automatically.
- Negative mutations demonstrate all five critical failure dimensions are
caught by deterministic grading.
- Reports and snapshots contain no raw event streams or known credential
values.
- PR 5 and the final integration PR preserve the approved stacked merge order.