The testing topic was 8 bytes under its 16 KB cap, so any edit would fail the router check. Move the E2E runbook, quality checklist and Windows verification sections verbatim into a new e2e-runbooks topic instead of raising the cap. Also upgrade scripts/ai-context.py to 0.2.0, which adds link checking, skipDirs and glob-based unrouted exceptions; the existing docs pass.
4.0 KiB
E2E Runbooks and Windows Verification
E2E Runbooks
Runbooks live in ai_docs/tests/*_runbook.md and are executed by mdproof. Before creating or changing one:
- Read
ai_docs/tests/mdproof.jsonwhen present and.mdproof/lessons-learned.md. - Verify every command and flag against source or
--helpinside the container. - Use YAML-free Markdown with Scope, Environment, Steps, and Pass Criteria.
- Give every step a
bashblock and a machine-checkableExpectedsection. - Prefer
--jsonor--format jsonwithjq:assertions over unstable human-readable text.
Execute a runbook with:
docker exec "$CONTAINER" /workspace/.devcontainer/ensure-mdproof.sh
docker exec "$CONTAINER" env SKILLSHARE_DEV_ALLOW_WORKSPACE_PROJECT=1 \
ssenv enter "$ENV_NAME" -- \
mdproof --report json /workspace/ai_docs/tests/<runbook>.md
Do not abort the whole runbook after the first failed step. Preserve every step result, then classify the failure:
- Runbook bug: stale flag, path, or assertion.
- Product bug: CLI behavior contradicts source or acceptance criteria.
- Environment issue: container, network, credential, or shared-path state.
Runbook Quality Checklist
- Verify every flag against
cmd/skillshare/or--help. - Verify project-init and global-init flag sets separately.
registry.yamlappears only after install/reconcile; installed resources do not belong inconfig.yaml.- Project state uses
.skillshare/and global state uses the configuration directory. - Audit customization uses rule IDs, not pattern-group names.
- Expected results use actual substrings, exit codes, regular expressions,
jq:, or snapshots rather than prose wishes. - Prefer
jq:for JSON assertions;log --jsonemits JSONL, not an array. - Never append YAML with a non-idempotent bare
cat >>; prefer the CLI or full replacement. - Writing through a symlink changes its target. Remove the exact symlink first or use another filename when a local file is required.
- Clean exact
/tmptargets at the beginning of a step. - Never assume a repository name equals an installed skill name; verify with
ss list --json.
Windows Verification
Linux tests cannot show Windows link behavior (junctions, file symlinks, Developer Mode). For changes to links, sync modes, or paths on Windows, run ai_docs/tests/windows_file_links_runbook.md on a real Windows machine; its script is scripts/windows/e2e-file-links.ps1. For first-level source links (follow_source_links), run ai_docs/tests/windows_follow_source_links_runbook.md (scripts/windows/e2e-follow-source-links.ps1). With a local UTM guest, scripts/windows/utm.sh handles probe, pinned builds, push, and running scripts as the desktop user (full or basic token); the skillshare-windows-utm skill walks through automated and hands-on runs. For child processes, console windows, or runs without a console (scheduled tasks), run ai_docs/tests/windows_console_windows_runbook.md (scripts/windows/e2e-console-windows.ps1).
- Cross-compile in the devcontainer (
GOOS=windows,GOARCHmatching the guest). The host drives a UTM guest withutmctl(file push,exec,file pull); this is the one allowed host-side execution path, and it never runs the product on macOS. utmctl execis asynchronous and runs as SYSTEM. Scripts write a report ending inDONE, and the host polls it withfile pull, which exits 0 even when the file is missing.- SYSTEM and administrators can create file symlinks. Run as the desktop user through a scheduled task, and use
runas /trustlevel:0x20000to test without symlink rights. - Isolate under
C:\Users\Public\sstest\by overridingUSERPROFILE,HOME,APPDATA,LOCALAPPDATA, andTEMP; confirm the config path before any mutating command. - Inspect results with
fsutil reparsepoint query(0xa0000003junction,0xa000000csymlink) and by reading the file. - Clean up with
rmdir /s /q, which does not follow junctions, and unregister the scheduled tasks.
Report which token (full or basic-user) and architecture each result came from.