Every container start installed vhs, ttyd, ffmpeg and chromium before the dev servers came up, which takes minutes on a fresh container while screenshots are only taken now and then. record-tui.sh now runs the install itself, so the cost moves to the first screenshot run and later runs skip it. Refs #200
8.1 KiB
Build, Test, and E2E
Use when building, running the skillshare CLI, executing Go or frontend tests, reproducing bugs, using the devcontainer/ssenv, or creating and running E2E runbooks.
Execution Boundary
The host is macOS; the project verification environment is the Linux devcontainer. Run these inside the devcontainer:
ssandskillsharecommands;go build,go test,make test, andmake check;- dashboard and website package scripts;
- bug reproductions and E2E runbooks.
The host may be used for source edits, git status/diff/log, read-only searches, python3 scripts/ai-context.py check, and documentation checks that do not execute the product binary.
Devcontainer
make devc # start, initialize, and enter
make devc-up # start without opening a shell
make devc-status
make devc-down
For programmatic access, resolve the container first:
CONTAINER=$(docker compose -f .devcontainer/docker-compose.yml ps -q skillshare-devcontainer)
If it is empty, stop and ask the user to run make devc-up before executing product commands. The source tree is bind-mounted at /workspace. The ss wrapper automatically builds current source, so ordinary CLI verification does not need a separate make build.
Narrow Verification First
Change to /workspace inside the container:
docker exec "$CONTAINER" bash -lc 'cd /workspace && go test ./internal/<package>/... -count=1'
docker exec "$CONTAINER" bash -lc 'cd /workspace && go test ./tests/integration -run TestName -count=1'
docker exec "$CONTAINER" bash -lc 'cd /workspace && make test-unit'
docker exec "$CONTAINER" bash -lc 'cd /workspace && make test-int'
docker exec "$CONTAINER" bash -lc 'cd /workspace && make check'
Start with the specific package or test that proves the change. Broaden only when new risk, failure evidence, or a release gate requires it. Base retries on new evidence and distinguish pre-existing failures from regressions.
Real-shell completion tests (TestCompletion_{Zsh,Fish}_Completes*) skip when zsh or fish is missing, as in the devcontainer. To run them, start a throwaway container from the devcontainer image with the checkout at /workspace and the Go cache volumes, install zsh fish with apt-get there only, then make build and go test ./tests/integration -run 'Completion' -count=1.
Stateful CLI Isolation
Use a fresh ssenv for tests that modify configuration or state:
ENV_NAME="task-<descriptive-id>"
docker exec "$CONTAINER" ssenv create "$ENV_NAME" --init
docker exec "$CONTAINER" ssenv enter "$ENV_NAME" -- ss status --json
Rules:
- Create a fresh environment for every E2E run; never reuse stale state.
ssenvisolates onlyHOME./tmpand other system paths are shared, so runbooks must use unique paths or clean an exact target first.- Use
bash -cfor multi-command sequences andcd /workspacebefore Go commands. --initalready performs global initialization and creates the defaultrulesextra; a runbook must not assume an empty environment.--seedfills the environment forinittesting: claude (2 skills), codex (1 skill), cursor, and a local repo at$HOME/remote/skills.gitwhosepdf-toolsclashes with claude's.- Report the environment after execution. Delete or preserve it for debugging according to user direction; never discard requested evidence silently.
E2E Runbooks
Runbooks live in ai_docs/tests/*_runbook.md and are executed by mdproof. Before creating or changing one:
- Read
ai_docs/tests/mdproof.jsonwhen present and.mdproof/lessons-learned.md. - Verify every command and flag against source or
--helpinside the container. - Use YAML-free Markdown with Scope, Environment, Steps, and Pass Criteria.
- Give every step a
bashblock and a machine-checkableExpectedsection. - Prefer
--jsonor--format jsonwithjq:assertions over unstable human-readable text.
Execute a runbook with:
docker exec "$CONTAINER" /workspace/.devcontainer/ensure-mdproof.sh
docker exec "$CONTAINER" env SKILLSHARE_DEV_ALLOW_WORKSPACE_PROJECT=1 \
ssenv enter "$ENV_NAME" -- \
mdproof --report json /workspace/ai_docs/tests/<runbook>.md
Do not abort the whole runbook after the first failed step. Preserve every step result, then classify the failure:
- Runbook bug: stale flag, path, or assertion.
- Product bug: CLI behavior contradicts source or acceptance criteria.
- Environment issue: container, network, credential, or shared-path state.
Runbook Quality Checklist
- Verify every flag against
cmd/skillshare/or--help. - Verify project-init and global-init flag sets separately.
registry.yamlappears only after install/reconcile; installed resources do not belong inconfig.yaml.- Project state uses
.skillshare/and global state uses the configuration directory. - Audit customization uses rule IDs, not pattern-group names.
- Expected results use actual substrings, exit codes, regular expressions,
jq:, or snapshots rather than prose wishes. - Prefer
jq:for JSON assertions;log --jsonemits JSONL, not an array. - Never append YAML with a non-idempotent bare
cat >>; prefer the CLI or full replacement. - Writing through a symlink changes its target. Remove the exact symlink first or use another filename when a local file is required.
- Clean exact
/tmptargets at the beginning of a step. - Never assume a repository name equals an installed skill name; verify with
ss list --json.
Windows Verification
Linux tests cannot show Windows link behavior (junctions, file symlinks, Developer Mode). For changes to links, sync modes, or paths on Windows, run ai_docs/tests/windows_file_links_runbook.md on a real Windows machine; its script is scripts/windows/e2e-file-links.ps1. With a local UTM guest, scripts/windows/utm.sh handles probe, pinned builds, push, and running scripts as the desktop user (full or basic token); the skillshare-windows-utm skill walks through automated and hands-on runs. For child processes, console windows, or runs without a console (scheduled tasks), run ai_docs/tests/windows_console_windows_runbook.md (scripts/windows/e2e-console-windows.ps1).
- Cross-compile in the devcontainer (
GOOS=windows,GOARCHmatching the guest). The host drives a UTM guest withutmctl(file push,exec,file pull); this is the one allowed host-side execution path, and it never runs the product on macOS. utmctl execis asynchronous and runs as SYSTEM. Scripts write a report ending inDONE, and the host polls it withfile pull, which exits 0 even when the file is missing.- SYSTEM and administrators can create file symlinks. Run as the desktop user through a scheduled task, and use
runas /trustlevel:0x20000to test without symlink rights. - Isolate under
C:\Users\Public\sstest\by overridingUSERPROFILE,HOME,APPDATA,LOCALAPPDATA, andTEMP; confirm the config path before any mutating command. - Inspect results with
fsutil reparsepoint query(0xa0000003junction,0xa000000csymlink) and by reading the file. - Clean up with
rmdir /s /q, which does not follow junctions, and unregister the scheduled tasks.
Report which token (full or basic-user) and architecture each result came from.
Frontend and Website Verification
Start the dashboard through the devcontainer ui command. Run website package commands according to website/AGENTS.md. Visual changes require inspected screenshots in addition to builds and tests. Load frontend for design-specific checks.
For full-screen TUI changes, screenshot every TUI inside the devcontainer with scripts/screenshots/record-tui.sh [name...]. It records the binary in BIN (default /tmp/tuibin) and writes PNGs to OUT (default /tmp/tui-shots). Each screen runs in a fresh ssenv that scripts/screenshots/tui-fixture.sh fills with real commands. The first run installs vhs and its tools through .devcontainer/ensure-recording-tools.sh, which takes a few minutes; later runs skip it.
Reporting
Report the exact commands, pass/fail status, failure location, and limitations. Starting a command is not proof of success. Never disable tests, hooks, audits, or trust prompts to obtain a pass.