Files
skillshare/wiki/testing.md
T
Willie 8c98ab3b51 test(completion): exercise zsh and fish completions
The completion tests only grepped the generated scripts, so a script
that parses but offers nothing, or a flag listed in the wrong place,
passed. These tests load the scripts in a real fish (complete -C) and a
real zsh (compinit under zpty, capturing what compadd receives) and
assert the mcp subcommands and the mcp check flags, including --live
and --timeout. They skip where the shells are not installed; the
testing wiki says how to run them in a throwaway container.

Refs #141
2026-09-30 14:37:43 +08:00

7.3 KiB

Build, Test, and E2E

Use when building, running the skillshare CLI, executing Go or frontend tests, reproducing bugs, using the devcontainer/ssenv, or creating and running E2E runbooks.

Execution Boundary

The host is macOS; the project verification environment is the Linux devcontainer. Run these inside the devcontainer:

  • ss and skillshare commands;
  • go build, go test, make test, and make check;
  • dashboard and website package scripts;
  • bug reproductions and E2E runbooks.

The host may be used for source edits, git status/diff/log, read-only searches, python3 scripts/ai-context.py check, and documentation checks that do not execute the product binary.

Devcontainer

make devc          # start, initialize, and enter
make devc-up       # start without opening a shell
make devc-status
make devc-down

For programmatic access, resolve the container first:

CONTAINER=$(docker compose -f .devcontainer/docker-compose.yml ps -q skillshare-devcontainer)

If it is empty, stop and ask the user to run make devc-up before executing product commands. The source tree is bind-mounted at /workspace. The ss wrapper automatically builds current source, so ordinary CLI verification does not need a separate make build.

Narrow Verification First

Change to /workspace inside the container:

docker exec "$CONTAINER" bash -lc 'cd /workspace && go test ./internal/<package>/... -count=1'
docker exec "$CONTAINER" bash -lc 'cd /workspace && go test ./tests/integration -run TestName -count=1'
docker exec "$CONTAINER" bash -lc 'cd /workspace && make test-unit'
docker exec "$CONTAINER" bash -lc 'cd /workspace && make test-int'
docker exec "$CONTAINER" bash -lc 'cd /workspace && make check'

Start with the specific package or test that proves the change. Broaden only when new risk, failure evidence, or a release gate requires it. Base retries on new evidence and distinguish pre-existing failures from regressions.

Real-shell completion tests (TestCompletion_{Zsh,Fish}_Completes*) skip when zsh or fish is missing, as in the devcontainer. To run them, start a throwaway container from the devcontainer image with the checkout at /workspace and the Go cache volumes, install zsh fish with apt-get there only, then make build and go test ./tests/integration -run 'Completion' -count=1.

Stateful CLI Isolation

Use a fresh ssenv for tests that modify configuration or state:

ENV_NAME="task-<descriptive-id>"
docker exec "$CONTAINER" ssenv create "$ENV_NAME" --init
docker exec "$CONTAINER" ssenv enter "$ENV_NAME" -- ss status --json

Rules:

  • Create a fresh environment for every E2E run; never reuse stale state.
  • ssenv isolates only HOME. /tmp and other system paths are shared, so runbooks must use unique paths or clean an exact target first.
  • Use bash -c for multi-command sequences and cd /workspace before Go commands.
  • --init already performs global initialization and creates the default rules extra; a runbook must not assume an empty environment.
  • Report the environment after execution. Delete or preserve it for debugging according to user direction; never discard requested evidence silently.

E2E Runbooks

Runbooks live in ai_docs/tests/*_runbook.md and are executed by mdproof. Before creating or changing one:

  1. Read ai_docs/tests/mdproof.json when present and .mdproof/lessons-learned.md.
  2. Verify every command and flag against source or --help inside the container.
  3. Use YAML-free Markdown with Scope, Environment, Steps, and Pass Criteria.
  4. Give every step a bash block and a machine-checkable Expected section.
  5. Prefer --json or --format json with jq: assertions over unstable human-readable text.

Execute a runbook with:

docker exec "$CONTAINER" /workspace/.devcontainer/ensure-mdproof.sh
docker exec "$CONTAINER" env SKILLSHARE_DEV_ALLOW_WORKSPACE_PROJECT=1 \
  ssenv enter "$ENV_NAME" -- \
  mdproof --report json /workspace/ai_docs/tests/<runbook>.md

Do not abort the whole runbook after the first failed step. Preserve every step result, then classify the failure:

  • Runbook bug: stale flag, path, or assertion.
  • Product bug: CLI behavior contradicts source or acceptance criteria.
  • Environment issue: container, network, credential, or shared-path state.

Runbook Quality Checklist

  • Verify every flag against cmd/skillshare/ or --help.
  • Verify project-init and global-init flag sets separately.
  • registry.yaml appears only after install/reconcile; installed resources do not belong in config.yaml.
  • Project state uses .skillshare/ and global state uses the configuration directory.
  • Audit customization uses rule IDs, not pattern-group names.
  • Expected results use actual substrings, exit codes, regular expressions, jq:, or snapshots rather than prose wishes.
  • Prefer jq: for JSON assertions; log --json emits JSONL, not an array.
  • Never append YAML with a non-idempotent bare cat >>; prefer the CLI or full replacement.
  • Writing through a symlink changes its target. Remove the exact symlink first or use another filename when a local file is required.
  • Clean exact /tmp targets at the beginning of a step.
  • Never assume a repository name equals an installed skill name; verify with ss list --json.

Windows Verification

Linux tests cannot show Windows link behavior (junctions, file symlinks, Developer Mode). For changes to links, sync modes, or paths on Windows, run ai_docs/tests/windows_file_links_runbook.md on a real Windows machine; its script is scripts/windows/e2e-file-links.ps1. With a local UTM guest, scripts/windows/utm.sh handles probe, pinned builds, push, and running scripts as the desktop user (full or basic token); the skillshare-windows-utm skill walks through automated and hands-on runs.

  • Cross-compile in the devcontainer (GOOS=windows, GOARCH matching the guest). The host drives a UTM guest with utmctl (file push, exec, file pull); this is the one allowed host-side execution path, and it never runs the product on macOS.
  • utmctl exec is asynchronous and runs as SYSTEM. Scripts write a report ending in DONE, and the host polls it with file pull, which exits 0 even when the file is missing.
  • SYSTEM and administrators can create file symlinks. Run as the desktop user through a scheduled task, and use runas /trustlevel:0x20000 to test without symlink rights.
  • Isolate under C:\Users\Public\sstest\ by overriding USERPROFILE, HOME, APPDATA, LOCALAPPDATA, and TEMP; confirm the config path before any mutating command.
  • Inspect results with fsutil reparsepoint query (0xa0000003 junction, 0xa000000c symlink) and by reading the file.
  • Clean up with rmdir /s /q, which does not follow junctions, and unregister the scheduled tasks.

Report which token (full or basic-user) and architecture each result came from.

Frontend and Website Verification

Start the dashboard through the devcontainer ui command. Run website package commands according to website/AGENTS.md. Visual changes require inspected screenshots in addition to builds and tests. Load frontend for design-specific checks.

Reporting

Report the exact commands, pass/fail status, failure location, and limitations. Starting a command is not proof of success. Never disable tests, hooks, audits, or trust prompts to obtain a pass.