Files
skillshare/wiki/testing.md
T
Willie 8c98ab3b51 test(completion): exercise zsh and fish completions
The completion tests only grepped the generated scripts, so a script
that parses but offers nothing, or a flag listed in the wrong place,
passed. These tests load the scripts in a real fish (complete -C) and a
real zsh (compinit under zpty, capturing what compadd receives) and
assert the mcp subcommands and the mcp check flags, including --live
and --timeout. They skip where the shells are not installed; the
testing wiki says how to run them in a throwaway container.

Refs #141
2026-09-30 14:37:43 +08:00

126 lines
7.3 KiB
Markdown

# Build, Test, and E2E
Use when building, running the skillshare CLI, executing Go or frontend tests, reproducing bugs, using the devcontainer/ssenv, or creating and running E2E runbooks.
## Execution Boundary
The host is macOS; the project verification environment is the Linux devcontainer. Run these inside the devcontainer:
- `ss` and `skillshare` commands;
- `go build`, `go test`, `make test`, and `make check`;
- dashboard and website package scripts;
- bug reproductions and E2E runbooks.
The host may be used for source edits, `git status/diff/log`, read-only searches, `python3 scripts/ai-context.py check`, and documentation checks that do not execute the product binary.
## Devcontainer
```sh
make devc # start, initialize, and enter
make devc-up # start without opening a shell
make devc-status
make devc-down
```
For programmatic access, resolve the container first:
```sh
CONTAINER=$(docker compose -f .devcontainer/docker-compose.yml ps -q skillshare-devcontainer)
```
If it is empty, stop and ask the user to run `make devc-up` before executing product commands. The source tree is bind-mounted at `/workspace`. The `ss` wrapper automatically builds current source, so ordinary CLI verification does not need a separate `make build`.
## Narrow Verification First
Change to `/workspace` inside the container:
```sh
docker exec "$CONTAINER" bash -lc 'cd /workspace && go test ./internal/<package>/... -count=1'
docker exec "$CONTAINER" bash -lc 'cd /workspace && go test ./tests/integration -run TestName -count=1'
docker exec "$CONTAINER" bash -lc 'cd /workspace && make test-unit'
docker exec "$CONTAINER" bash -lc 'cd /workspace && make test-int'
docker exec "$CONTAINER" bash -lc 'cd /workspace && make check'
```
Start with the specific package or test that proves the change. Broaden only when new risk, failure evidence, or a release gate requires it. Base retries on new evidence and distinguish pre-existing failures from regressions.
Real-shell completion tests (`TestCompletion_{Zsh,Fish}_Completes*`) skip when zsh or fish is missing, as in the devcontainer. To run them, start a throwaway container from the devcontainer image with the checkout at `/workspace` and the Go cache volumes, install `zsh fish` with apt-get there only, then `make build` and `go test ./tests/integration -run 'Completion' -count=1`.
## Stateful CLI Isolation
Use a fresh `ssenv` for tests that modify configuration or state:
```sh
ENV_NAME="task-<descriptive-id>"
docker exec "$CONTAINER" ssenv create "$ENV_NAME" --init
docker exec "$CONTAINER" ssenv enter "$ENV_NAME" -- ss status --json
```
Rules:
- Create a fresh environment for every E2E run; never reuse stale state.
- `ssenv` isolates only `HOME`. `/tmp` and other system paths are shared, so runbooks must use unique paths or clean an exact target first.
- Use `bash -c` for multi-command sequences and `cd /workspace` before Go commands.
- `--init` already performs global initialization and creates the default `rules` extra; a runbook must not assume an empty environment.
- Report the environment after execution. Delete or preserve it for debugging according to user direction; never discard requested evidence silently.
## E2E Runbooks
Runbooks live in `ai_docs/tests/*_runbook.md` and are executed by mdproof. Before creating or changing one:
1. Read `ai_docs/tests/mdproof.json` when present and `.mdproof/lessons-learned.md`.
2. Verify every command and flag against source or `--help` inside the container.
3. Use YAML-free Markdown with Scope, Environment, Steps, and Pass Criteria.
4. Give every step a `bash` block and a machine-checkable `Expected` section.
5. Prefer `--json` or `--format json` with `jq:` assertions over unstable human-readable text.
Execute a runbook with:
```sh
docker exec "$CONTAINER" /workspace/.devcontainer/ensure-mdproof.sh
docker exec "$CONTAINER" env SKILLSHARE_DEV_ALLOW_WORKSPACE_PROJECT=1 \
ssenv enter "$ENV_NAME" -- \
mdproof --report json /workspace/ai_docs/tests/<runbook>.md
```
Do not abort the whole runbook after the first failed step. Preserve every step result, then classify the failure:
- **Runbook bug:** stale flag, path, or assertion.
- **Product bug:** CLI behavior contradicts source or acceptance criteria.
- **Environment issue:** container, network, credential, or shared-path state.
## Runbook Quality Checklist
- Verify every flag against `cmd/skillshare/` or `--help`.
- Verify project-init and global-init flag sets separately.
- `registry.yaml` appears only after install/reconcile; installed resources do not belong in `config.yaml`.
- Project state uses `.skillshare/` and global state uses the configuration directory.
- Audit customization uses rule IDs, not pattern-group names.
- Expected results use actual substrings, exit codes, regular expressions, `jq:`, or snapshots rather than prose wishes.
- Prefer `jq:` for JSON assertions; `log --json` emits JSONL, not an array.
- Never append YAML with a non-idempotent bare `cat >>`; prefer the CLI or full replacement.
- Writing through a symlink changes its target. Remove the exact symlink first or use another filename when a local file is required.
- Clean exact `/tmp` targets at the beginning of a step.
- Never assume a repository name equals an installed skill name; verify with `ss list --json`.
## Windows Verification
Linux tests cannot show Windows link behavior (junctions, file symlinks, Developer Mode). For changes to links, sync modes, or paths on Windows, run `ai_docs/tests/windows_file_links_runbook.md` on a real Windows machine; its script is `scripts/windows/e2e-file-links.ps1`. With a local UTM guest, `scripts/windows/utm.sh` handles probe, pinned builds, push, and running scripts as the desktop user (full or basic token); the `skillshare-windows-utm` skill walks through automated and hands-on runs.
- Cross-compile in the devcontainer (`GOOS=windows`, `GOARCH` matching the guest). The host drives a UTM guest with `utmctl` (`file push`, `exec`, `file pull`); this is the one allowed host-side execution path, and it never runs the product on macOS.
- `utmctl exec` is asynchronous and runs as SYSTEM. Scripts write a report ending in `DONE`, and the host polls it with `file pull`, which exits 0 even when the file is missing.
- SYSTEM and administrators can create file symlinks. Run as the desktop user through a scheduled task, and use `runas /trustlevel:0x20000` to test without symlink rights.
- Isolate under `C:\Users\Public\sstest\` by overriding `USERPROFILE`, `HOME`, `APPDATA`, `LOCALAPPDATA`, and `TEMP`; confirm the config path before any mutating command.
- Inspect results with `fsutil reparsepoint query` (`0xa0000003` junction, `0xa000000c` symlink) and by reading the file.
- Clean up with `rmdir /s /q`, which does not follow junctions, and unregister the scheduled tasks.
Report which token (full or basic-user) and architecture each result came from.
## Frontend and Website Verification
Start the dashboard through the devcontainer `ui` command. Run website package commands according to `website/AGENTS.md`. Visual changes require inspected screenshots in addition to builds and tests. Load `frontend` for design-specific checks.
## Reporting
Report the exact commands, pass/fail status, failure location, and limitations. Starting a command is not proof of success. Never disable tests, hooks, audits, or trust prompts to obtain a pass.