From 7de9fa4fb0c77a4f6e898e9619ef1c6610ac4931 Mon Sep 17 00:00:00 2001 From: Willie Date: Thu, 21 May 2026 15:08:40 +0800 Subject: [PATCH] =?UTF-8?q?test(e2e):=20add=20global=20sources=20map=20?= =?UTF-8?q?=E2=86=94=20legacy=20fields=20interop=20runbook?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Six-step mdproof runbook exercising the v0.19.16 global sources map: 1. Fresh init -g writes sources: map (no legacy source/agents_source/extras_source) 2. Legacy-only config (source: + agents_source:) — ss list still works 3. New sources map (sources.skills + sources.agents) — ss list works 4. Mixed config — sources.skills wins over legacy source 5. ss sync writes to a custom target with sources.skills set 6. ss sync rejects configs that set neither source nor sources.skills Runs in 0.9s and prevents regressions on either format. Also: record an mdproof lessons-learned entry for `ss sync --json` output — it has no top-level `.errors` field; assert `.linked` / `.details` or rely on exit_code instead. --- .mdproof/lessons-learned.md | 7 + ai_docs/tests/global_sources_map_runbook.md | 234 ++++++++++++++++++++ 2 files changed, 241 insertions(+) create mode 100644 ai_docs/tests/global_sources_map_runbook.md diff --git a/.mdproof/lessons-learned.md b/.mdproof/lessons-learned.md index 76c5d28cd..379b45787 100644 --- a/.mdproof/lessons-learned.md +++ b/.mdproof/lessons-learned.md @@ -92,3 +92,10 @@ - **Discovery**: In a full run of 22 runbooks, the first clean run had 2 failures. But the second run (same ssenv, re-running mdproof) had 10 failures — all due to accumulated state from the first run (1257 trash items, stale registry entries, leftover extras). Even within a single full run, alphabetically-later runbooks can fail because of state left by earlier ones - **Fix**: (1) Each runbook should clean up its own footprint in Step 1 (rm -rf /tmp/ paths, clear trash, reset config sections). (2) For authoritative results, run each runbook in its own fresh ssenv. (3) Full-directory runs are useful as smoke tests but failures should be re-verified in isolation before treating them as real bugs - **Runbooks affected**: extras_refactor_json (file_count mismatch from extras_commands leftovers), gitlab_hosts_config (trash from previous runbooks broke go test), registry_yaml_split (/tmp/ state from prior runs) + +### [assertion] `ss sync --json` has no `.errors` field — use `.linked` / `.details` + +- **Context**: `global_sources_map_runbook` step 5 asserted `jq: .errors == 0` on `ss sync --json` to verify a successful sync. The assertion failed because the output schema doesn't include an `errors` key at all (jq returns `null`, which compares unequal to `0`) +- **Discovery**: `ss sync --json` emits `{targets, linked, local, updated, pruned, ignored_count, ignored_skills, dry_run, duration, details[], context_cost}`. Failure is signalled by the process exit code and per-target details, not by a top-level `errors` field. Many AI-generated assertions assume a generic `.errors == 0` pattern from REST APIs — that pattern doesn't apply here +- **Fix**: Assert positive evidence of the sync outcome instead. Examples: `jq: .linked == ` for "N skills got linked", `jq: .details[0].name == ""` for target-level reporting, `jq: .updated == 0` for "nothing changed in this re-run". Pair with `exit_code: 0` for the failure-case guard +- **Runbooks affected**: global_sources_map_runbook.md (step 5 fixed before merging) diff --git a/ai_docs/tests/global_sources_map_runbook.md b/ai_docs/tests/global_sources_map_runbook.md new file mode 100644 index 000000000..3eb1d75c6 --- /dev/null +++ b/ai_docs/tests/global_sources_map_runbook.md @@ -0,0 +1,234 @@ +# CLI E2E Runbook: Global Config — sources map ↔ legacy fields interop + +Validates v0.19.16's `sources:` map for global config alongside the legacy +`source:` / `agents_source:` / `extras_source:` top-level fields. Confirms +both formats produce identical CLI behavior, mixed configs prefer the new +format, and fresh `init -g` emits the new shape. + +**Origin**: v0.19.16 global config sources map feature. + +## Scope + +- Fresh `init -g` writes only the new `sources:` map (no legacy fields) +- Legacy-only config (`source:` / `agents_source:`) continues to work +- New-format config (`sources.skills:` / `sources.agents:`) works +- Mixed config: `sources.skills` wins over the corresponding legacy field +- `ss list` / `ss sync` resolve paths via `EffectiveSkillsSource()` / + `EffectiveAgentsSource()` regardless of format +- `ss sync` validation rejects configs with neither `source` nor `sources.skills` + +## Environment + +Run inside devcontainer. mdproof setup auto-runs +`ss init -g --force --all-targets --no-git --no-skill` before each step, so +each step starts with a fresh default config in the new format. + +## Steps + +### 1. Fresh init -g writes the new sources map and omits legacy fields + +```bash +config=~/.config/skillshare/config.yaml +echo "sources_map=$(grep -c '^sources:' $config)" +echo "skills_subkey=$(grep -c '^ skills:' $config)" +echo "agents_subkey=$(grep -c '^ agents:' $config)" +echo "legacy_source=$(grep -c '^source:' $config)" +echo "legacy_agents_source=$(grep -c '^agents_source:' $config)" +echo "legacy_extras_source=$(grep -c '^extras_source:' $config)" +``` + +Expected: +- exit_code: 0 +- sources_map=1 +- skills_subkey=1 +- agents_subkey=1 +- legacy_source=0 +- legacy_agents_source=0 +- legacy_extras_source=0 + +### 2. Legacy-only config — ss list resolves skills via top-level source + +```bash +SRC=/tmp/sources-runbook-legacy-skills +AGENTS=/tmp/sources-runbook-legacy-agents +rm -rf $SRC $AGENTS +mkdir -p $SRC/test-legacy $AGENTS + +cat > ~/.config/skillshare/config.yaml < $SRC/test-legacy/SKILL.md <<'SKILL' +--- +name: test-legacy +description: Skill placed under legacy source path +--- +# Body +SKILL + +ss list --json +``` + +Expected: +- exit_code: 0 +- jq: [.[].name] | contains(["test-legacy"]) + +### 3. New sources map — ss list resolves skills via sources.skills + +```bash +SRC=/tmp/sources-runbook-new-skills +AGENTS=/tmp/sources-runbook-new-agents +rm -rf $SRC $AGENTS +mkdir -p $SRC/test-new $AGENTS + +cat > ~/.config/skillshare/config.yaml < $SRC/test-new/SKILL.md <<'SKILL' +--- +name: test-new +description: Skill placed under sources.skills path +--- +# Body +SKILL + +ss list --json +``` + +Expected: +- exit_code: 0 +- jq: [.[].name] | contains(["test-new"]) + +### 4. Mixed config — sources.skills wins over legacy source + +```bash +WIN=/tmp/sources-runbook-win +LOSE=/tmp/sources-runbook-lose +AGENTS=/tmp/sources-runbook-mixed-agents +rm -rf $WIN $LOSE $AGENTS +mkdir -p $WIN/winning $LOSE/losing $AGENTS + +cat > ~/.config/skillshare/config.yaml < $WIN/winning/SKILL.md <<'SKILL' +--- +name: winning +description: Skill under sources.skills (should appear) +--- +# Body +SKILL + +cat > $LOSE/losing/SKILL.md <<'SKILL' +--- +name: losing +description: Skill under legacy source (should NOT appear) +--- +# Body +SKILL + +ss list --json +``` + +Expected: +- exit_code: 0 +- jq: [.[].name] | contains(["winning"]) +- jq: ([.[].name] | contains(["losing"])) | not + +### 5. Sync works with new sources map and a custom target path + +```bash +SRC=/tmp/sources-runbook-sync-skills +AGENTS=/tmp/sources-runbook-sync-agents +TARGET=/tmp/sources-runbook-sync-target +rm -rf $SRC $AGENTS $TARGET +mkdir -p $SRC/sync-skill $AGENTS $TARGET + +cat > ~/.config/skillshare/config.yaml < $SRC/sync-skill/SKILL.md <<'SKILL' +--- +name: sync-skill +description: Verify sync writes to target when sources.skills is set +--- +# Body +SKILL + +ss sync --json +``` + +Expected: +- exit_code: 0 +- jq: .linked == 1 +- jq: .details[0].name == "claude" + +### 6. Sync rejects configs that set neither source nor sources.skills + +```bash +AGENTS=/tmp/sources-runbook-empty-agents +rm -rf $AGENTS +mkdir -p $AGENTS + +# Config with NEITHER legacy `source:` NOR `sources.skills:` +cat > ~/.config/skillshare/config.yaml <&1 +EXIT=$? +echo "exit_status=$EXIT" +``` + +Expected: +- exit_status=1 +- source path is empty + +## Pass Criteria + +All six steps pass. Together they prove that the global config sources map +shipped in v0.19.16 is wired through the runtime helpers, the legacy +top-level fields still work unchanged, mixed configs prefer the new keys, +and the validation surface still rejects configs that explicitly omit both +shapes.