test(e2e): add global sources map ↔ legacy fields interop runbook

Six-step mdproof runbook exercising the v0.19.16 global sources map:

1. Fresh init -g writes sources: map (no legacy source/agents_source/extras_source)
2. Legacy-only config (source: + agents_source:) — ss list still works
3. New sources map (sources.skills + sources.agents) — ss list works
4. Mixed config — sources.skills wins over legacy source
5. ss sync writes to a custom target with sources.skills set
6. ss sync rejects configs that set neither source nor sources.skills

Runs in 0.9s and prevents regressions on either format.

Also: record an mdproof lessons-learned entry for `ss sync --json`
output — it has no top-level `.errors` field; assert `.linked` /
`.details` or rely on exit_code instead.
This commit is contained in:
Willie
2026-05-21 15:08:40 +08:00
parent 8e0117a29d
commit 7de9fa4fb0
2 changed files with 241 additions and 0 deletions
+7
View File
@@ -92,3 +92,10 @@
- **Discovery**: In a full run of 22 runbooks, the first clean run had 2 failures. But the second run (same ssenv, re-running mdproof) had 10 failures — all due to accumulated state from the first run (1257 trash items, stale registry entries, leftover extras). Even within a single full run, alphabetically-later runbooks can fail because of state left by earlier ones
- **Fix**: (1) Each runbook should clean up its own footprint in Step 1 (rm -rf /tmp/ paths, clear trash, reset config sections). (2) For authoritative results, run each runbook in its own fresh ssenv. (3) Full-directory runs are useful as smoke tests but failures should be re-verified in isolation before treating them as real bugs
- **Runbooks affected**: extras_refactor_json (file_count mismatch from extras_commands leftovers), gitlab_hosts_config (trash from previous runbooks broke go test), registry_yaml_split (/tmp/ state from prior runs)
### [assertion] `ss sync --json` has no `.errors` field — use `.linked` / `.details`
- **Context**: `global_sources_map_runbook` step 5 asserted `jq: .errors == 0` on `ss sync --json` to verify a successful sync. The assertion failed because the output schema doesn't include an `errors` key at all (jq returns `null`, which compares unequal to `0`)
- **Discovery**: `ss sync --json` emits `{targets, linked, local, updated, pruned, ignored_count, ignored_skills, dry_run, duration, details[], context_cost}`. Failure is signalled by the process exit code and per-target details, not by a top-level `errors` field. Many AI-generated assertions assume a generic `.errors == 0` pattern from REST APIs — that pattern doesn't apply here
- **Fix**: Assert positive evidence of the sync outcome instead. Examples: `jq: .linked == <N>` for "N skills got linked", `jq: .details[0].name == "<target>"` for target-level reporting, `jq: .updated == 0` for "nothing changed in this re-run". Pair with `exit_code: 0` for the failure-case guard
- **Runbooks affected**: global_sources_map_runbook.md (step 5 fixed before merging)
+234
View File
@@ -0,0 +1,234 @@
# CLI E2E Runbook: Global Config — sources map ↔ legacy fields interop
Validates v0.19.16's `sources:` map for global config alongside the legacy
`source:` / `agents_source:` / `extras_source:` top-level fields. Confirms
both formats produce identical CLI behavior, mixed configs prefer the new
format, and fresh `init -g` emits the new shape.
**Origin**: v0.19.16 global config sources map feature.
## Scope
- Fresh `init -g` writes only the new `sources:` map (no legacy fields)
- Legacy-only config (`source:` / `agents_source:`) continues to work
- New-format config (`sources.skills:` / `sources.agents:`) works
- Mixed config: `sources.skills` wins over the corresponding legacy field
- `ss list` / `ss sync` resolve paths via `EffectiveSkillsSource()` /
`EffectiveAgentsSource()` regardless of format
- `ss sync` validation rejects configs with neither `source` nor `sources.skills`
## Environment
Run inside devcontainer. mdproof setup auto-runs
`ss init -g --force --all-targets --no-git --no-skill` before each step, so
each step starts with a fresh default config in the new format.
## Steps
### 1. Fresh init -g writes the new sources map and omits legacy fields
```bash
config=~/.config/skillshare/config.yaml
echo "sources_map=$(grep -c '^sources:' $config)"
echo "skills_subkey=$(grep -c '^ skills:' $config)"
echo "agents_subkey=$(grep -c '^ agents:' $config)"
echo "legacy_source=$(grep -c '^source:' $config)"
echo "legacy_agents_source=$(grep -c '^agents_source:' $config)"
echo "legacy_extras_source=$(grep -c '^extras_source:' $config)"
```
Expected:
- exit_code: 0
- sources_map=1
- skills_subkey=1
- agents_subkey=1
- legacy_source=0
- legacy_agents_source=0
- legacy_extras_source=0
### 2. Legacy-only config — ss list resolves skills via top-level source
```bash
SRC=/tmp/sources-runbook-legacy-skills
AGENTS=/tmp/sources-runbook-legacy-agents
rm -rf $SRC $AGENTS
mkdir -p $SRC/test-legacy $AGENTS
cat > ~/.config/skillshare/config.yaml <<EOF
# yaml-language-server: \$schema=https://raw.githubusercontent.com/runkids/skillshare/main/schemas/config.schema.json
source: $SRC
agents_source: $AGENTS
mode: merge
targets:
claude:
skills:
path: ~/.claude/skills
EOF
cat > $SRC/test-legacy/SKILL.md <<'SKILL'
---
name: test-legacy
description: Skill placed under legacy source path
---
# Body
SKILL
ss list --json
```
Expected:
- exit_code: 0
- jq: [.[].name] | contains(["test-legacy"])
### 3. New sources map — ss list resolves skills via sources.skills
```bash
SRC=/tmp/sources-runbook-new-skills
AGENTS=/tmp/sources-runbook-new-agents
rm -rf $SRC $AGENTS
mkdir -p $SRC/test-new $AGENTS
cat > ~/.config/skillshare/config.yaml <<EOF
# yaml-language-server: \$schema=https://raw.githubusercontent.com/runkids/skillshare/main/schemas/config.schema.json
sources:
skills: $SRC
agents: $AGENTS
mode: merge
targets:
claude:
skills:
path: ~/.claude/skills
EOF
cat > $SRC/test-new/SKILL.md <<'SKILL'
---
name: test-new
description: Skill placed under sources.skills path
---
# Body
SKILL
ss list --json
```
Expected:
- exit_code: 0
- jq: [.[].name] | contains(["test-new"])
### 4. Mixed config — sources.skills wins over legacy source
```bash
WIN=/tmp/sources-runbook-win
LOSE=/tmp/sources-runbook-lose
AGENTS=/tmp/sources-runbook-mixed-agents
rm -rf $WIN $LOSE $AGENTS
mkdir -p $WIN/winning $LOSE/losing $AGENTS
cat > ~/.config/skillshare/config.yaml <<EOF
# yaml-language-server: \$schema=https://raw.githubusercontent.com/runkids/skillshare/main/schemas/config.schema.json
source: $LOSE
agents_source: $AGENTS
sources:
skills: $WIN
mode: merge
targets:
claude:
skills:
path: ~/.claude/skills
EOF
cat > $WIN/winning/SKILL.md <<'SKILL'
---
name: winning
description: Skill under sources.skills (should appear)
---
# Body
SKILL
cat > $LOSE/losing/SKILL.md <<'SKILL'
---
name: losing
description: Skill under legacy source (should NOT appear)
---
# Body
SKILL
ss list --json
```
Expected:
- exit_code: 0
- jq: [.[].name] | contains(["winning"])
- jq: ([.[].name] | contains(["losing"])) | not
### 5. Sync works with new sources map and a custom target path
```bash
SRC=/tmp/sources-runbook-sync-skills
AGENTS=/tmp/sources-runbook-sync-agents
TARGET=/tmp/sources-runbook-sync-target
rm -rf $SRC $AGENTS $TARGET
mkdir -p $SRC/sync-skill $AGENTS $TARGET
cat > ~/.config/skillshare/config.yaml <<EOF
# yaml-language-server: \$schema=https://raw.githubusercontent.com/runkids/skillshare/main/schemas/config.schema.json
sources:
skills: $SRC
agents: $AGENTS
mode: merge
targets:
claude:
skills:
path: $TARGET
EOF
cat > $SRC/sync-skill/SKILL.md <<'SKILL'
---
name: sync-skill
description: Verify sync writes to target when sources.skills is set
---
# Body
SKILL
ss sync --json
```
Expected:
- exit_code: 0
- jq: .linked == 1
- jq: .details[0].name == "claude"
### 6. Sync rejects configs that set neither source nor sources.skills
```bash
AGENTS=/tmp/sources-runbook-empty-agents
rm -rf $AGENTS
mkdir -p $AGENTS
# Config with NEITHER legacy `source:` NOR `sources.skills:`
cat > ~/.config/skillshare/config.yaml <<EOF
# yaml-language-server: \$schema=https://raw.githubusercontent.com/runkids/skillshare/main/schemas/config.schema.json
agents_source: $AGENTS
mode: merge
targets:
claude:
skills:
path: ~/.claude/skills
EOF
ss sync 2>&1
EXIT=$?
echo "exit_status=$EXIT"
```
Expected:
- exit_status=1
- source path is empty
## Pass Criteria
All six steps pass. Together they prove that the global config sources map
shipped in v0.19.16 is wired through the runtime helpers, the legacy
top-level fields still work unchanged, mixed configs prefer the new keys,
and the validation surface still rejects configs that explicitly omit both
shapes.