mirror of
https://github.com/runkids/skillshare.git
synced 2026-11-18 18:04:17 +00:00
test(e2e): add global sources map ↔ legacy fields interop runbook
Six-step mdproof runbook exercising the v0.19.16 global sources map: 1. Fresh init -g writes sources: map (no legacy source/agents_source/extras_source) 2. Legacy-only config (source: + agents_source:) — ss list still works 3. New sources map (sources.skills + sources.agents) — ss list works 4. Mixed config — sources.skills wins over legacy source 5. ss sync writes to a custom target with sources.skills set 6. ss sync rejects configs that set neither source nor sources.skills Runs in 0.9s and prevents regressions on either format. Also: record an mdproof lessons-learned entry for `ss sync --json` output — it has no top-level `.errors` field; assert `.linked` / `.details` or rely on exit_code instead.
This commit is contained in:
@@ -92,3 +92,10 @@
|
||||
- **Discovery**: In a full run of 22 runbooks, the first clean run had 2 failures. But the second run (same ssenv, re-running mdproof) had 10 failures — all due to accumulated state from the first run (1257 trash items, stale registry entries, leftover extras). Even within a single full run, alphabetically-later runbooks can fail because of state left by earlier ones
|
||||
- **Fix**: (1) Each runbook should clean up its own footprint in Step 1 (rm -rf /tmp/ paths, clear trash, reset config sections). (2) For authoritative results, run each runbook in its own fresh ssenv. (3) Full-directory runs are useful as smoke tests but failures should be re-verified in isolation before treating them as real bugs
|
||||
- **Runbooks affected**: extras_refactor_json (file_count mismatch from extras_commands leftovers), gitlab_hosts_config (trash from previous runbooks broke go test), registry_yaml_split (/tmp/ state from prior runs)
|
||||
|
||||
### [assertion] `ss sync --json` has no `.errors` field — use `.linked` / `.details`
|
||||
|
||||
- **Context**: `global_sources_map_runbook` step 5 asserted `jq: .errors == 0` on `ss sync --json` to verify a successful sync. The assertion failed because the output schema doesn't include an `errors` key at all (jq returns `null`, which compares unequal to `0`)
|
||||
- **Discovery**: `ss sync --json` emits `{targets, linked, local, updated, pruned, ignored_count, ignored_skills, dry_run, duration, details[], context_cost}`. Failure is signalled by the process exit code and per-target details, not by a top-level `errors` field. Many AI-generated assertions assume a generic `.errors == 0` pattern from REST APIs — that pattern doesn't apply here
|
||||
- **Fix**: Assert positive evidence of the sync outcome instead. Examples: `jq: .linked == <N>` for "N skills got linked", `jq: .details[0].name == "<target>"` for target-level reporting, `jq: .updated == 0` for "nothing changed in this re-run". Pair with `exit_code: 0` for the failure-case guard
|
||||
- **Runbooks affected**: global_sources_map_runbook.md (step 5 fixed before merging)
|
||||
|
||||
@@ -0,0 +1,234 @@
|
||||
# CLI E2E Runbook: Global Config — sources map ↔ legacy fields interop
|
||||
|
||||
Validates v0.19.16's `sources:` map for global config alongside the legacy
|
||||
`source:` / `agents_source:` / `extras_source:` top-level fields. Confirms
|
||||
both formats produce identical CLI behavior, mixed configs prefer the new
|
||||
format, and fresh `init -g` emits the new shape.
|
||||
|
||||
**Origin**: v0.19.16 global config sources map feature.
|
||||
|
||||
## Scope
|
||||
|
||||
- Fresh `init -g` writes only the new `sources:` map (no legacy fields)
|
||||
- Legacy-only config (`source:` / `agents_source:`) continues to work
|
||||
- New-format config (`sources.skills:` / `sources.agents:`) works
|
||||
- Mixed config: `sources.skills` wins over the corresponding legacy field
|
||||
- `ss list` / `ss sync` resolve paths via `EffectiveSkillsSource()` /
|
||||
`EffectiveAgentsSource()` regardless of format
|
||||
- `ss sync` validation rejects configs with neither `source` nor `sources.skills`
|
||||
|
||||
## Environment
|
||||
|
||||
Run inside devcontainer. mdproof setup auto-runs
|
||||
`ss init -g --force --all-targets --no-git --no-skill` before each step, so
|
||||
each step starts with a fresh default config in the new format.
|
||||
|
||||
## Steps
|
||||
|
||||
### 1. Fresh init -g writes the new sources map and omits legacy fields
|
||||
|
||||
```bash
|
||||
config=~/.config/skillshare/config.yaml
|
||||
echo "sources_map=$(grep -c '^sources:' $config)"
|
||||
echo "skills_subkey=$(grep -c '^ skills:' $config)"
|
||||
echo "agents_subkey=$(grep -c '^ agents:' $config)"
|
||||
echo "legacy_source=$(grep -c '^source:' $config)"
|
||||
echo "legacy_agents_source=$(grep -c '^agents_source:' $config)"
|
||||
echo "legacy_extras_source=$(grep -c '^extras_source:' $config)"
|
||||
```
|
||||
|
||||
Expected:
|
||||
- exit_code: 0
|
||||
- sources_map=1
|
||||
- skills_subkey=1
|
||||
- agents_subkey=1
|
||||
- legacy_source=0
|
||||
- legacy_agents_source=0
|
||||
- legacy_extras_source=0
|
||||
|
||||
### 2. Legacy-only config — ss list resolves skills via top-level source
|
||||
|
||||
```bash
|
||||
SRC=/tmp/sources-runbook-legacy-skills
|
||||
AGENTS=/tmp/sources-runbook-legacy-agents
|
||||
rm -rf $SRC $AGENTS
|
||||
mkdir -p $SRC/test-legacy $AGENTS
|
||||
|
||||
cat > ~/.config/skillshare/config.yaml <<EOF
|
||||
# yaml-language-server: \$schema=https://raw.githubusercontent.com/runkids/skillshare/main/schemas/config.schema.json
|
||||
source: $SRC
|
||||
agents_source: $AGENTS
|
||||
mode: merge
|
||||
targets:
|
||||
claude:
|
||||
skills:
|
||||
path: ~/.claude/skills
|
||||
EOF
|
||||
|
||||
cat > $SRC/test-legacy/SKILL.md <<'SKILL'
|
||||
---
|
||||
name: test-legacy
|
||||
description: Skill placed under legacy source path
|
||||
---
|
||||
# Body
|
||||
SKILL
|
||||
|
||||
ss list --json
|
||||
```
|
||||
|
||||
Expected:
|
||||
- exit_code: 0
|
||||
- jq: [.[].name] | contains(["test-legacy"])
|
||||
|
||||
### 3. New sources map — ss list resolves skills via sources.skills
|
||||
|
||||
```bash
|
||||
SRC=/tmp/sources-runbook-new-skills
|
||||
AGENTS=/tmp/sources-runbook-new-agents
|
||||
rm -rf $SRC $AGENTS
|
||||
mkdir -p $SRC/test-new $AGENTS
|
||||
|
||||
cat > ~/.config/skillshare/config.yaml <<EOF
|
||||
# yaml-language-server: \$schema=https://raw.githubusercontent.com/runkids/skillshare/main/schemas/config.schema.json
|
||||
sources:
|
||||
skills: $SRC
|
||||
agents: $AGENTS
|
||||
mode: merge
|
||||
targets:
|
||||
claude:
|
||||
skills:
|
||||
path: ~/.claude/skills
|
||||
EOF
|
||||
|
||||
cat > $SRC/test-new/SKILL.md <<'SKILL'
|
||||
---
|
||||
name: test-new
|
||||
description: Skill placed under sources.skills path
|
||||
---
|
||||
# Body
|
||||
SKILL
|
||||
|
||||
ss list --json
|
||||
```
|
||||
|
||||
Expected:
|
||||
- exit_code: 0
|
||||
- jq: [.[].name] | contains(["test-new"])
|
||||
|
||||
### 4. Mixed config — sources.skills wins over legacy source
|
||||
|
||||
```bash
|
||||
WIN=/tmp/sources-runbook-win
|
||||
LOSE=/tmp/sources-runbook-lose
|
||||
AGENTS=/tmp/sources-runbook-mixed-agents
|
||||
rm -rf $WIN $LOSE $AGENTS
|
||||
mkdir -p $WIN/winning $LOSE/losing $AGENTS
|
||||
|
||||
cat > ~/.config/skillshare/config.yaml <<EOF
|
||||
# yaml-language-server: \$schema=https://raw.githubusercontent.com/runkids/skillshare/main/schemas/config.schema.json
|
||||
source: $LOSE
|
||||
agents_source: $AGENTS
|
||||
sources:
|
||||
skills: $WIN
|
||||
mode: merge
|
||||
targets:
|
||||
claude:
|
||||
skills:
|
||||
path: ~/.claude/skills
|
||||
EOF
|
||||
|
||||
cat > $WIN/winning/SKILL.md <<'SKILL'
|
||||
---
|
||||
name: winning
|
||||
description: Skill under sources.skills (should appear)
|
||||
---
|
||||
# Body
|
||||
SKILL
|
||||
|
||||
cat > $LOSE/losing/SKILL.md <<'SKILL'
|
||||
---
|
||||
name: losing
|
||||
description: Skill under legacy source (should NOT appear)
|
||||
---
|
||||
# Body
|
||||
SKILL
|
||||
|
||||
ss list --json
|
||||
```
|
||||
|
||||
Expected:
|
||||
- exit_code: 0
|
||||
- jq: [.[].name] | contains(["winning"])
|
||||
- jq: ([.[].name] | contains(["losing"])) | not
|
||||
|
||||
### 5. Sync works with new sources map and a custom target path
|
||||
|
||||
```bash
|
||||
SRC=/tmp/sources-runbook-sync-skills
|
||||
AGENTS=/tmp/sources-runbook-sync-agents
|
||||
TARGET=/tmp/sources-runbook-sync-target
|
||||
rm -rf $SRC $AGENTS $TARGET
|
||||
mkdir -p $SRC/sync-skill $AGENTS $TARGET
|
||||
|
||||
cat > ~/.config/skillshare/config.yaml <<EOF
|
||||
# yaml-language-server: \$schema=https://raw.githubusercontent.com/runkids/skillshare/main/schemas/config.schema.json
|
||||
sources:
|
||||
skills: $SRC
|
||||
agents: $AGENTS
|
||||
mode: merge
|
||||
targets:
|
||||
claude:
|
||||
skills:
|
||||
path: $TARGET
|
||||
EOF
|
||||
|
||||
cat > $SRC/sync-skill/SKILL.md <<'SKILL'
|
||||
---
|
||||
name: sync-skill
|
||||
description: Verify sync writes to target when sources.skills is set
|
||||
---
|
||||
# Body
|
||||
SKILL
|
||||
|
||||
ss sync --json
|
||||
```
|
||||
|
||||
Expected:
|
||||
- exit_code: 0
|
||||
- jq: .linked == 1
|
||||
- jq: .details[0].name == "claude"
|
||||
|
||||
### 6. Sync rejects configs that set neither source nor sources.skills
|
||||
|
||||
```bash
|
||||
AGENTS=/tmp/sources-runbook-empty-agents
|
||||
rm -rf $AGENTS
|
||||
mkdir -p $AGENTS
|
||||
|
||||
# Config with NEITHER legacy `source:` NOR `sources.skills:`
|
||||
cat > ~/.config/skillshare/config.yaml <<EOF
|
||||
# yaml-language-server: \$schema=https://raw.githubusercontent.com/runkids/skillshare/main/schemas/config.schema.json
|
||||
agents_source: $AGENTS
|
||||
mode: merge
|
||||
targets:
|
||||
claude:
|
||||
skills:
|
||||
path: ~/.claude/skills
|
||||
EOF
|
||||
|
||||
ss sync 2>&1
|
||||
EXIT=$?
|
||||
echo "exit_status=$EXIT"
|
||||
```
|
||||
|
||||
Expected:
|
||||
- exit_status=1
|
||||
- source path is empty
|
||||
|
||||
## Pass Criteria
|
||||
|
||||
All six steps pass. Together they prove that the global config sources map
|
||||
shipped in v0.19.16 is wired through the runtime helpers, the legacy
|
||||
top-level fields still work unchanged, mixed configs prefer the new keys,
|
||||
and the validation surface still rejects configs that explicitly omit both
|
||||
shapes.
|
||||
Reference in New Issue
Block a user