diff --git a/website/blog/2026-03-06-e2e-testing-for-ai-agents.md b/website/blog/2026-03-06-e2e-testing-for-ai-agents.md deleted file mode 100644 index 35ea9e11f..000000000 --- a/website/blog/2026-03-06-e2e-testing-for-ai-agents.md +++ /dev/null @@ -1,185 +0,0 @@ ---- -slug: e2e-testing-for-ai-agents -title: "What I Learned Building a Testing Infrastructure for AI-Driven CLI Development" -authors: [runkids] -tags: [testing, ai-agents, devcontainer, e2e] ---- - -Over the past few months building [skillshare](https://github.com/runkids/skillshare), I've been experimenting with letting AI agents (Claude Code, mainly) handle more and more of the development cycle — writing features, fixing bugs, even running verification. - -Along the way I learned some things about what makes AI agents productive (and what doesn't). Sharing here in case it's useful to others building CLI tools. - - - -## The Problem I Kept Hitting - -AI agents write code fast. But "fast" doesn't mean "correct." I kept finding myself manually verifying things the agent said were done — re-running commands, checking files, eyeballing output. The bottleneck wasn't code generation. **It was verification.** - -So I started building infrastructure to let the agent verify its own work. - -## Three Layers of Testing - -``` - E2E (Docker Sandbox) - ───────────────────────── - Integration (testutil.Sandbox) - ───────────────────────────────── - Unit Tests (go test) -``` - -Nothing revolutionary here. The key was making **every layer runnable by the agent itself** — no human in the loop for verification. - -### Layer 1: Unit Tests - -Standard `go test` for pure functions. AI agents can write and run these instantly. - -```bash -go test ./internal/sync/... -go test ./internal/audit/... -``` - -### Layer 2: Integration Tests with `testutil.Sandbox` - -This is where things started clicking. `testutil.Sandbox` creates a **fully isolated fake HOME directory** for each test — complete with `.claude/`, `.cursor/`, and 50+ other AI CLI target directories. - -```go -sb := testutil.NewSandbox(t) -defer sb.Cleanup() - -// Create a skill in the source directory -sb.CreateSkill("my-skill", map[string]string{ - "SKILL.md": "---\nname: my-skill\n---\n# Content", -}) - -// Run the CLI and verify -result := sb.RunCLI("sync") -result.AssertSuccess() -``` - -Why this matters for AI agents: - -- **No side effects** — each test gets a fresh environment, agents can't break the host -- **Fast feedback** — `go test -run TestXxx` takes seconds -- **Self-contained** — no external dependencies, no network, no cleanup needed - -The agent writes a feature, adds an integration test, runs it, gets immediate pass/fail. No human needed. - -### Layer 3: E2E with Docker Sandbox - -Integration tests run on the host, which may have a different environment than real users. So the top layer is a **Docker-based devcontainer** — a clean Debian environment that simulates a real user's first experience. - -```bash -make devc # Start devcontainer + enter shell (one step) -make devc-reset # Full reset if something breaks -``` - -Inside the devcontainer, the AI agent can run any command in complete isolation — install skills from Git repos, sync to targets, run the full CLI — without any risk to the host environment. - -## The Thing I Didn't Expect: Runbooks - -I started writing **E2E runbooks** — step-by-step test scripts in `ai_docs/tests/` — mostly for my own documentation. Turns out they're exactly what AI agents need: - -```markdown -## Test: Install a tracked skill - -### Setup -1. Start with a clean environment - -### Steps -1. Run: `skillshare install runkids/skillshare` -2. Verify: skill directory exists in `~/.config/skillshare/skills/_runkids__skillshare/` -3. Verify: SKILL.md contains expected frontmatter -4. Run: `skillshare list` -5. Verify: output shows the installed skill with [tracked] badge -6. Run: `skillshare sync` -7. Verify: symlinks exist in all enabled targets - -### Expected Result -- Skill is installed, tracked, and synced to all targets -``` - -Explicit steps + expected results = no ambiguity. The agent stops guessing and just follows the script. **The runbook became the spec.** - -## Eating Our Own Dog Food - -The part I'm most happy about. The runbooks and devcontainer don't work by magic — the AI agent needs to know _how_ to use them. That's where **built-in skills** come in. - -skillshare uses its own skill system to teach AI agents the entire workflow: - -``` -.skillshare/skills/ - devcontainer/SKILL.md # How to enter, run commands, use ssenv isolation - cli-e2e-test/SKILL.md # How to pick a runbook, execute it, report results -``` - -The **devcontainer skill** teaches the agent: -- All CLI execution must happen inside the container (`docker exec`, never on host) -- Use `ssenv` for HOME isolation — each test gets a clean `~/.config/skillshare/` -- Zero-rebuild workflow — edit code on host, `docker exec` picks up changes instantly -- Common mistakes to avoid (wrong arch, forgetting `cd /workspace`, etc.) - -The **cli-e2e-test skill** orchestrates the full E2E flow: -- Phase 0: Check devcontainer is running -- Phase 1: Detect which runbooks are relevant to recent code changes -- Phase 2: Let the agent pick or auto-generate a test -- Phase 3: Execute steps with `ssenv` isolation, verify each assertion -- Phase 4: Report results, clean up, and run a retrospective - -A skill management tool that uses its own skill system to teach AI agents how to develop and test itself. When I improve the testing workflow, I update the skill. Next time the agent picks up a task, it gets the updated instructions. No stale READMEs. - -## The Full Loop - -Here's the development cycle the agent runs autonomously: - -``` -Spec / Issue - | - v -Write code - | - v -Run unit tests (go test ./internal/...) - | - v -Run integration tests (go test ./tests/integration/ -run TestXxx) - | - v -Enter devcontainer (make devc) - | - v -Execute E2E runbook - | - v -All green? -> Open PR -``` - -My job becomes: - -1. **Write the spec** — what should the feature do? -2. **Write the runbook** — how do we verify it works? -3. **Review the PR** — does the implementation make sense? - -## Why CLIs Make This Possible - -This workflow works _because_ skillshare is a CLI: - -- **Text in, text out** — AI agents can read command output and verify assertions naturally -- **Deterministic behavior** — same input produces same output, no UI flakiness -- **Container-friendly** — CLIs run perfectly in Docker, no display server or browser needed -- **Composable** — agents can chain commands, pipe output, and script complex scenarios - -GUIs need screenshot comparison, browser automation, and flaky selectors. CLIs need `assert output contains "success"`. The verification surface is fundamentally simpler. - -## What I'd Tell Someone Starting Out - -1. **Invest in test isolation early** — agents need a sandbox they can't break -2. **Write runbooks** — explicit verification beats vague expectations -3. **Teach agents your workflow via skills/context files** — not just your API -4. **Use containers** — cheaper than debugging environment differences -5. **Add `--json` flags** — structured output lets agents verify programmatically - -## Honest Takeaway - -AI agents are fast, confident, and wrong in new ways. I don't assume they write perfect code — I assume they can fix their own mistakes, **as long as the feedback loop is fast enough.** Runbooks, sandboxes, and built-in skills are that feedback loop. - -Still learning and iterating on this. If you're building CLI tools with AI agents, I'd love to hear what's working for you. diff --git a/website/blog/2026-03-12-e2e-tests-now-powered-by-mdproof.md b/website/blog/2026-03-12-e2e-tests-now-powered-by-mdproof.md deleted file mode 100644 index 771a5cfa5..000000000 --- a/website/blog/2026-03-12-e2e-tests-now-powered-by-mdproof.md +++ /dev/null @@ -1,248 +0,0 @@ ---- -slug: e2e-tests-now-powered-by-mdproof -title: "skillshare E2E Tests Are Now Fully Powered by mdproof" -authors: [runkids] -tags: [testing, e2e, mdproof, ai-agents] ---- - -A week ago I wrote about [building testing infrastructure for AI-driven CLI development](/blog/e2e-testing-for-ai-agents). The runbooks I described — Markdown files with step-by-step commands and expected results — turned out to be the most useful piece of that entire infrastructure. - -So useful, in fact, that I extracted the runner into its own tool: [**mdproof**](https://github.com/runkids/mdproof). - -As of today, all 22 of skillshare's E2E runbooks run through mdproof. The built-in runner is gone. - - - -## From Informal Runbooks to a Real Test Runner - -The original runbooks were just documentation. They looked like this: - -```markdown -## Steps -1. Run: `skillshare install runkids/skillshare` -2. Verify: skill directory exists -3. Verify: SKILL.md contains expected frontmatter -``` - -A human (or an AI agent) would read the steps and manually execute them. That worked, but it was slow and error-prone. The agent had to interpret "verify: skill directory exists" and figure out the right `test -d` command on its own. - -What I wanted was something that could **execute the commands and check the assertions automatically** — but still look like a readable Markdown document. - -## What mdproof Does - -mdproof treats Markdown files as executable test runbooks. Write your test as a document, run it as a real test: - -````markdown -### Step 2: First init - -```bash -ss init --no-copy --targets claude,cursor --mode merge --no-git --no-skill -``` - -Expected: -- exit_code: 0 -- Initialized successfully -- regex: Config:.*\.config/skillshare/config\.yaml -```` - -That's an actual step from skillshare's [`first_use_cli_e2e_runbook.md`](https://github.com/runkids/skillshare/blob/main/ai_docs/tests/first_use_cli_e2e_runbook.md). mdproof parses it, runs the bash block, and checks each assertion. No test framework. No assertion library. The test IS the document. - -``` -$ mdproof ai_docs/tests/first_use_cli_e2e_runbook.md - - ✓ first_use_cli_e2e_runbook.md - ────────────────────────────────────────────────── - ✓ Step 0 Verify command entrypoint 12ms - ✓ Step 1 Create isolated first-use HOME 3ms - ✓ Step 2 First init 84ms - ✓ Step 3 Verify config was created 8ms - ✓ Step 4 Add one demo skill to source 4ms - ✓ Step 5 First sync 62ms - ✓ Step 6 Verify skill reached both targets 5ms - ✓ Step 7 Per-target compatibility tuning 94ms - ✓ Step 8 Run doctor 31ms - ────────────────────────────────────────────────── - 9/9 passed 303ms -``` - -## Four Assertion Types - -Through building skillshare's test suite, four assertion patterns emerged naturally. mdproof supports all of them: - -### Substring + Negation - -The simplest form. Check if output contains (or doesn't contain) a string: - -```markdown -Expected: -- Initialized successfully -- config_created=yes -- Not Exec format error -``` - -### Exit Code - -Explicit exit code checking — essential for CLI tools: - -```markdown -Expected: -- exit_code: 0 -``` - -### Regex - -Pattern matching with multiline mode (`(?m)`) enabled by default: - -```markdown -Expected: -- regex: v(dev|\d+\.\d+) -- regex: Config:.*\.config/skillshare/config\.yaml -``` - -### jq (JSON assertions) - -This is where it gets powerful. skillshare's `--json` flags produce structured output, and `jq:` assertions let you validate the structure precisely: - -````markdown -### Step 2: status --json outputs pure JSON - -```bash -OUTPUT=$(ss status --json) -echo "$OUTPUT" | jq -e ".skill_count >= 2" && echo "FIELD_CHECK=OK" -echo "$OUTPUT" | jq -e ".targets | length >= 1" && echo "TARGETS=OK" -FIRST=$(echo "$OUTPUT" | head -c1) -[ "$FIRST" = "{" ] && echo "PURE_JSON=OK" -``` - -Expected: -- exit_code: 0 -- FIELD_CHECK=OK -- TARGETS=OK -- PURE_JSON=OK -```` - -This pattern — run `jq -e` inside the bash block, echo a sentinel, assert on the sentinel — emerged from skillshare's [JSON output purity runbook](https://github.com/runkids/skillshare/blob/main/ai_docs/tests/json_output_purity_runbook.md). It's ugly but reliable: if the `jq` expression fails, the sentinel never prints, and the assertion catches it. - -## Lifecycle Hooks - -Real E2E tests need setup and teardown. mdproof supports a `runbook.json` config that runs hooks automatically: - -```json -{ - "build": "cd /workspace && make build && /workspace/.devcontainer/ensure-mdproof.sh", - "setup": "ss init -g --no-copy --all-targets --no-git --no-skill --force", - "teardown": "ss uninstall --all --force", - "timeout": "5m" -} -``` - -- **build**: Runs once before all runbooks (compile the binary) -- **setup**: Runs before each runbook (clean slate) -- **teardown**: Runs after each runbook (cleanup) -- **timeout**: Global safety net - -This means runbook files stay focused on the actual test logic. No boilerplate. - -## JSON Reports for AI Agents - -The whole point of extracting mdproof was to close the agent feedback loop. `--report json` outputs structured results that agents can parse directly: - -```json -{ - "version": "1", - "runbook": "first_use_cli_e2e_runbook.md", - "duration_ms": 303, - "summary": { "total": 9, "passed": 9, "failed": 0, "skipped": 0 }, - "steps": [ - { - "step": { "number": 0, "title": "Verify command entrypoint", "command": "..." }, - "status": "passed", - "exit_code": 0, - "stdout": "skillshare version vdev (...)", - "stderr": "" - } - ] -} -``` - -Agent writes code → agent runs `mdproof --report json` → agent reads the JSON → agent fixes failures → re-run. No human interprets terminal colors. No regex on test output. The agent gets exactly the data it needs. - -## Why I Extracted It - -The runbook runner started as ~200 lines inside skillshare's codebase. Over three days (March 9–11) it grew features that had nothing to do with skillshare: - -- Persistent bash sessions (variables survive across steps) -- Typed assertion engine (exit_code, regex, jq, substring) -- Lifecycle hooks (build/setup/teardown) -- Selective step execution (`--steps 3,5` or `--from 4`) -- Retry and depends directives -- Container-only safety checks -- JSON and JUnit XML reporting - -At that point it was clear: **this isn't a skillshare feature — it's a testing tool.** So I pulled it out into [runkids/mdproof](https://github.com/runkids/mdproof) and replaced the built-in runner with a single dependency. - -The migration commit was one line of real logic: swap the executor. - -## What skillshare's Test Suite Looks Like Now - -22 runbooks covering the full CLI surface: - -| Category | Runbooks | Example | -|----------|----------|---------| -| First-time user flow | 1 | `first_use_cli_e2e_runbook.md` | -| JSON output purity | 2 | `json_output_purity_runbook.md`, `extras_refactor_json_runbook.md` | -| Security audit | 1 | `audit_output_antigravity_runbook.md` | -| Install/update | 2 | `issue46_install_update_optimization_runbook.md` | -| Sync behavior | 3 | `symlinked_dir_sync_runbook.md`, `sync_extras_runbook.md` | -| URL parsing | 3 | `gitlab_subgroup_url_parse_runbook.md`, `azure_devops_url_parse_runbook.md` | -| Config features | 2 | `gitlab_hosts_config_runbook.md`, `registry_yaml_split_runbook.md` | -| Cleanup operations | 3 | `uninstall_sync_orphan_runbook.md`, `uninstall_all_glob_runbook.md` | -| Precision checks | 2 | `check_treehash_precision_runbook.md`, `update_prune_check_stale_runbook.md` | -| Other | 4 | target paths, gitignore perf, precommit hooks, discovery | - -All executed via: - -```bash -mdproof --report json ai_docs/tests/ -``` - -One command. 22 runbooks. Structured results. - -## The Loop - -In the [previous post](/blog/e2e-testing-for-ai-agents) I described a development loop where the agent writes code, runs tests, and verifies its own work. mdproof makes the E2E layer of that loop concrete: - -``` -Agent writes code - ↓ -go test ./... (unit + integration) - ↓ -mdproof --report json ai_docs/tests/relevant_runbook.md - ↓ -Agent reads JSON → all passed? → PR - → failed? → fix and re-run -``` - -The runbooks haven't changed much since that post. What changed is that they're no longer documentation that an agent interprets — they're **executable specifications** that produce machine-readable results. - -## Try It - -If you're building a CLI tool and want executable Markdown tests: - -```bash -curl -fsSL https://raw.githubusercontent.com/runkids/mdproof/main/install.sh | sh -``` - -Write a `test.md`, add some steps with `Expected:` blocks, and run `mdproof test.md`. That's it. - -If you use skillshare, mdproof is already available as a skill: - -```bash -skillshare install runkids/mdproof -``` - -Your AI agent gets the full syntax reference and can start writing runbooks immediately. - ---- - -[mdproof on GitHub](https://github.com/runkids/mdproof) · [skillshare's runbook suite](https://github.com/runkids/skillshare/tree/main/ai_docs/tests) diff --git a/website/blog/2026-04-05-multi-account-claude-code.md b/website/blog/2026-04-05-multi-account-claude-code.md index 89ccc0bed..08271aaf1 100644 --- a/website/blog/2026-04-05-multi-account-claude-code.md +++ b/website/blog/2026-04-05-multi-account-claude-code.md @@ -281,7 +281,7 @@ This opens a dashboard at `localhost:19420` where you can: - Run sync, audit, and check operations visually - Edit target include/exclude filters without touching YAML -![skillshare UI dashboard](/static/img/web-dashboard-demo2.png) +![skillshare UI dashboard](/img/web-dashboard-demo2.png) Everything you can do in the CLI, you can do in the UI — including setting up the multi-account extras config described above.