mirror of
https://github.com/runkids/skillshare.git
synced 2026-11-19 02:17:09 +00:00
fix(blog): correct image path and remove outdated posts
Fix Docusaurus build error: /static/img/ → /img/ for markdown images. Remove e2e-testing-for-ai-agents and e2e-tests-now-powered-by-mdproof blog posts.
This commit is contained in:
@@ -1,185 +0,0 @@
|
||||
---
|
||||
slug: e2e-testing-for-ai-agents
|
||||
title: "What I Learned Building a Testing Infrastructure for AI-Driven CLI Development"
|
||||
authors: [runkids]
|
||||
tags: [testing, ai-agents, devcontainer, e2e]
|
||||
---
|
||||
|
||||
Over the past few months building [skillshare](https://github.com/runkids/skillshare), I've been experimenting with letting AI agents (Claude Code, mainly) handle more and more of the development cycle — writing features, fixing bugs, even running verification.
|
||||
|
||||
Along the way I learned some things about what makes AI agents productive (and what doesn't). Sharing here in case it's useful to others building CLI tools.
|
||||
|
||||
<!-- truncate -->
|
||||
|
||||
## The Problem I Kept Hitting
|
||||
|
||||
AI agents write code fast. But "fast" doesn't mean "correct." I kept finding myself manually verifying things the agent said were done — re-running commands, checking files, eyeballing output. The bottleneck wasn't code generation. **It was verification.**
|
||||
|
||||
So I started building infrastructure to let the agent verify its own work.
|
||||
|
||||
## Three Layers of Testing
|
||||
|
||||
```
|
||||
E2E (Docker Sandbox)
|
||||
─────────────────────────
|
||||
Integration (testutil.Sandbox)
|
||||
─────────────────────────────────
|
||||
Unit Tests (go test)
|
||||
```
|
||||
|
||||
Nothing revolutionary here. The key was making **every layer runnable by the agent itself** — no human in the loop for verification.
|
||||
|
||||
### Layer 1: Unit Tests
|
||||
|
||||
Standard `go test` for pure functions. AI agents can write and run these instantly.
|
||||
|
||||
```bash
|
||||
go test ./internal/sync/...
|
||||
go test ./internal/audit/...
|
||||
```
|
||||
|
||||
### Layer 2: Integration Tests with `testutil.Sandbox`
|
||||
|
||||
This is where things started clicking. `testutil.Sandbox` creates a **fully isolated fake HOME directory** for each test — complete with `.claude/`, `.cursor/`, and 50+ other AI CLI target directories.
|
||||
|
||||
```go
|
||||
sb := testutil.NewSandbox(t)
|
||||
defer sb.Cleanup()
|
||||
|
||||
// Create a skill in the source directory
|
||||
sb.CreateSkill("my-skill", map[string]string{
|
||||
"SKILL.md": "---\nname: my-skill\n---\n# Content",
|
||||
})
|
||||
|
||||
// Run the CLI and verify
|
||||
result := sb.RunCLI("sync")
|
||||
result.AssertSuccess()
|
||||
```
|
||||
|
||||
Why this matters for AI agents:
|
||||
|
||||
- **No side effects** — each test gets a fresh environment, agents can't break the host
|
||||
- **Fast feedback** — `go test -run TestXxx` takes seconds
|
||||
- **Self-contained** — no external dependencies, no network, no cleanup needed
|
||||
|
||||
The agent writes a feature, adds an integration test, runs it, gets immediate pass/fail. No human needed.
|
||||
|
||||
### Layer 3: E2E with Docker Sandbox
|
||||
|
||||
Integration tests run on the host, which may have a different environment than real users. So the top layer is a **Docker-based devcontainer** — a clean Debian environment that simulates a real user's first experience.
|
||||
|
||||
```bash
|
||||
make devc # Start devcontainer + enter shell (one step)
|
||||
make devc-reset # Full reset if something breaks
|
||||
```
|
||||
|
||||
Inside the devcontainer, the AI agent can run any command in complete isolation — install skills from Git repos, sync to targets, run the full CLI — without any risk to the host environment.
|
||||
|
||||
## The Thing I Didn't Expect: Runbooks
|
||||
|
||||
I started writing **E2E runbooks** — step-by-step test scripts in `ai_docs/tests/` — mostly for my own documentation. Turns out they're exactly what AI agents need:
|
||||
|
||||
```markdown
|
||||
## Test: Install a tracked skill
|
||||
|
||||
### Setup
|
||||
1. Start with a clean environment
|
||||
|
||||
### Steps
|
||||
1. Run: `skillshare install runkids/skillshare`
|
||||
2. Verify: skill directory exists in `~/.config/skillshare/skills/_runkids__skillshare/`
|
||||
3. Verify: SKILL.md contains expected frontmatter
|
||||
4. Run: `skillshare list`
|
||||
5. Verify: output shows the installed skill with [tracked] badge
|
||||
6. Run: `skillshare sync`
|
||||
7. Verify: symlinks exist in all enabled targets
|
||||
|
||||
### Expected Result
|
||||
- Skill is installed, tracked, and synced to all targets
|
||||
```
|
||||
|
||||
Explicit steps + expected results = no ambiguity. The agent stops guessing and just follows the script. **The runbook became the spec.**
|
||||
|
||||
## Eating Our Own Dog Food
|
||||
|
||||
The part I'm most happy about. The runbooks and devcontainer don't work by magic — the AI agent needs to know _how_ to use them. That's where **built-in skills** come in.
|
||||
|
||||
skillshare uses its own skill system to teach AI agents the entire workflow:
|
||||
|
||||
```
|
||||
.skillshare/skills/
|
||||
devcontainer/SKILL.md # How to enter, run commands, use ssenv isolation
|
||||
cli-e2e-test/SKILL.md # How to pick a runbook, execute it, report results
|
||||
```
|
||||
|
||||
The **devcontainer skill** teaches the agent:
|
||||
- All CLI execution must happen inside the container (`docker exec`, never on host)
|
||||
- Use `ssenv` for HOME isolation — each test gets a clean `~/.config/skillshare/`
|
||||
- Zero-rebuild workflow — edit code on host, `docker exec` picks up changes instantly
|
||||
- Common mistakes to avoid (wrong arch, forgetting `cd /workspace`, etc.)
|
||||
|
||||
The **cli-e2e-test skill** orchestrates the full E2E flow:
|
||||
- Phase 0: Check devcontainer is running
|
||||
- Phase 1: Detect which runbooks are relevant to recent code changes
|
||||
- Phase 2: Let the agent pick or auto-generate a test
|
||||
- Phase 3: Execute steps with `ssenv` isolation, verify each assertion
|
||||
- Phase 4: Report results, clean up, and run a retrospective
|
||||
|
||||
A skill management tool that uses its own skill system to teach AI agents how to develop and test itself. When I improve the testing workflow, I update the skill. Next time the agent picks up a task, it gets the updated instructions. No stale READMEs.
|
||||
|
||||
## The Full Loop
|
||||
|
||||
Here's the development cycle the agent runs autonomously:
|
||||
|
||||
```
|
||||
Spec / Issue
|
||||
|
|
||||
v
|
||||
Write code
|
||||
|
|
||||
v
|
||||
Run unit tests (go test ./internal/...)
|
||||
|
|
||||
v
|
||||
Run integration tests (go test ./tests/integration/ -run TestXxx)
|
||||
|
|
||||
v
|
||||
Enter devcontainer (make devc)
|
||||
|
|
||||
v
|
||||
Execute E2E runbook
|
||||
|
|
||||
v
|
||||
All green? -> Open PR
|
||||
```
|
||||
|
||||
My job becomes:
|
||||
|
||||
1. **Write the spec** — what should the feature do?
|
||||
2. **Write the runbook** — how do we verify it works?
|
||||
3. **Review the PR** — does the implementation make sense?
|
||||
|
||||
## Why CLIs Make This Possible
|
||||
|
||||
This workflow works _because_ skillshare is a CLI:
|
||||
|
||||
- **Text in, text out** — AI agents can read command output and verify assertions naturally
|
||||
- **Deterministic behavior** — same input produces same output, no UI flakiness
|
||||
- **Container-friendly** — CLIs run perfectly in Docker, no display server or browser needed
|
||||
- **Composable** — agents can chain commands, pipe output, and script complex scenarios
|
||||
|
||||
GUIs need screenshot comparison, browser automation, and flaky selectors. CLIs need `assert output contains "success"`. The verification surface is fundamentally simpler.
|
||||
|
||||
## What I'd Tell Someone Starting Out
|
||||
|
||||
1. **Invest in test isolation early** — agents need a sandbox they can't break
|
||||
2. **Write runbooks** — explicit verification beats vague expectations
|
||||
3. **Teach agents your workflow via skills/context files** — not just your API
|
||||
4. **Use containers** — cheaper than debugging environment differences
|
||||
5. **Add `--json` flags** — structured output lets agents verify programmatically
|
||||
|
||||
## Honest Takeaway
|
||||
|
||||
AI agents are fast, confident, and wrong in new ways. I don't assume they write perfect code — I assume they can fix their own mistakes, **as long as the feedback loop is fast enough.** Runbooks, sandboxes, and built-in skills are that feedback loop.
|
||||
|
||||
Still learning and iterating on this. If you're building CLI tools with AI agents, I'd love to hear what's working for you.
|
||||
@@ -1,248 +0,0 @@
|
||||
---
|
||||
slug: e2e-tests-now-powered-by-mdproof
|
||||
title: "skillshare E2E Tests Are Now Fully Powered by mdproof"
|
||||
authors: [runkids]
|
||||
tags: [testing, e2e, mdproof, ai-agents]
|
||||
---
|
||||
|
||||
A week ago I wrote about [building testing infrastructure for AI-driven CLI development](/blog/e2e-testing-for-ai-agents). The runbooks I described — Markdown files with step-by-step commands and expected results — turned out to be the most useful piece of that entire infrastructure.
|
||||
|
||||
So useful, in fact, that I extracted the runner into its own tool: [**mdproof**](https://github.com/runkids/mdproof).
|
||||
|
||||
As of today, all 22 of skillshare's E2E runbooks run through mdproof. The built-in runner is gone.
|
||||
|
||||
<!-- truncate -->
|
||||
|
||||
## From Informal Runbooks to a Real Test Runner
|
||||
|
||||
The original runbooks were just documentation. They looked like this:
|
||||
|
||||
```markdown
|
||||
## Steps
|
||||
1. Run: `skillshare install runkids/skillshare`
|
||||
2. Verify: skill directory exists
|
||||
3. Verify: SKILL.md contains expected frontmatter
|
||||
```
|
||||
|
||||
A human (or an AI agent) would read the steps and manually execute them. That worked, but it was slow and error-prone. The agent had to interpret "verify: skill directory exists" and figure out the right `test -d` command on its own.
|
||||
|
||||
What I wanted was something that could **execute the commands and check the assertions automatically** — but still look like a readable Markdown document.
|
||||
|
||||
## What mdproof Does
|
||||
|
||||
mdproof treats Markdown files as executable test runbooks. Write your test as a document, run it as a real test:
|
||||
|
||||
````markdown
|
||||
### Step 2: First init
|
||||
|
||||
```bash
|
||||
ss init --no-copy --targets claude,cursor --mode merge --no-git --no-skill
|
||||
```
|
||||
|
||||
Expected:
|
||||
- exit_code: 0
|
||||
- Initialized successfully
|
||||
- regex: Config:.*\.config/skillshare/config\.yaml
|
||||
````
|
||||
|
||||
That's an actual step from skillshare's [`first_use_cli_e2e_runbook.md`](https://github.com/runkids/skillshare/blob/main/ai_docs/tests/first_use_cli_e2e_runbook.md). mdproof parses it, runs the bash block, and checks each assertion. No test framework. No assertion library. The test IS the document.
|
||||
|
||||
```
|
||||
$ mdproof ai_docs/tests/first_use_cli_e2e_runbook.md
|
||||
|
||||
✓ first_use_cli_e2e_runbook.md
|
||||
──────────────────────────────────────────────────
|
||||
✓ Step 0 Verify command entrypoint 12ms
|
||||
✓ Step 1 Create isolated first-use HOME 3ms
|
||||
✓ Step 2 First init 84ms
|
||||
✓ Step 3 Verify config was created 8ms
|
||||
✓ Step 4 Add one demo skill to source 4ms
|
||||
✓ Step 5 First sync 62ms
|
||||
✓ Step 6 Verify skill reached both targets 5ms
|
||||
✓ Step 7 Per-target compatibility tuning 94ms
|
||||
✓ Step 8 Run doctor 31ms
|
||||
──────────────────────────────────────────────────
|
||||
9/9 passed 303ms
|
||||
```
|
||||
|
||||
## Four Assertion Types
|
||||
|
||||
Through building skillshare's test suite, four assertion patterns emerged naturally. mdproof supports all of them:
|
||||
|
||||
### Substring + Negation
|
||||
|
||||
The simplest form. Check if output contains (or doesn't contain) a string:
|
||||
|
||||
```markdown
|
||||
Expected:
|
||||
- Initialized successfully
|
||||
- config_created=yes
|
||||
- Not Exec format error
|
||||
```
|
||||
|
||||
### Exit Code
|
||||
|
||||
Explicit exit code checking — essential for CLI tools:
|
||||
|
||||
```markdown
|
||||
Expected:
|
||||
- exit_code: 0
|
||||
```
|
||||
|
||||
### Regex
|
||||
|
||||
Pattern matching with multiline mode (`(?m)`) enabled by default:
|
||||
|
||||
```markdown
|
||||
Expected:
|
||||
- regex: v(dev|\d+\.\d+)
|
||||
- regex: Config:.*\.config/skillshare/config\.yaml
|
||||
```
|
||||
|
||||
### jq (JSON assertions)
|
||||
|
||||
This is where it gets powerful. skillshare's `--json` flags produce structured output, and `jq:` assertions let you validate the structure precisely:
|
||||
|
||||
````markdown
|
||||
### Step 2: status --json outputs pure JSON
|
||||
|
||||
```bash
|
||||
OUTPUT=$(ss status --json)
|
||||
echo "$OUTPUT" | jq -e ".skill_count >= 2" && echo "FIELD_CHECK=OK"
|
||||
echo "$OUTPUT" | jq -e ".targets | length >= 1" && echo "TARGETS=OK"
|
||||
FIRST=$(echo "$OUTPUT" | head -c1)
|
||||
[ "$FIRST" = "{" ] && echo "PURE_JSON=OK"
|
||||
```
|
||||
|
||||
Expected:
|
||||
- exit_code: 0
|
||||
- FIELD_CHECK=OK
|
||||
- TARGETS=OK
|
||||
- PURE_JSON=OK
|
||||
````
|
||||
|
||||
This pattern — run `jq -e` inside the bash block, echo a sentinel, assert on the sentinel — emerged from skillshare's [JSON output purity runbook](https://github.com/runkids/skillshare/blob/main/ai_docs/tests/json_output_purity_runbook.md). It's ugly but reliable: if the `jq` expression fails, the sentinel never prints, and the assertion catches it.
|
||||
|
||||
## Lifecycle Hooks
|
||||
|
||||
Real E2E tests need setup and teardown. mdproof supports a `runbook.json` config that runs hooks automatically:
|
||||
|
||||
```json
|
||||
{
|
||||
"build": "cd /workspace && make build && /workspace/.devcontainer/ensure-mdproof.sh",
|
||||
"setup": "ss init -g --no-copy --all-targets --no-git --no-skill --force",
|
||||
"teardown": "ss uninstall --all --force",
|
||||
"timeout": "5m"
|
||||
}
|
||||
```
|
||||
|
||||
- **build**: Runs once before all runbooks (compile the binary)
|
||||
- **setup**: Runs before each runbook (clean slate)
|
||||
- **teardown**: Runs after each runbook (cleanup)
|
||||
- **timeout**: Global safety net
|
||||
|
||||
This means runbook files stay focused on the actual test logic. No boilerplate.
|
||||
|
||||
## JSON Reports for AI Agents
|
||||
|
||||
The whole point of extracting mdproof was to close the agent feedback loop. `--report json` outputs structured results that agents can parse directly:
|
||||
|
||||
```json
|
||||
{
|
||||
"version": "1",
|
||||
"runbook": "first_use_cli_e2e_runbook.md",
|
||||
"duration_ms": 303,
|
||||
"summary": { "total": 9, "passed": 9, "failed": 0, "skipped": 0 },
|
||||
"steps": [
|
||||
{
|
||||
"step": { "number": 0, "title": "Verify command entrypoint", "command": "..." },
|
||||
"status": "passed",
|
||||
"exit_code": 0,
|
||||
"stdout": "skillshare version vdev (...)",
|
||||
"stderr": ""
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Agent writes code → agent runs `mdproof --report json` → agent reads the JSON → agent fixes failures → re-run. No human interprets terminal colors. No regex on test output. The agent gets exactly the data it needs.
|
||||
|
||||
## Why I Extracted It
|
||||
|
||||
The runbook runner started as ~200 lines inside skillshare's codebase. Over three days (March 9–11) it grew features that had nothing to do with skillshare:
|
||||
|
||||
- Persistent bash sessions (variables survive across steps)
|
||||
- Typed assertion engine (exit_code, regex, jq, substring)
|
||||
- Lifecycle hooks (build/setup/teardown)
|
||||
- Selective step execution (`--steps 3,5` or `--from 4`)
|
||||
- Retry and depends directives
|
||||
- Container-only safety checks
|
||||
- JSON and JUnit XML reporting
|
||||
|
||||
At that point it was clear: **this isn't a skillshare feature — it's a testing tool.** So I pulled it out into [runkids/mdproof](https://github.com/runkids/mdproof) and replaced the built-in runner with a single dependency.
|
||||
|
||||
The migration commit was one line of real logic: swap the executor.
|
||||
|
||||
## What skillshare's Test Suite Looks Like Now
|
||||
|
||||
22 runbooks covering the full CLI surface:
|
||||
|
||||
| Category | Runbooks | Example |
|
||||
|----------|----------|---------|
|
||||
| First-time user flow | 1 | `first_use_cli_e2e_runbook.md` |
|
||||
| JSON output purity | 2 | `json_output_purity_runbook.md`, `extras_refactor_json_runbook.md` |
|
||||
| Security audit | 1 | `audit_output_antigravity_runbook.md` |
|
||||
| Install/update | 2 | `issue46_install_update_optimization_runbook.md` |
|
||||
| Sync behavior | 3 | `symlinked_dir_sync_runbook.md`, `sync_extras_runbook.md` |
|
||||
| URL parsing | 3 | `gitlab_subgroup_url_parse_runbook.md`, `azure_devops_url_parse_runbook.md` |
|
||||
| Config features | 2 | `gitlab_hosts_config_runbook.md`, `registry_yaml_split_runbook.md` |
|
||||
| Cleanup operations | 3 | `uninstall_sync_orphan_runbook.md`, `uninstall_all_glob_runbook.md` |
|
||||
| Precision checks | 2 | `check_treehash_precision_runbook.md`, `update_prune_check_stale_runbook.md` |
|
||||
| Other | 4 | target paths, gitignore perf, precommit hooks, discovery |
|
||||
|
||||
All executed via:
|
||||
|
||||
```bash
|
||||
mdproof --report json ai_docs/tests/
|
||||
```
|
||||
|
||||
One command. 22 runbooks. Structured results.
|
||||
|
||||
## The Loop
|
||||
|
||||
In the [previous post](/blog/e2e-testing-for-ai-agents) I described a development loop where the agent writes code, runs tests, and verifies its own work. mdproof makes the E2E layer of that loop concrete:
|
||||
|
||||
```
|
||||
Agent writes code
|
||||
↓
|
||||
go test ./... (unit + integration)
|
||||
↓
|
||||
mdproof --report json ai_docs/tests/relevant_runbook.md
|
||||
↓
|
||||
Agent reads JSON → all passed? → PR
|
||||
→ failed? → fix and re-run
|
||||
```
|
||||
|
||||
The runbooks haven't changed much since that post. What changed is that they're no longer documentation that an agent interprets — they're **executable specifications** that produce machine-readable results.
|
||||
|
||||
## Try It
|
||||
|
||||
If you're building a CLI tool and want executable Markdown tests:
|
||||
|
||||
```bash
|
||||
curl -fsSL https://raw.githubusercontent.com/runkids/mdproof/main/install.sh | sh
|
||||
```
|
||||
|
||||
Write a `test.md`, add some steps with `Expected:` blocks, and run `mdproof test.md`. That's it.
|
||||
|
||||
If you use skillshare, mdproof is already available as a skill:
|
||||
|
||||
```bash
|
||||
skillshare install runkids/mdproof
|
||||
```
|
||||
|
||||
Your AI agent gets the full syntax reference and can start writing runbooks immediately.
|
||||
|
||||
---
|
||||
|
||||
[mdproof on GitHub](https://github.com/runkids/mdproof) · [skillshare's runbook suite](https://github.com/runkids/skillshare/tree/main/ai_docs/tests)
|
||||
@@ -281,7 +281,7 @@ This opens a dashboard at `localhost:19420` where you can:
|
||||
- Run sync, audit, and check operations visually
|
||||
- Edit target include/exclude filters without touching YAML
|
||||
|
||||

|
||||

|
||||
|
||||
Everything you can do in the CLI, you can do in the UI — including setting up the multi-account extras config described above.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user