mirror of
https://github.com/runkids/skillshare.git
synced 2026-11-20 20:34:19 +00:00
docs(runbook): add runbook runner usage guide
Covers CLI flags, safety mechanism, runbook markdown format, all 5 assertion types (substring, negated, exit_code, regex, jq), JSON output schema, session executor behavior, and common patterns.
This commit is contained in:
@@ -0,0 +1,446 @@
|
||||
# Runbook Runner Guide
|
||||
|
||||
`bin/runbook` is a CLI tool that parses, executes, and validates Markdown-based E2E test runbooks.
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
# Build
|
||||
go build -o bin/runbook ./tools/runbook
|
||||
|
||||
# Dry-run (parse only, no execution)
|
||||
bin/runbook --dry-run --report json ai_docs/tests/atomgit_install_runbook.md
|
||||
|
||||
# Execute inside a container (required for execution)
|
||||
bin/runbook --report json --no-tui ai_docs/tests/atomgit_install_runbook.md
|
||||
|
||||
# Batch execute an entire directory
|
||||
bin/runbook --report json --no-tui ai_docs/tests/
|
||||
```
|
||||
|
||||
## CLI Flags
|
||||
|
||||
| Flag | Description |
|
||||
|------|-------------|
|
||||
| `--dry-run` | Parse the runbook only; do not execute any commands |
|
||||
| `--report json` | Output a JSON report |
|
||||
| `--no-tui` | Disable interactive TUI; use plain text output |
|
||||
| `--timeout 5m` | Per-step timeout (default: 2 minutes) |
|
||||
|
||||
Input can be a single `.md` file or a directory (auto-discovers `*_runbook.md` / `*-runbook.md`).
|
||||
|
||||
## Safety Mechanism
|
||||
|
||||
The runbook runner only executes inside containers (detected via `/.dockerenv`). Running outside a container is refused:
|
||||
|
||||
```
|
||||
runbook: refusing to execute outside a container
|
||||
Use --dry-run to parse without executing, or set
|
||||
RUNBOOK_ALLOW_EXECUTE=1 to override this safety check.
|
||||
```
|
||||
|
||||
- `--dry-run` works in any environment
|
||||
- Set `RUNBOOK_ALLOW_EXECUTE=1` to override (for testing only)
|
||||
|
||||
---
|
||||
|
||||
## Writing Runbooks
|
||||
|
||||
### Basic Structure
|
||||
|
||||
```markdown
|
||||
# Runbook Title
|
||||
|
||||
## Scope
|
||||
|
||||
Describe the test scope.
|
||||
|
||||
## Environment
|
||||
|
||||
- Requirements (e.g., devcontainer, network access)
|
||||
|
||||
## Steps
|
||||
|
||||
### Step 1: Step title
|
||||
|
||||
Optional description.
|
||||
|
||||
\`\`\`bash
|
||||
ss install https://github.com/user/repo --all
|
||||
\`\`\`
|
||||
|
||||
**Expected:**
|
||||
- exit_code: 0
|
||||
- Installed
|
||||
- my-skill
|
||||
|
||||
### Step 2: Verify result
|
||||
|
||||
\`\`\`bash
|
||||
ss list --no-tui
|
||||
\`\`\`
|
||||
|
||||
**Expected:**
|
||||
- exit_code: 0
|
||||
- my-skill
|
||||
|
||||
## Pass Criteria
|
||||
|
||||
All steps pass.
|
||||
```
|
||||
|
||||
### Step Heading Formats
|
||||
|
||||
The parser supports multiple heading formats:
|
||||
|
||||
```markdown
|
||||
### Step 1: Title <- standard
|
||||
## Step 0: Setup <- ## level works too
|
||||
### 1. Title <- number with period
|
||||
### 1b. Sub-step <- with suffix letter
|
||||
```
|
||||
|
||||
### Code Blocks
|
||||
|
||||
- Supports ` ```bash `, ` ```sh `, or unmarked (defaults to bash)
|
||||
- Multiple code blocks within a single step are merged with `---` separator
|
||||
- Non-bash/sh language tags (e.g., `yaml`, `json`) are classified as `manual` and skipped during execution
|
||||
|
||||
### Expected Blocks
|
||||
|
||||
Write `**Expected:**` below the code block, followed by `- ` prefixed assertion lines.
|
||||
|
||||
---
|
||||
|
||||
## Assertion Syntax
|
||||
|
||||
### Five Assertion Types
|
||||
|
||||
#### 1. Substring (default)
|
||||
|
||||
```markdown
|
||||
- Installed
|
||||
- my-skill
|
||||
```
|
||||
|
||||
Case-insensitive substring match against stdout + stderr combined.
|
||||
|
||||
#### 2. Negated Substring
|
||||
|
||||
```markdown
|
||||
- Not error
|
||||
- NOT warning
|
||||
- not found
|
||||
```
|
||||
|
||||
Supported negation prefixes (matched longest-first):
|
||||
|
||||
| Prefix | Example |
|
||||
|--------|---------|
|
||||
| `Should NOT ` | `Should NOT contain errors` |
|
||||
| `should not ` | `should not fail` |
|
||||
| `Must NOT ` | `Must NOT crash` |
|
||||
| `must not ` | `must not hang` |
|
||||
| `Does not ` | `Does not contain secrets` |
|
||||
| `does not ` | `does not leak` |
|
||||
| `NOT ` | `NOT Enumerating objects` |
|
||||
| `Not ` | `Not error` |
|
||||
| `not ` | `not failed` |
|
||||
| `No ` | `No warnings` |
|
||||
| `no ` | `no errors` |
|
||||
|
||||
#### 3. Exit Code
|
||||
|
||||
```markdown
|
||||
- exit_code: 0 <- exact match
|
||||
- exit_code: 1 <- expect failure
|
||||
- exit_code: !0 <- any non-zero value
|
||||
- exit_code: !1 <- any value except 1
|
||||
```
|
||||
|
||||
Checks only the process exit code, not stdout/stderr.
|
||||
|
||||
#### 4. Regex
|
||||
|
||||
```markdown
|
||||
- regex: \d+ skills
|
||||
- regex: v\d+\.\d+\.\d+
|
||||
- regex: (installed|updated)\s+successfully
|
||||
```
|
||||
|
||||
Uses Go `regexp` syntax. Matches against stdout + stderr combined. Case-sensitive.
|
||||
|
||||
#### 5. jq Expression
|
||||
|
||||
```markdown
|
||||
- jq: .count > 0
|
||||
- jq: .skills | length >= 2
|
||||
- jq: .status == "ok"
|
||||
- jq: .installed == true
|
||||
```
|
||||
|
||||
Runs `jq -e <expr>` against **stdout only** (stderr excluded). Requires `jq` to be installed.
|
||||
|
||||
### Match Targets
|
||||
|
||||
| Type | Matches Against | Case |
|
||||
|------|----------------|------|
|
||||
| substring | stdout + stderr | insensitive |
|
||||
| negated | stdout + stderr | insensitive |
|
||||
| exit_code | exit code value | N/A |
|
||||
| regex | stdout + stderr | sensitive |
|
||||
| jq | stdout only | N/A |
|
||||
|
||||
### Assertion vs Exit Code Behavior
|
||||
|
||||
- **With assertions**: Assertions determine pass/fail, **regardless of exit code**
|
||||
- Example: `exit_code: 1` + `error` -> passes even with exit code 1
|
||||
- **Without assertions**: exit code 0 -> pass, non-zero -> fail
|
||||
|
||||
---
|
||||
|
||||
## JSON Output Format
|
||||
|
||||
```json
|
||||
{
|
||||
"version": "1",
|
||||
"runbook": "atomgit_install_runbook.md",
|
||||
"duration_ms": 12345,
|
||||
"summary": {
|
||||
"total": 7,
|
||||
"passed": 5,
|
||||
"failed": 1,
|
||||
"skipped": 1
|
||||
},
|
||||
"steps": [
|
||||
{
|
||||
"step": {
|
||||
"number": 1,
|
||||
"title": "Install from AtomGit",
|
||||
"command": "ss install https://atomgit.com/...",
|
||||
"lang": "bash",
|
||||
"expected": ["exit_code: 0", "Installed"],
|
||||
"executor": "auto"
|
||||
},
|
||||
"status": "passed",
|
||||
"duration_ms": 3200,
|
||||
"stdout": "...",
|
||||
"stderr": "",
|
||||
"exit_code": 0,
|
||||
"assertions": [
|
||||
{
|
||||
"pattern": "exit_code: 0",
|
||||
"type": "exit_code",
|
||||
"matched": true
|
||||
},
|
||||
{
|
||||
"pattern": "Installed",
|
||||
"type": "substring",
|
||||
"matched": true
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### AssertionResult Fields
|
||||
|
||||
| Field | Description |
|
||||
|-------|-------------|
|
||||
| `pattern` | Original assertion text |
|
||||
| `type` | `substring`, `exit_code`, `regex`, or `jq` |
|
||||
| `matched` | Whether the assertion passed |
|
||||
| `negated` | Whether it's a negated assertion (`Not ...` or `exit_code: !N`) |
|
||||
| `detail` | Extra info on failure (e.g., `got exit_code=1, expected 0`) |
|
||||
|
||||
---
|
||||
|
||||
## Session Executor (Cross-Step Variable Persistence)
|
||||
|
||||
In non-TUI mode, all `auto` steps run sequentially within a single bash process. Each step executes in a subshell `()` for isolation, but environment variables are persisted via an env file:
|
||||
|
||||
```
|
||||
Step 1: export FOO=bar -> saved to /tmp/env
|
||||
Step 2: echo $FOO -> sources /tmp/env first -> prints "bar"
|
||||
```
|
||||
|
||||
Steps in a runbook can naturally reference variables set by earlier steps without any extra handling.
|
||||
|
||||
### Step Failure Does Not Abort
|
||||
|
||||
A failed step (non-zero exit code) does not stop subsequent steps. All steps run to completion, and the final report aggregates all results.
|
||||
|
||||
### Manual Steps
|
||||
|
||||
Steps without a bash code block, or with non-bash language tags, are classified as `manual`, automatically skipped, and marked as `skipped`.
|
||||
|
||||
---
|
||||
|
||||
## Full Example Runbook
|
||||
|
||||
```markdown
|
||||
# My Feature E2E Test
|
||||
|
||||
## Scope
|
||||
|
||||
Verify the full install -> list -> sync -> uninstall lifecycle.
|
||||
|
||||
## Environment
|
||||
|
||||
- Devcontainer with ssenv isolation
|
||||
- Network access required
|
||||
|
||||
## Steps
|
||||
|
||||
### Step 1: Install skill
|
||||
|
||||
\`\`\`bash
|
||||
ss install https://github.com/user/repo --all
|
||||
\`\`\`
|
||||
|
||||
**Expected:**
|
||||
- exit_code: 0
|
||||
- Installed
|
||||
|
||||
### Step 2: Verify in list
|
||||
|
||||
\`\`\`bash
|
||||
ss list --json | jq -e '.[] | select(.name == "my-skill")'
|
||||
\`\`\`
|
||||
|
||||
**Expected:**
|
||||
- exit_code: 0
|
||||
- jq: .name == "my-skill"
|
||||
|
||||
### Step 3: Check sync
|
||||
|
||||
\`\`\`bash
|
||||
ss sync
|
||||
\`\`\`
|
||||
|
||||
**Expected:**
|
||||
- exit_code: 0
|
||||
- regex: \d+ linked
|
||||
|
||||
### Step 4: Uninstall
|
||||
|
||||
\`\`\`bash
|
||||
ss uninstall my-skill --force
|
||||
\`\`\`
|
||||
|
||||
**Expected:**
|
||||
- exit_code: 0
|
||||
- Moved to trash
|
||||
|
||||
### Step 5: Verify cleanup
|
||||
|
||||
\`\`\`bash
|
||||
ss list --no-tui
|
||||
\`\`\`
|
||||
|
||||
**Expected:**
|
||||
- exit_code: 0
|
||||
- Not my-skill
|
||||
|
||||
## Pass Criteria
|
||||
|
||||
All 5 steps pass.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Common Patterns
|
||||
|
||||
### Verify File Exists
|
||||
|
||||
```bash
|
||||
test -f ~/.config/skillshare/registry.yaml && echo "EXISTS" || echo "MISSING"
|
||||
```
|
||||
|
||||
```markdown
|
||||
**Expected:**
|
||||
- exit_code: 0
|
||||
- EXISTS
|
||||
- Not MISSING
|
||||
```
|
||||
|
||||
### JSON Field Validation
|
||||
|
||||
```bash
|
||||
ss status --json
|
||||
```
|
||||
|
||||
```markdown
|
||||
**Expected:**
|
||||
- exit_code: 0
|
||||
- jq: .skill_count >= 1
|
||||
- jq: .targets | length > 0
|
||||
```
|
||||
|
||||
### Expect a Failing Step
|
||||
|
||||
```bash
|
||||
ss install nonexistent-repo
|
||||
```
|
||||
|
||||
```markdown
|
||||
**Expected:**
|
||||
- exit_code: !0
|
||||
- Not Installed
|
||||
- error
|
||||
```
|
||||
|
||||
### Multi-Line Script
|
||||
|
||||
```bash
|
||||
SKILL_DIR=~/.config/skillshare/skills/my-skill
|
||||
test -d "$SKILL_DIR" && echo "DIR_OK" || echo "DIR_MISSING"
|
||||
cat "$SKILL_DIR/.skillshare-meta.json" | jq -e '.tree_hash'
|
||||
echo "META_OK"
|
||||
```
|
||||
|
||||
```markdown
|
||||
**Expected:**
|
||||
- exit_code: 0
|
||||
- DIR_OK
|
||||
- META_OK
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Usage in CI / E2E Skill
|
||||
|
||||
### Build and Execute Inside Devcontainer
|
||||
|
||||
```bash
|
||||
# Build Linux binary
|
||||
docker exec $CONTAINER bash -c 'cd /workspace && go build -o bin/runbook ./tools/runbook'
|
||||
|
||||
# Execute in ssenv-isolated environment
|
||||
docker exec $CONTAINER ssenv enter "$ENV_NAME" -- \
|
||||
/workspace/bin/runbook --report json --no-tui \
|
||||
/workspace/ai_docs/tests/my_runbook.md
|
||||
```
|
||||
|
||||
### Dry-Run Preview
|
||||
|
||||
```bash
|
||||
bin/runbook --dry-run --report json ai_docs/tests/ | jq -s '
|
||||
[.[] | select(.runbook) | {
|
||||
runbook: .runbook,
|
||||
steps: .summary.total
|
||||
}]'
|
||||
```
|
||||
|
||||
### Analyze Failed Steps
|
||||
|
||||
```bash
|
||||
bin/runbook --report json --no-tui my_runbook.md | jq '
|
||||
.steps[] | select(.status == "failed") | {
|
||||
step: .step.number,
|
||||
title: .step.title,
|
||||
exit_code: .exit_code,
|
||||
failed_assertions: [.assertions[] | select(.matched == false)]
|
||||
}'
|
||||
```
|
||||
Reference in New Issue
Block a user