Files
skillshare/ai_docs/tests/mcp_check_runbook.md
T
Willie cb4b55a91b test(mcp): cover --live in the mcp check runbook
Unit tests use a fake server built from the test binary, so nothing
exercised --live against separate real processes, a real HTTP server
or the process tree a server leaves behind. Steps 8-13 use sh stdio
servers (server/discover, initialize only, one that ignores SIGTERM and
starts a child, one that crashes), a Python server that answers 401,
and an unreachable URL, and check that nothing starts without --live.

Refs #141
2026-09-30 14:42:50 +08:00

16 KiB
Raw Blame History

CLI E2E Runbook: skillshare mcp check

Scope

Verify the read-only MCP check against a global config whose servers cover every finding: an unset fromEnv variable, a command missing from PATH, a .invalid host, unsynced targets, a client rule refusal, targets: [], and one mcp.projects root. Covers human output, the --json shape including project, --no-dns, name filtering, exit codes 0 and 1, and GET /api/mcp/check with and without ?dns=0. Steps 1–7 launch no server, run no command, and sync nothing.

Steps 8–13 cover --live against a second config: tiny local stdio servers written as sh scripts (one answers server/discover, one only the initialize handshake, one never answers, one crashes), a local HTTP server that answers 401 with resource metadata, and an unreachable HTTP URL. They check that no server starts without --live, the live JSON object, the human output, the 401 warning, the timeout and that it leaves no process behind, and that values are redacted.

Environment

Run inside the devcontainer with a fresh ssenv HOME:

ssenv create mcp-check-e2e --init
cd /tmp
ssenv enter mcp-check-e2e -- mdproof --report json /workspace/ai_docs/tests/mcp_check_runbook.md

Human-output steps set NO_COLOR=1 so the status marks can be matched as plain text. Every step works from $HOME and points SKILLSHARE_CONFIG at $HOME/mcp-check/config.yaml. Never run these steps with the repository root as the working directory: the CLI would pick up the repository's .skillshare/ as project mode. The check needs DNS only for Step 4; .invalid never resolves, online or not. Step 7 starts skillshare ui on port 47931 and Step 11 a Python HTTP server on port 47932; pick other free ports if they are taken. Port 1 on 127.0.0.1 must refuse connections.

Steps

Step 1: Write the global config

The skills directory must exist before skillshare ui starts. mcp list would refuse this config because of the workspace server, so the step reads it back through a check of the one server that has no problems.

set -eu
cd "$HOME"
CASE="$HOME/mcp-check"
rm -rf "$CASE"
mkdir -p "$CASE/skills" "$CASE/app"
export SKILLSHARE_CONFIG="$CASE/config.yaml"
cat > "$SKILLSHARE_CONFIG" <<YAML
source: $CASE/skills
targets: {}
mcp:
  targets: [claude, cursor]
  servers:
    docs:
      url: https://example.com/docs
    envless:
      command: sh
      env:
        API_KEY: {fromEnv: SS_E2E_CHECK_TOKEN}
    ghost:
      command: no-such-mcp-binary
    offline:
      url: https://mcp.skillshare.invalid/mcp
      targets: [cursor]
    parked:
      url: https://example.com/parked
      targets: []
    workspace:
      url: https://example.com/workspace
  projects:
    ~/mcp-check/app:
      targets: [claude]
      servers:
        docs:
          command: no-such-project-binary
YAML
ss mcp check parked --json -g

Expected:

  • exit_code: 0
  • jq: [.servers[].name] == ["parked"]
  • jq: .summary == {"errors": 0, "warnings": 0}

Step 2: Human output lists every server and exits 1

cd "$HOME"
export SKILLSHARE_CONFIG="$HOME/mcp-check/config.yaml"
unset SS_E2E_CHECK_TOKEN
NO_COLOR=1 ss mcp check --no-dns -g

Expected:

  • exit_code: 1
  • ✗ envless
  • ✗ env API_KEY reads SS_E2E_CHECK_TOKEN, which is not set
  • ✗ command no-such-mcp-binary was not found on PATH
  • ✗ claude: Claude MCP workspace: Claude Code reserves this name
  • ! cursor: not synced yet; run skillshare sync mcp
  • · kept in Skillshare only; no Agent receives it
  • ✗ docs (project ~/mcp-check/app)
  • ✗ command no-such-project-binary was not found on PATH
  • 7 server(s) checked: 4 error(s), 9 warning(s)

Step 3: JSON shape and the project field

cd "$HOME"
export SKILLSHARE_CONFIG="$HOME/mcp-check/config.yaml"
unset SS_E2E_CHECK_TOKEN
ss mcp check --json --no-dns -g

Expected:

  • exit_code: 1
  • jq: [.servers[].name] == ["docs", "envless", "ghost", "offline", "parked", "workspace", "docs"]
  • jq: [.servers[] | select(has("project"))] | length == 1
  • jq: .servers[6].project | startswith("/") and endswith("/mcp-check/app")
  • jq: .servers[6].ok == false and .servers[6].findings[0].check == "command" and .servers[6].findings[0].subject == "no-such-project-binary"
  • jq: .servers[6].findings[1] == {"level": "warning", "check": "sync", "target": "claude", "message": "not synced yet; run skillshare sync mcp"}
  • jq: .servers[1].findings[0] | .check == "env" and .subject == "SS_E2E_CHECK_TOKEN" and .target == ""
  • jq: .servers[5].findings[0] | .check == "client-rule" and .target == "claude"
  • jq: .servers[4] | .ok == true and .findings == [{"level": "info", "check": "targets", "target": "", "message": "kept in Skillshare only; no Agent receives it"}]
  • jq: [.servers[].findings[] | select(.check == "dns")] == []
  • jq: .summary == {"errors": 4, "warnings": 9}

Step 4: DNS warning, and --no-dns skips it

A DNS warning never fails the check, so both runs exit 0.

set -eu
cd "$HOME"
export SKILLSHARE_CONFIG="$HOME/mcp-check/config.yaml"
ss mcp check offline --json -g > "$HOME/mcp-check/dns.json"
ss mcp check offline --json --no-dns -g > "$HOME/mcp-check/no-dns.json"
jq -n --slurpfile dns "$HOME/mcp-check/dns.json" --slurpfile skip "$HOME/mcp-check/no-dns.json" '{dns: $dns[0], skip: $skip[0]}'

Expected:

  • exit_code: 0
  • jq: .dns.servers[0].findings[0] == {"level": "warning", "check": "dns", "target": "", "message": "host mcp.skillshare.invalid did not resolve", "subject": "mcp.skillshare.invalid"}
  • jq: .dns.servers[0].ok == true and .dns.summary == {"errors": 0, "warnings": 2}
  • jq: [.skip.servers[0].findings[].check] == ["sync"]

Step 5: Name filtering, unknown names, and exit 0

set -eu
cd "$HOME"
export SKILLSHARE_CONFIG="$HOME/mcp-check/config.yaml"
ss mcp check docs --json --no-dns -g > "$HOME/mcp-check/docs.json" || test $? -eq 1
if ss mcp check nope -g > "$HOME/mcp-check/unknown.txt" 2>&1; then echo "unknown name must fail"; exit 1; fi
ss mcp check parked -g > "$HOME/mcp-check/parked.txt"
jq -c '[.servers[] | {name, project}]' "$HOME/mcp-check/docs.json"
cat "$HOME/mcp-check/unknown.txt" "$HOME/mcp-check/parked.txt"

Expected:

  • exit_code: 0
  • regex: [{"name":"docs","project":null},{"name":"docs","project":"[^"]+/mcp-check/app"}]
  • unknown MCP server "nope"; known servers: docs, envless, ghost, offline, parked, workspace
  • 1 server(s) checked: 0 error(s), 0 warning(s)

Step 6: A set variable passes and its value is never printed

set -eu
cd "$HOME"
export SKILLSHARE_CONFIG="$HOME/mcp-check/config.yaml"
SS_E2E_CHECK_TOKEN=s3cr3t-e2e-value NO_COLOR=1 ss mcp check envless --no-dns -g > "$HOME/mcp-check/envless.txt"
SS_E2E_CHECK_TOKEN=s3cr3t-e2e-value ss mcp check envless --json --no-dns -g >> "$HOME/mcp-check/envless.txt"
if grep -q s3cr3t-e2e-value "$HOME/mcp-check/envless.txt"; then echo "value leaked"; exit 1; fi
head -n 5 "$HOME/mcp-check/envless.txt"

Expected:

  • exit_code: 0
  • ! envless
  • 1 server(s) checked: 0 error(s), 2 warning(s)

Step 7: The dashboard endpoint, with and without ?dns=0

set -eu
cd "$HOME"
export SKILLSHARE_CONFIG="$HOME/mcp-check/config.yaml"
unset SS_E2E_CHECK_TOKEN
PORT=47931
ss ui --port "$PORT" --no-open -g > "$HOME/mcp-check/ui.log" 2>&1 &
UI_PID=$!
trap 'kill "$UI_PID" 2>/dev/null || true' EXIT
for _ in $(seq 1 100); do
  curl -fsS "http://127.0.0.1:$PORT/api/mcp/check?dns=0" > "$HOME/mcp-check/api-no-dns.json" 2>/dev/null && break
  sleep 0.2
done
curl -fsS "http://127.0.0.1:$PORT/api/mcp/check" > "$HOME/mcp-check/api-dns.json"
jq -n --slurpfile skip "$HOME/mcp-check/api-no-dns.json" --slurpfile dns "$HOME/mcp-check/api-dns.json" '{skip: $skip[0], dns: $dns[0]}'

Expected:

  • exit_code: 0
  • jq: .skip.summary == {"errors": 4, "warnings": 9}
  • jq: [.skip.servers[] | select(.project)] | length == 1
  • jq: [.skip.servers[].findings[] | select(.check == "dns")] == []
  • jq: .dns.summary == {"errors": 4, "warnings": 10}
  • jq: [.dns.servers[].findings[] | select(.check == "dns") | .subject] == ["mcp.skillshare.invalid"]

Step 8: Write the live config; without --live nothing starts

Every script writes a marker file when it starts, so the step can prove that a check without --live starts none of them.

set -eu
cd "$HOME"
CASE="$HOME/mcp-check-live"
rm -rf "$CASE"
mkdir -p "$CASE/skills"
export SKILLSHARE_CONFIG="$CASE/config.yaml"
cat > "$CASE/tiny.sh" <<'SH'
#!/bin/sh
touch "$MARKER"
while IFS= read -r line; do
  id=$(printf '%s\n' "$line" | sed -n 's/.*"id":\([0-9][0-9]*\).*/\1/p')
  case "$line" in
    *'"server/discover"'*) printf '{"jsonrpc":"2.0","id":%s,"result":{"resultType":"complete","supportedVersions":["2026-07-28"],"capabilities":{"tools":{}},"_meta":{"io.modelcontextprotocol/serverInfo":{"name":"tiny","version":"0.1.0"}}}}\n' "$id" ;;
    *'"tools/list"'*) printf '{"jsonrpc":"2.0","id":%s,"result":{"resultType":"complete","tools":[{"name":"echo","inputSchema":{"type":"object"}}]}}\n' "$id" ;;
  esac
done
SH
cat > "$CASE/legacy.sh" <<'SH'
#!/bin/sh
touch "$MARKER"
while IFS= read -r line; do
  id=$(printf '%s\n' "$line" | sed -n 's/.*"id":\([0-9][0-9]*\).*/\1/p')
  case "$line" in
    *'"notifications/initialized"'*) ;;
    *'"method":"initialize"'*) printf '{"jsonrpc":"2.0","id":%s,"result":{"protocolVersion":"2025-11-25","capabilities":{"tools":{}},"serverInfo":{"name":"legacy","version":"0.9"}}}\n' "$id" ;;
    *'"tools/list"'*) printf '{"jsonrpc":"2.0","id":%s,"result":{"tools":[{"name":"a"},{"name":"b"}]}}\n' "$id" ;;
    *) printf '{"jsonrpc":"2.0","id":%s,"error":{"code":-32601,"message":"Method not found"}}\n' "$id" ;;
  esac
done
SH
cat > "$CASE/hang.sh" <<'SH'
#!/bin/sh
touch "$MARKER"
trap '' TERM
sleep 3001 &
while :; do sleep 1; done
SH
cat > "$CASE/crash.sh" <<'SH'
#!/bin/sh
touch "$MARKER"
echo "fatal: token $API_TOKEN rejected" >&2
exit 3
SH
cat > "$SKILLSHARE_CONFIG" <<YAML
source: $CASE/skills
targets: {}
mcp:
  targets: [claude]
  servers:
    crash:
      command: sh
      args: [$CASE/crash.sh]
      env:
        MARKER: $CASE/started-crash
        API_TOKEN: {fromEnv: SS_E2E_LIVE_TOKEN}
    down:
      url: http://127.0.0.1:1/mcp
    hang:
      command: sh
      args: [$CASE/hang.sh]
      env:
        MARKER: $CASE/started-hang
    legacy:
      command: sh
      args: [$CASE/legacy.sh]
      env:
        MARKER: $CASE/started-legacy
    signin:
      url: http://127.0.0.1:47932/mcp
      bearerToken: {fromEnv: SS_E2E_LIVE_TOKEN}
    tiny:
      command: sh
      args: [$CASE/tiny.sh]
      env:
        MARKER: $CASE/started-tiny
YAML
SS_E2E_LIVE_TOKEN=s3cr3t-live-e2e ss mcp check --json --no-dns -g > "$CASE/static.json"
ls "$CASE" | grep -c '^started-' || true
jq -c '{live: [.servers[] | select(has("live"))], checks: [.servers[].findings[].check] | unique}' "$CASE/static.json"

Expected:

  • exit_code: 0
  • regex: ^0$
  • jq: .live == [] and .checks == ["sync"]

Step 9: --live reports serverInfo, protocol and tools

set -eu
cd "$HOME"
export SKILLSHARE_CONFIG="$HOME/mcp-check-live/config.yaml"
ss mcp check legacy tiny --live --json -g

Expected:

  • exit_code: 0
  • jq: [.servers[].name] == ["legacy", "tiny"]
  • jq: .servers[1].live == {"protocolVersion": "2026-07-28", "serverInfo": {"name": "tiny", "version": "0.1.0"}, "tools": 1}
  • jq: .servers[0].live == {"protocolVersion": "2025-11-25", "serverInfo": {"name": "legacy", "version": "0.9"}, "tools": 2}
  • jq: [.servers[].findings[] | select(.check == "live") | .level] == ["info", "info"]
  • jq: .summary.errors == 0

Step 10: Human output of a live probe

cd "$HOME"
export SKILLSHARE_CONFIG="$HOME/mcp-check-live/config.yaml"
NO_COLOR=1 ss mcp check tiny --live -g

Expected:

  • exit_code: 0
  • · responds: tiny 0.1.0, protocol 2026-07-28, 1 tool(s)
  • 1 server(s) checked: 0 error(s), 1 warning(s)

Step 11: 401 is a sign-in warning; an unreachable URL is an error

set -eu
cd "$HOME"
CASE="$HOME/mcp-check-live"
export SKILLSHARE_CONFIG="$CASE/config.yaml"
cat > "$CASE/signin.py" <<'PY'
import http.server
class Handler(http.server.BaseHTTPRequestHandler):
    def do_POST(self):
        self.send_response(401)
        self.send_header("WWW-Authenticate", 'Bearer resource_metadata="http://127.0.0.1:47932/.well-known/oauth-protected-resource", scope="read"')
        self.send_header("Content-Length", "0")
        self.end_headers()
    def log_message(self, *args):
        pass
http.server.HTTPServer(("127.0.0.1", 47932), Handler).serve_forever()
PY
python3 "$CASE/signin.py" &
PY_PID=$!
trap 'kill "$PY_PID" 2>/dev/null || true' EXIT
for _ in $(seq 1 50); do
  curl -s -o /dev/null -X POST http://127.0.0.1:47932/mcp && break
  sleep 0.1
done
SS_E2E_LIVE_TOKEN=s3cr3t-live-e2e ss mcp check down signin --live --json -g > "$CASE/http.json" || test $? -eq 1
if grep -q s3cr3t-live-e2e "$CASE/http.json"; then echo "value leaked"; exit 1; fi
cat "$CASE/http.json"

Expected:

  • exit_code: 0
  • jq: [.servers[].name] == ["down", "signin"]
  • jq: .servers[1].ok == true and (.servers[1] | has("live") | not)
  • jq: [.servers[1].findings[] | select(.check == "live")][0] == {"level": "warning", "check": "live", "target": "", "message": "sign-in required (HTTP 401): resource metadata at http://127.0.0.1:47932/.well-known/oauth-protected-resource; skillshare does not sign in", "subject": "http://127.0.0.1:47932/.well-known/oauth-protected-resource"}
  • jq: .servers[0].ok == false and ([.servers[0].findings[] | select(.check == "live")][0].message | startswith("live probe failed:") and contains("connection refused"))
  • jq: .summary.errors == 1

Step 12: The timeout stops the server and everything it started

hang.sh ignores SIGTERM and starts sleep 3001, so only the process-group SIGKILL stops both.

set -eu
cd "$HOME"
CASE="$HOME/mcp-check-live"
export SKILLSHARE_CONFIG="$CASE/config.yaml"
START=$(date +%s)
ss mcp check hang --live --timeout 2s --json -g > "$CASE/hang.json" || test $? -eq 1
ELAPSED=$(( $(date +%s) - START ))
test "$ELAPSED" -lt 8
sleep 0.5
# The brackets keep pgrep from matching this script's own command line.
LEFT=$(pgrep -fc 'sleep 300[1]|hang[.]sh' || true)
jq -c --arg left "$LEFT" '{left: $left, finding: [.servers[0].findings[] | select(.check == "live")][0]}' "$CASE/hang.json"

Expected:

  • exit_code: 0
  • jq: .left == "0"
  • jq: .finding == {"level": "error", "check": "live", "target": "", "message": "live probe failed: no answer within 2s"}

Step 13: A crash shows redacted stderr; a static error skips the probe

set -eu
cd "$HOME"
CASE="$HOME/mcp-check-live"
export SKILLSHARE_CONFIG="$CASE/config.yaml"
rm -f "$CASE/started-crash"
SS_E2E_LIVE_TOKEN=s3cr3t-live-e2e ss mcp check crash --live --json -g > "$CASE/crash.json" || test $? -eq 1
env -u SS_E2E_LIVE_TOKEN ss mcp check crash --live --json -g > "$CASE/skipped.json" || test $? -eq 1
if grep -q s3cr3t-live-e2e "$CASE/crash.json"; then echo "value leaked"; exit 1; fi
jq -n --slurpfile crash "$CASE/crash.json" --slurpfile skip "$CASE/skipped.json" '{crash: [$crash[0].servers[0].findings[] | select(.check == "live")][0], skip: [$skip[0].servers[0].findings[] | select(.check == "live")][0]}'

Expected:

  • exit_code: 0
  • jq: .crash.level == "error" and (.crash.message | contains("exit status 3")) and (.crash.message | endswith("stderr: fatal: token [redacted] rejected"))
  • jq: .skip == {"level": "info", "check": "live", "target": "", "message": "not probed live: fix the errors above first"}

Pass Criteria

  • Steps 1–13 pass.
  • Errors exit 1 and warnings alone exit 0.
  • project appears only on the mcp.projects server and holds the expanded root.
  • No step prints the value of SS_E2E_CHECK_TOKEN or SS_E2E_LIVE_TOKEN.
  • Without --live no server starts; with it, stdio probes fall back to initialize, a 401 is a warning, and a timed-out server leaves no process behind.