Unit tests use a fake server built from the test binary, so nothing exercised --live against separate real processes, a real HTTP server or the process tree a server leaves behind. Steps 8-13 use sh stdio servers (server/discover, initialize only, one that ignores SIGTERM and starts a child, one that crashes), a Python server that answers 401, and an unreachable URL, and check that nothing starts without --live. Refs #141
16 KiB
CLI E2E Runbook: skillshare mcp check
Scope
Verify the read-only MCP check against a global config whose servers cover every
finding: an unset fromEnv variable, a command missing from PATH, a .invalid host,
unsynced targets, a client rule refusal, targets: [], and one mcp.projects root.
Covers human output, the --json shape including project, --no-dns, name
filtering, exit codes 0 and 1, and GET /api/mcp/check with and without ?dns=0.
Steps 1–7 launch no server, run no command, and sync nothing.
Steps 8–13 cover --live against a second config: tiny local stdio servers written as
sh scripts (one answers server/discover, one only the initialize handshake, one
never answers, one crashes), a local HTTP server that answers 401 with resource
metadata, and an unreachable HTTP URL. They check that no server starts without
--live, the live JSON object, the human output, the 401 warning, the timeout and
that it leaves no process behind, and that values are redacted.
Environment
Run inside the devcontainer with a fresh ssenv HOME:
ssenv create mcp-check-e2e --init
cd /tmp
ssenv enter mcp-check-e2e -- mdproof --report json /workspace/ai_docs/tests/mcp_check_runbook.md
Human-output steps set NO_COLOR=1 so the status marks can be matched as plain text.
Every step works from $HOME and points SKILLSHARE_CONFIG at
$HOME/mcp-check/config.yaml. Never run these steps with the repository root as the
working directory: the CLI would pick up the repository's .skillshare/ as project
mode. The check needs DNS only for Step 4; .invalid never resolves, online or not.
Step 7 starts skillshare ui on port 47931 and Step 11 a Python HTTP server on port
47932; pick other free ports if they are taken. Port 1 on 127.0.0.1 must refuse
connections.
Steps
Step 1: Write the global config
The skills directory must exist before skillshare ui starts. mcp list would refuse
this config because of the workspace server, so the step reads it back through a check
of the one server that has no problems.
set -eu
cd "$HOME"
CASE="$HOME/mcp-check"
rm -rf "$CASE"
mkdir -p "$CASE/skills" "$CASE/app"
export SKILLSHARE_CONFIG="$CASE/config.yaml"
cat > "$SKILLSHARE_CONFIG" <<YAML
source: $CASE/skills
targets: {}
mcp:
targets: [claude, cursor]
servers:
docs:
url: https://example.com/docs
envless:
command: sh
env:
API_KEY: {fromEnv: SS_E2E_CHECK_TOKEN}
ghost:
command: no-such-mcp-binary
offline:
url: https://mcp.skillshare.invalid/mcp
targets: [cursor]
parked:
url: https://example.com/parked
targets: []
workspace:
url: https://example.com/workspace
projects:
~/mcp-check/app:
targets: [claude]
servers:
docs:
command: no-such-project-binary
YAML
ss mcp check parked --json -g
Expected:
- exit_code: 0
- jq: [.servers[].name] == ["parked"]
- jq: .summary == {"errors": 0, "warnings": 0}
Step 2: Human output lists every server and exits 1
cd "$HOME"
export SKILLSHARE_CONFIG="$HOME/mcp-check/config.yaml"
unset SS_E2E_CHECK_TOKEN
NO_COLOR=1 ss mcp check --no-dns -g
Expected:
- exit_code: 1
- ✗ envless
- ✗ env API_KEY reads SS_E2E_CHECK_TOKEN, which is not set
- ✗ command no-such-mcp-binary was not found on PATH
- ✗ claude: Claude MCP workspace: Claude Code reserves this name
- ! cursor: not synced yet; run skillshare sync mcp
- · kept in Skillshare only; no Agent receives it
- ✗ docs (project ~/mcp-check/app)
- ✗ command no-such-project-binary was not found on PATH
- 7 server(s) checked: 4 error(s), 9 warning(s)
Step 3: JSON shape and the project field
cd "$HOME"
export SKILLSHARE_CONFIG="$HOME/mcp-check/config.yaml"
unset SS_E2E_CHECK_TOKEN
ss mcp check --json --no-dns -g
Expected:
- exit_code: 1
- jq: [.servers[].name] == ["docs", "envless", "ghost", "offline", "parked", "workspace", "docs"]
- jq: [.servers[] | select(has("project"))] | length == 1
- jq: .servers[6].project | startswith("/") and endswith("/mcp-check/app")
- jq: .servers[6].ok == false and .servers[6].findings[0].check == "command" and .servers[6].findings[0].subject == "no-such-project-binary"
- jq: .servers[6].findings[1] == {"level": "warning", "check": "sync", "target": "claude", "message": "not synced yet; run skillshare sync mcp"}
- jq: .servers[1].findings[0] | .check == "env" and .subject == "SS_E2E_CHECK_TOKEN" and .target == ""
- jq: .servers[5].findings[0] | .check == "client-rule" and .target == "claude"
- jq: .servers[4] | .ok == true and .findings == [{"level": "info", "check": "targets", "target": "", "message": "kept in Skillshare only; no Agent receives it"}]
- jq: [.servers[].findings[] | select(.check == "dns")] == []
- jq: .summary == {"errors": 4, "warnings": 9}
Step 4: DNS warning, and --no-dns skips it
A DNS warning never fails the check, so both runs exit 0.
set -eu
cd "$HOME"
export SKILLSHARE_CONFIG="$HOME/mcp-check/config.yaml"
ss mcp check offline --json -g > "$HOME/mcp-check/dns.json"
ss mcp check offline --json --no-dns -g > "$HOME/mcp-check/no-dns.json"
jq -n --slurpfile dns "$HOME/mcp-check/dns.json" --slurpfile skip "$HOME/mcp-check/no-dns.json" '{dns: $dns[0], skip: $skip[0]}'
Expected:
- exit_code: 0
- jq: .dns.servers[0].findings[0] == {"level": "warning", "check": "dns", "target": "", "message": "host mcp.skillshare.invalid did not resolve", "subject": "mcp.skillshare.invalid"}
- jq: .dns.servers[0].ok == true and .dns.summary == {"errors": 0, "warnings": 2}
- jq: [.skip.servers[0].findings[].check] == ["sync"]
Step 5: Name filtering, unknown names, and exit 0
set -eu
cd "$HOME"
export SKILLSHARE_CONFIG="$HOME/mcp-check/config.yaml"
ss mcp check docs --json --no-dns -g > "$HOME/mcp-check/docs.json" || test $? -eq 1
if ss mcp check nope -g > "$HOME/mcp-check/unknown.txt" 2>&1; then echo "unknown name must fail"; exit 1; fi
ss mcp check parked -g > "$HOME/mcp-check/parked.txt"
jq -c '[.servers[] | {name, project}]' "$HOME/mcp-check/docs.json"
cat "$HOME/mcp-check/unknown.txt" "$HOME/mcp-check/parked.txt"
Expected:
- exit_code: 0
- regex: [{"name":"docs","project":null},{"name":"docs","project":"[^"]+/mcp-check/app"}]
- unknown MCP server "nope"; known servers: docs, envless, ghost, offline, parked, workspace
- 1 server(s) checked: 0 error(s), 0 warning(s)
Step 6: A set variable passes and its value is never printed
set -eu
cd "$HOME"
export SKILLSHARE_CONFIG="$HOME/mcp-check/config.yaml"
SS_E2E_CHECK_TOKEN=s3cr3t-e2e-value NO_COLOR=1 ss mcp check envless --no-dns -g > "$HOME/mcp-check/envless.txt"
SS_E2E_CHECK_TOKEN=s3cr3t-e2e-value ss mcp check envless --json --no-dns -g >> "$HOME/mcp-check/envless.txt"
if grep -q s3cr3t-e2e-value "$HOME/mcp-check/envless.txt"; then echo "value leaked"; exit 1; fi
head -n 5 "$HOME/mcp-check/envless.txt"
Expected:
- exit_code: 0
- ! envless
- 1 server(s) checked: 0 error(s), 2 warning(s)
Step 7: The dashboard endpoint, with and without ?dns=0
set -eu
cd "$HOME"
export SKILLSHARE_CONFIG="$HOME/mcp-check/config.yaml"
unset SS_E2E_CHECK_TOKEN
PORT=47931
ss ui --port "$PORT" --no-open -g > "$HOME/mcp-check/ui.log" 2>&1 &
UI_PID=$!
trap 'kill "$UI_PID" 2>/dev/null || true' EXIT
for _ in $(seq 1 100); do
curl -fsS "http://127.0.0.1:$PORT/api/mcp/check?dns=0" > "$HOME/mcp-check/api-no-dns.json" 2>/dev/null && break
sleep 0.2
done
curl -fsS "http://127.0.0.1:$PORT/api/mcp/check" > "$HOME/mcp-check/api-dns.json"
jq -n --slurpfile skip "$HOME/mcp-check/api-no-dns.json" --slurpfile dns "$HOME/mcp-check/api-dns.json" '{skip: $skip[0], dns: $dns[0]}'
Expected:
- exit_code: 0
- jq: .skip.summary == {"errors": 4, "warnings": 9}
- jq: [.skip.servers[] | select(.project)] | length == 1
- jq: [.skip.servers[].findings[] | select(.check == "dns")] == []
- jq: .dns.summary == {"errors": 4, "warnings": 10}
- jq: [.dns.servers[].findings[] | select(.check == "dns") | .subject] == ["mcp.skillshare.invalid"]
Step 8: Write the live config; without --live nothing starts
Every script writes a marker file when it starts, so the step can prove that a
check without --live starts none of them.
set -eu
cd "$HOME"
CASE="$HOME/mcp-check-live"
rm -rf "$CASE"
mkdir -p "$CASE/skills"
export SKILLSHARE_CONFIG="$CASE/config.yaml"
cat > "$CASE/tiny.sh" <<'SH'
#!/bin/sh
touch "$MARKER"
while IFS= read -r line; do
id=$(printf '%s\n' "$line" | sed -n 's/.*"id":\([0-9][0-9]*\).*/\1/p')
case "$line" in
*'"server/discover"'*) printf '{"jsonrpc":"2.0","id":%s,"result":{"resultType":"complete","supportedVersions":["2026-07-28"],"capabilities":{"tools":{}},"_meta":{"io.modelcontextprotocol/serverInfo":{"name":"tiny","version":"0.1.0"}}}}\n' "$id" ;;
*'"tools/list"'*) printf '{"jsonrpc":"2.0","id":%s,"result":{"resultType":"complete","tools":[{"name":"echo","inputSchema":{"type":"object"}}]}}\n' "$id" ;;
esac
done
SH
cat > "$CASE/legacy.sh" <<'SH'
#!/bin/sh
touch "$MARKER"
while IFS= read -r line; do
id=$(printf '%s\n' "$line" | sed -n 's/.*"id":\([0-9][0-9]*\).*/\1/p')
case "$line" in
*'"notifications/initialized"'*) ;;
*'"method":"initialize"'*) printf '{"jsonrpc":"2.0","id":%s,"result":{"protocolVersion":"2025-11-25","capabilities":{"tools":{}},"serverInfo":{"name":"legacy","version":"0.9"}}}\n' "$id" ;;
*'"tools/list"'*) printf '{"jsonrpc":"2.0","id":%s,"result":{"tools":[{"name":"a"},{"name":"b"}]}}\n' "$id" ;;
*) printf '{"jsonrpc":"2.0","id":%s,"error":{"code":-32601,"message":"Method not found"}}\n' "$id" ;;
esac
done
SH
cat > "$CASE/hang.sh" <<'SH'
#!/bin/sh
touch "$MARKER"
trap '' TERM
sleep 3001 &
while :; do sleep 1; done
SH
cat > "$CASE/crash.sh" <<'SH'
#!/bin/sh
touch "$MARKER"
echo "fatal: token $API_TOKEN rejected" >&2
exit 3
SH
cat > "$SKILLSHARE_CONFIG" <<YAML
source: $CASE/skills
targets: {}
mcp:
targets: [claude]
servers:
crash:
command: sh
args: [$CASE/crash.sh]
env:
MARKER: $CASE/started-crash
API_TOKEN: {fromEnv: SS_E2E_LIVE_TOKEN}
down:
url: http://127.0.0.1:1/mcp
hang:
command: sh
args: [$CASE/hang.sh]
env:
MARKER: $CASE/started-hang
legacy:
command: sh
args: [$CASE/legacy.sh]
env:
MARKER: $CASE/started-legacy
signin:
url: http://127.0.0.1:47932/mcp
bearerToken: {fromEnv: SS_E2E_LIVE_TOKEN}
tiny:
command: sh
args: [$CASE/tiny.sh]
env:
MARKER: $CASE/started-tiny
YAML
SS_E2E_LIVE_TOKEN=s3cr3t-live-e2e ss mcp check --json --no-dns -g > "$CASE/static.json"
ls "$CASE" | grep -c '^started-' || true
jq -c '{live: [.servers[] | select(has("live"))], checks: [.servers[].findings[].check] | unique}' "$CASE/static.json"
Expected:
- exit_code: 0
- regex: ^0$
- jq: .live == [] and .checks == ["sync"]
Step 9: --live reports serverInfo, protocol and tools
set -eu
cd "$HOME"
export SKILLSHARE_CONFIG="$HOME/mcp-check-live/config.yaml"
ss mcp check legacy tiny --live --json -g
Expected:
- exit_code: 0
- jq: [.servers[].name] == ["legacy", "tiny"]
- jq: .servers[1].live == {"protocolVersion": "2026-07-28", "serverInfo": {"name": "tiny", "version": "0.1.0"}, "tools": 1}
- jq: .servers[0].live == {"protocolVersion": "2025-11-25", "serverInfo": {"name": "legacy", "version": "0.9"}, "tools": 2}
- jq: [.servers[].findings[] | select(.check == "live") | .level] == ["info", "info"]
- jq: .summary.errors == 0
Step 10: Human output of a live probe
cd "$HOME"
export SKILLSHARE_CONFIG="$HOME/mcp-check-live/config.yaml"
NO_COLOR=1 ss mcp check tiny --live -g
Expected:
- exit_code: 0
- · responds: tiny 0.1.0, protocol 2026-07-28, 1 tool(s)
- 1 server(s) checked: 0 error(s), 1 warning(s)
Step 11: 401 is a sign-in warning; an unreachable URL is an error
set -eu
cd "$HOME"
CASE="$HOME/mcp-check-live"
export SKILLSHARE_CONFIG="$CASE/config.yaml"
cat > "$CASE/signin.py" <<'PY'
import http.server
class Handler(http.server.BaseHTTPRequestHandler):
def do_POST(self):
self.send_response(401)
self.send_header("WWW-Authenticate", 'Bearer resource_metadata="http://127.0.0.1:47932/.well-known/oauth-protected-resource", scope="read"')
self.send_header("Content-Length", "0")
self.end_headers()
def log_message(self, *args):
pass
http.server.HTTPServer(("127.0.0.1", 47932), Handler).serve_forever()
PY
python3 "$CASE/signin.py" &
PY_PID=$!
trap 'kill "$PY_PID" 2>/dev/null || true' EXIT
for _ in $(seq 1 50); do
curl -s -o /dev/null -X POST http://127.0.0.1:47932/mcp && break
sleep 0.1
done
SS_E2E_LIVE_TOKEN=s3cr3t-live-e2e ss mcp check down signin --live --json -g > "$CASE/http.json" || test $? -eq 1
if grep -q s3cr3t-live-e2e "$CASE/http.json"; then echo "value leaked"; exit 1; fi
cat "$CASE/http.json"
Expected:
- exit_code: 0
- jq: [.servers[].name] == ["down", "signin"]
- jq: .servers[1].ok == true and (.servers[1] | has("live") | not)
- jq: [.servers[1].findings[] | select(.check == "live")][0] == {"level": "warning", "check": "live", "target": "", "message": "sign-in required (HTTP 401): resource metadata at http://127.0.0.1:47932/.well-known/oauth-protected-resource; skillshare does not sign in", "subject": "http://127.0.0.1:47932/.well-known/oauth-protected-resource"}
- jq: .servers[0].ok == false and ([.servers[0].findings[] | select(.check == "live")][0].message | startswith("live probe failed:") and contains("connection refused"))
- jq: .summary.errors == 1
Step 12: The timeout stops the server and everything it started
hang.sh ignores SIGTERM and starts sleep 3001, so only the process-group SIGKILL
stops both.
set -eu
cd "$HOME"
CASE="$HOME/mcp-check-live"
export SKILLSHARE_CONFIG="$CASE/config.yaml"
START=$(date +%s)
ss mcp check hang --live --timeout 2s --json -g > "$CASE/hang.json" || test $? -eq 1
ELAPSED=$(( $(date +%s) - START ))
test "$ELAPSED" -lt 8
sleep 0.5
# The brackets keep pgrep from matching this script's own command line.
LEFT=$(pgrep -fc 'sleep 300[1]|hang[.]sh' || true)
jq -c --arg left "$LEFT" '{left: $left, finding: [.servers[0].findings[] | select(.check == "live")][0]}' "$CASE/hang.json"
Expected:
- exit_code: 0
- jq: .left == "0"
- jq: .finding == {"level": "error", "check": "live", "target": "", "message": "live probe failed: no answer within 2s"}
Step 13: A crash shows redacted stderr; a static error skips the probe
set -eu
cd "$HOME"
CASE="$HOME/mcp-check-live"
export SKILLSHARE_CONFIG="$CASE/config.yaml"
rm -f "$CASE/started-crash"
SS_E2E_LIVE_TOKEN=s3cr3t-live-e2e ss mcp check crash --live --json -g > "$CASE/crash.json" || test $? -eq 1
env -u SS_E2E_LIVE_TOKEN ss mcp check crash --live --json -g > "$CASE/skipped.json" || test $? -eq 1
if grep -q s3cr3t-live-e2e "$CASE/crash.json"; then echo "value leaked"; exit 1; fi
jq -n --slurpfile crash "$CASE/crash.json" --slurpfile skip "$CASE/skipped.json" '{crash: [$crash[0].servers[0].findings[] | select(.check == "live")][0], skip: [$skip[0].servers[0].findings[] | select(.check == "live")][0]}'
Expected:
- exit_code: 0
- jq: .crash.level == "error" and (.crash.message | contains("exit status 3")) and (.crash.message | endswith("stderr: fatal: token [redacted] rejected"))
- jq: .skip == {"level": "info", "check": "live", "target": "", "message": "not probed live: fix the errors above first"}
Pass Criteria
- Steps 1–13 pass.
- Errors exit 1 and warnings alone exit 0.
projectappears only on themcp.projectsserver and holds the expanded root.- No step prints the value of
SS_E2E_CHECK_TOKENorSS_E2E_LIVE_TOKEN. - Without
--liveno server starts; with it, stdio probes fall back toinitialize, a 401 is a warning, and a timed-out server leaves no process behind.