Add --json flag with structured JSON output to: create, enable, disable,
eval, sync, tap (add/remove/list), translate, save, read.
Add loading spinners to: audit, fix, test, plugin, team, methodology,
hook, remove. Commands that already had spinners get --json-aware no-op
spinners when --json is active to suppress visual output.
- action.yml: pass inputs via env vars, whitelist allowed commands
- All --json commands: use no-op spinner to prevent stdout pollution
- remove: run AGENTS.md update before JSON output (was skipped)
- eval: gate spinner on json format, wrap in try/catch for cleanup
- scan: wrap spinner in try/catch for cleanup on failure
* feat: add multi-tier eval engine (skillkit eval)
6-tier evaluation system for comprehensive skill assessment:
- Tier 1: LLM quality scoring (G-Eval pattern, 6 dimensions)
- Tier 2: Contradiction detection (formal rules + LLM semantic)
- Tier 3: Behavioral security (code analysis + taint tracking + LLM)
- Tier 4: Sandbox execution testing (Docker, graceful skip)
- Tier 5: Dynamic marketplace benchmarks (percentile ranking)
- Tier 6: Community signals (freshness, metadata, content health)
Works without API keys (heuristic fallback for Tier 1, Tiers 5-6).
Full LLM evaluation with --provider flag.
* fix: address Devin review — input validation for eval CLI
- --min-score NaN bypass: validate parseInt result before comparison
- --timeout NaN propagation: skip NaN timeout instead of passing to Docker
- --tier invalid input: graceful error message instead of unhandled throw
- scoreContentSize formula: cap sub-optimal at 99 to distinguish from optimal
* fix: address CodeRabbit review findings on eval system
- Harden contradiction/security prompts against prompt injection by
wrapping untrusted content in <skill_content> XML tags and adding
explicit "treat as untrusted data" instructions
- Use head+tail content sampling in security prompt instead of head-only
truncation to catch malicious payloads at end of content
- Add missing Tier 4 sandbox formatter to reporter.ts
- Fix misleading sampleSize fallback in dynamic-benchmark.ts
- Move --min-score NaN validation before engine.evaluate() to avoid
wasting LLM calls on invalid input
- Add truncation indicator to quality-cot.ts prompts
- Add term length limits (>100 chars) in contradiction.ts to mitigate
ReDoS risk on dynamically constructed regex
* fix: address round 2 review findings on eval system
- Wrap quality-cot prompts with <skill_content> XML tags and untrusted
data instructions, consistent with contradiction/security prompts
- Change truncateContent to head+tail sampling (4000+2000) instead of
head-only, so completeness/token-efficiency evaluators see both
beginning and end of long skills
- Fix escapeXmlTags regex to handle whitespace variants before closing
angle bracket (</skill_content >, </skill_content\n>)
- Type DIMENSION_PROMPTS with Record<EvalDimension, PromptBuilder> for
compile-time key validation
- Clamp estimatePercentile return to [0,100] range defensively
- Make buildStatsFromMarketplace async with periodic yields (every 50
skills) to avoid blocking the event loop on first-run cache miss
- Extract parsedTimeout local variable in eval.ts to avoid double
parseInt parsing
* fix: address round 3 Devin + CodeRabbit findings on eval system
- Scan both code blocks AND full content for dangerous patterns in
behavioral-security.ts, with deduplication by description+snippet
to avoid double-reporting (Devin: security scanner skipped prose)
- Use non-greedy regex /\[[\s\S]*?\]/ in parseSemanticFindings to
match first complete JSON array instead of greedily spanning to
last bracket in LLM response (Devin: greedy regex corrupted JSON)
- Fix formatSandboxDetails to use r.testCase matching SandboxResult
type instead of r.name which doesn't exist (Devin: property mismatch)
- Make MarketplaceSkill.description optional to match marketplace
schema where only id/name/source/tags are required (CodeRabbit)
- Rename sandbox test case from "code example execution" to "code
block content validation" since it validates content not execution
(CodeRabbit)
* fix: address round 4 Devin + CodeRabbit findings on eval system
Inline fixes:
- Replace greedy JSON regex in behavioral-security.ts runLLMAnalysis
with balanced bracket extraction that correctly handles ] inside
JSON string values
- Replace non-greedy regex in contradiction.ts parseSemanticFindings
with same balanced bracket extraction (non-greedy also breaks on
inner brackets)
- Fix crossValidatedCount in behavioral-security.ts to accumulate
(+= uniqueEngines.size) instead of replacing (Math.max)
- Fix gradeDeterministic: cleanExit && hasOutput without matching
expectedOutcome now returns passed:false score:50 instead of
passed:true score:75
- Docker unavailable returns score:-1 sentinel instead of 0/F;
engine.ts filters score<0 tiers from overall average so
Docker-less machines aren't penalized
Duplicate fix:
- Add score field to SandboxResult type; store gradeResult.score
in each result; compute tier score from average of numeric scores
instead of boolean pass rate
Nitpick fixes:
- Distinguish "Docker unavailable" from "No tests executed" in
formatSandboxDetails
- Tighten base64 obfuscation regex to require 4-char groups
- Only push 'llm' to engines when LLM actually produced findings
- Bound \s+ to \s{1,10} in dynamically constructed regexes in
contradiction.ts to prevent catastrophic backtracking
- Validate cache timestamp with Number.isFinite() in loadCache to
reject corrupted cache files with NaN timestamps