Commit Graph
4 Commits
Author SHA1 Message Date
Rohit Ghumare 0c82c58905 feat(cli): add --json output and loading spinners to remaining commands
Add --json flag with structured JSON output to: create, enable, disable,
eval, sync, tap (add/remove/list), translate, save, read.

Add loading spinners to: audit, fix, test, plugin, team, methodology,
hook, remove. Commands that already had spinners get --json-aware no-op
spinners when --json is active to suppress visual output.
2026-04-12 12:02:38 +01:00
Rohit Ghumare 9e50c8abe1 fix: prevent shell injection in action, suppress stdout in --json mode
- action.yml: pass inputs via env vars, whitelist allowed commands
- All --json commands: use no-op spinner to prevent stdout pollution
- remove: run AGENTS.md update before JSON output (was skipped)
- eval: gate spinner on json format, wrap in try/catch for cleanup
- scan: wrap spinner in try/catch for cleanup on failure
2026-04-12 10:56:35 +01:00
Rohit Ghumare 49fcc532d2 feat: add loading spinners to translate, sync, eval, publish, and scan commands (#88) 2026-04-12 10:41:46 +01:00
Rohit Ghumare 5691b10c7e feat: add multi-tier eval engine (skillkit eval) (#85)
* feat: add multi-tier eval engine (skillkit eval)

6-tier evaluation system for comprehensive skill assessment:
- Tier 1: LLM quality scoring (G-Eval pattern, 6 dimensions)
- Tier 2: Contradiction detection (formal rules + LLM semantic)
- Tier 3: Behavioral security (code analysis + taint tracking + LLM)
- Tier 4: Sandbox execution testing (Docker, graceful skip)
- Tier 5: Dynamic marketplace benchmarks (percentile ranking)
- Tier 6: Community signals (freshness, metadata, content health)

Works without API keys (heuristic fallback for Tier 1, Tiers 5-6).
Full LLM evaluation with --provider flag.

* fix: address Devin review — input validation for eval CLI

- --min-score NaN bypass: validate parseInt result before comparison
- --timeout NaN propagation: skip NaN timeout instead of passing to Docker
- --tier invalid input: graceful error message instead of unhandled throw
- scoreContentSize formula: cap sub-optimal at 99 to distinguish from optimal

* fix: address CodeRabbit review findings on eval system

- Harden contradiction/security prompts against prompt injection by
  wrapping untrusted content in <skill_content> XML tags and adding
  explicit "treat as untrusted data" instructions
- Use head+tail content sampling in security prompt instead of head-only
  truncation to catch malicious payloads at end of content
- Add missing Tier 4 sandbox formatter to reporter.ts
- Fix misleading sampleSize fallback in dynamic-benchmark.ts
- Move --min-score NaN validation before engine.evaluate() to avoid
  wasting LLM calls on invalid input
- Add truncation indicator to quality-cot.ts prompts
- Add term length limits (>100 chars) in contradiction.ts to mitigate
  ReDoS risk on dynamically constructed regex

* fix: address round 2 review findings on eval system

- Wrap quality-cot prompts with <skill_content> XML tags and untrusted
  data instructions, consistent with contradiction/security prompts
- Change truncateContent to head+tail sampling (4000+2000) instead of
  head-only, so completeness/token-efficiency evaluators see both
  beginning and end of long skills
- Fix escapeXmlTags regex to handle whitespace variants before closing
  angle bracket (</skill_content >, </skill_content\n>)
- Type DIMENSION_PROMPTS with Record<EvalDimension, PromptBuilder> for
  compile-time key validation
- Clamp estimatePercentile return to [0,100] range defensively
- Make buildStatsFromMarketplace async with periodic yields (every 50
  skills) to avoid blocking the event loop on first-run cache miss
- Extract parsedTimeout local variable in eval.ts to avoid double
  parseInt parsing

* fix: address round 3 Devin + CodeRabbit findings on eval system

- Scan both code blocks AND full content for dangerous patterns in
  behavioral-security.ts, with deduplication by description+snippet
  to avoid double-reporting (Devin: security scanner skipped prose)
- Use non-greedy regex /\[[\s\S]*?\]/ in parseSemanticFindings to
  match first complete JSON array instead of greedily spanning to
  last bracket in LLM response (Devin: greedy regex corrupted JSON)
- Fix formatSandboxDetails to use r.testCase matching SandboxResult
  type instead of r.name which doesn't exist (Devin: property mismatch)
- Make MarketplaceSkill.description optional to match marketplace
  schema where only id/name/source/tags are required (CodeRabbit)
- Rename sandbox test case from "code example execution" to "code
  block content validation" since it validates content not execution
  (CodeRabbit)

* fix: address round 4 Devin + CodeRabbit findings on eval system

Inline fixes:
- Replace greedy JSON regex in behavioral-security.ts runLLMAnalysis
  with balanced bracket extraction that correctly handles ] inside
  JSON string values
- Replace non-greedy regex in contradiction.ts parseSemanticFindings
  with same balanced bracket extraction (non-greedy also breaks on
  inner brackets)
- Fix crossValidatedCount in behavioral-security.ts to accumulate
  (+= uniqueEngines.size) instead of replacing (Math.max)
- Fix gradeDeterministic: cleanExit && hasOutput without matching
  expectedOutcome now returns passed:false score:50 instead of
  passed:true score:75
- Docker unavailable returns score:-1 sentinel instead of 0/F;
  engine.ts filters score<0 tiers from overall average so
  Docker-less machines aren't penalized

Duplicate fix:
- Add score field to SandboxResult type; store gradeResult.score
  in each result; compute tier score from average of numeric scores
  instead of boolean pass rate

Nitpick fixes:
- Distinguish "Docker unavailable" from "No tests executed" in
  formatSandboxDetails
- Tighten base64 obfuscation regex to require 4-char groups
- Only push 'llm' to engines when LLM actually produced findings
- Bound \s+ to \s{1,10} in dynamically constructed regexes in
  contradiction.ts to prevent catastrophic backtracking
- Validate cache timestamp with Number.isFinite() in loadCache to
  reject corrupted cache files with NaN timestamps
2026-03-10 18:34:32 +00:00