Rohit Ghumare 5691b10c7e feat: add multi-tier eval engine (skillkit eval) (#85)
* feat: add multi-tier eval engine (skillkit eval)

6-tier evaluation system for comprehensive skill assessment:
- Tier 1: LLM quality scoring (G-Eval pattern, 6 dimensions)
- Tier 2: Contradiction detection (formal rules + LLM semantic)
- Tier 3: Behavioral security (code analysis + taint tracking + LLM)
- Tier 4: Sandbox execution testing (Docker, graceful skip)
- Tier 5: Dynamic marketplace benchmarks (percentile ranking)
- Tier 6: Community signals (freshness, metadata, content health)

Works without API keys (heuristic fallback for Tier 1, Tiers 5-6).
Full LLM evaluation with --provider flag.

* fix: address Devin review — input validation for eval CLI

- --min-score NaN bypass: validate parseInt result before comparison
- --timeout NaN propagation: skip NaN timeout instead of passing to Docker
- --tier invalid input: graceful error message instead of unhandled throw
- scoreContentSize formula: cap sub-optimal at 99 to distinguish from optimal

* fix: address CodeRabbit review findings on eval system

- Harden contradiction/security prompts against prompt injection by
  wrapping untrusted content in <skill_content> XML tags and adding
  explicit "treat as untrusted data" instructions
- Use head+tail content sampling in security prompt instead of head-only
  truncation to catch malicious payloads at end of content
- Add missing Tier 4 sandbox formatter to reporter.ts
- Fix misleading sampleSize fallback in dynamic-benchmark.ts
- Move --min-score NaN validation before engine.evaluate() to avoid
  wasting LLM calls on invalid input
- Add truncation indicator to quality-cot.ts prompts
- Add term length limits (>100 chars) in contradiction.ts to mitigate
  ReDoS risk on dynamically constructed regex

* fix: address round 2 review findings on eval system

- Wrap quality-cot prompts with <skill_content> XML tags and untrusted
  data instructions, consistent with contradiction/security prompts
- Change truncateContent to head+tail sampling (4000+2000) instead of
  head-only, so completeness/token-efficiency evaluators see both
  beginning and end of long skills
- Fix escapeXmlTags regex to handle whitespace variants before closing
  angle bracket (</skill_content >, </skill_content\n>)
- Type DIMENSION_PROMPTS with Record<EvalDimension, PromptBuilder> for
  compile-time key validation
- Clamp estimatePercentile return to [0,100] range defensively
- Make buildStatsFromMarketplace async with periodic yields (every 50
  skills) to avoid blocking the event loop on first-run cache miss
- Extract parsedTimeout local variable in eval.ts to avoid double
  parseInt parsing

* fix: address round 3 Devin + CodeRabbit findings on eval system

- Scan both code blocks AND full content for dangerous patterns in
  behavioral-security.ts, with deduplication by description+snippet
  to avoid double-reporting (Devin: security scanner skipped prose)
- Use non-greedy regex /\[[\s\S]*?\]/ in parseSemanticFindings to
  match first complete JSON array instead of greedily spanning to
  last bracket in LLM response (Devin: greedy regex corrupted JSON)
- Fix formatSandboxDetails to use r.testCase matching SandboxResult
  type instead of r.name which doesn't exist (Devin: property mismatch)
- Make MarketplaceSkill.description optional to match marketplace
  schema where only id/name/source/tags are required (CodeRabbit)
- Rename sandbox test case from "code example execution" to "code
  block content validation" since it validates content not execution
  (CodeRabbit)

* fix: address round 4 Devin + CodeRabbit findings on eval system

Inline fixes:
- Replace greedy JSON regex in behavioral-security.ts runLLMAnalysis
  with balanced bracket extraction that correctly handles ] inside
  JSON string values
- Replace non-greedy regex in contradiction.ts parseSemanticFindings
  with same balanced bracket extraction (non-greedy also breaks on
  inner brackets)
- Fix crossValidatedCount in behavioral-security.ts to accumulate
  (+= uniqueEngines.size) instead of replacing (Math.max)
- Fix gradeDeterministic: cleanExit && hasOutput without matching
  expectedOutcome now returns passed:false score:50 instead of
  passed:true score:75
- Docker unavailable returns score:-1 sentinel instead of 0/F;
  engine.ts filters score<0 tiers from overall average so
  Docker-less machines aren't penalized

Duplicate fix:
- Add score field to SandboxResult type; store gradeResult.score
  in each result; compute tier score from average of numeric scores
  instead of boolean pass rate

Nitpick fixes:
- Distinguish "Docker unavailable" from "No tests executed" in
  formatSandboxDetails
- Tighten base64 obfuscation regex to require 4-char groups
- Only push 'llm' to engines when LLM actually produced findings
- Bound \s+ to \s{1,10} in dynamically constructed regexes in
  contradiction.ts to prevent catastrophic backtracking
- Validate cache timestamp with Number.isFinite() in loadCache to
  reject corrupted cache files with NaN timestamps
2026-03-10 18:34:32 +00:00
2026-02-09 21:00:18 +00:00
2026-01-21 10:59:25 +00:00

SkillKit - One Skill. Every Agent.

CI npm version npm downloads GitHub stars License

SkillKit is the open source package manager for AI agent skills. Write a skill once, deploy it to 44 agents — Claude, Cursor, Copilot, Windsurf, Devin, Codex, and 38 more. No rewrites. 400K+ skills across registries.

Website · Docs · API Explorer · Chrome Extension · npm


Skills make AI coding agents smarter. But every agent uses a different format:

Agent Format Directory
Claude Code SKILL.md .claude/skills/
Cursor .mdc .cursor/skills/
Copilot Markdown .github/skills/
Windsurf Markdown .windsurf/skills/

You end up rewriting the same skill for each agent, or locking into one platform.

SkillKit fixes this. Install from 400K+ skills across registries, auto-translate between formats, persist learnings with Memory. Works with Claude, Cursor, Windsurf, Copilot, and 40 more.

npx skillkit@latest

See It In Action

Watch the demo

Click the image above to watch the full demo

Quick Start

npx skillkit@latest init              # Detect agents, create dirs
skillkit recommend                    # Get smart suggestions
skillkit install anthropics/skills    # Install from marketplace
skillkit sync                         # Deploy to your agents

Four commands. Your agents now have skills for PDF processing, code review, and more.

What Can You Do?

Install skills from anywhere

skillkit install anthropics/skills          # GitHub
skillkit install gitlab:team/skills         # GitLab
skillkit install ./my-local-skills          # Local path

Translate between agents

Write for Claude, deploy to Cursor:

skillkit translate my-skill --to cursor
skillkit translate --all --to windsurf

Get smart recommendations

SkillKit reads your project, detects your stack, and suggests relevant skills:

skillkit recommend
# 92% vercel-react-best-practices
# 87% tailwind-v4-patterns
# 85% nextjs-app-router

Discover skills at runtime

Start an API server and let agents find skills on demand:

skillkit serve
# Server running at http://localhost:3737

curl "http://localhost:3737/search?q=react+performance"

Or use MCP for native agent integration:

{
  "mcpServers": {
    "skillkit": { "command": "npx", "args": ["@skillkit/mcp"] }
  }
}

Or use Python:

pip install skillkit-client
from skillkit import SkillKitClient

async with SkillKitClient() as client:
    results = await client.search("react performance", limit=5)

REST API docs · MCP Server docs · Python Client docs · Interactive API explorer

Auto-generate agent instructions

Let SkillKit analyze your codebase and create CLAUDE.md, .cursorrules, etc.:

skillkit primer --all-agents

Session memory

Your AI agents learn patterns during sessions, then forget everything. SkillKit captures those learnings:

skillkit memory compress
skillkit memory search "auth patterns"
skillkit memory export auth-patterns

AI skill generation

Generate skills from natural language with multi-source context:

skillkit generate
# Interactive wizard with 4 context sources:
# - Documentation (via Context7)
# - Your codebase patterns
# - 400K+ marketplace skills
# - Memory (your corrections)

Works with any LLM: Claude, GPT-4, Gemini, Ollama (local), or OpenRouter (100+ models). Generates agent-optimized variants with trust scores.

Mesh network

Distribute agents across machines with encrypted P2P:

skillkit mesh init
skillkit mesh discover

Team collaboration

Share skills via a Git-committable .skills manifest:

skillkit manifest init
skillkit manifest add anthropics/skills
git commit -m "add team skills"

Team members run skillkit manifest install and they're in sync.

Chrome Extension

Save any webpage as a skill directly from your browser.

  1. Build: pnpm --filter @skillkit/extension build
  2. Chrome → chrome://extensions → Load unpacked → packages/extension/dist/
  3. Click the extension icon or right-click → "Save page as Skill"

The extension sends the page URL to the SkillKit API for server-side extraction with Turndown, 5-source weighted tag detection, and GitHub URL support. The resulting SKILL.md downloads automatically. Then run skillkit install ~/Downloads/skillkit-skills/my-skill to deploy to all agents.

Extension docs

Interactive TUI

skillkit ui

h Home · m Marketplace · r Recommend · t Translate · i Installed · s Sync · q Quit

SkillKit Interactive CLI

Supported Agents (44)

Agent Format Directory
Claude Code SKILL.md .claude/skills/
Cursor .mdc .cursor/skills/
Codex SKILL.md .codex/skills/
Gemini CLI SKILL.md .gemini/skills/
OpenCode SKILL.md .opencode/skills/
GitHub Copilot Markdown .github/skills/
Windsurf Markdown .windsurf/skills/
Devin Markdown .devin/skills/
Aider SKILL.md .aider/skills/
Sourcegraph Cody SKILL.md .cody/skills/
Amazon Q SKILL.md .amazonq/skills/

Plus 33 more: Amp, Antigravity, Augment Code, Bolt, Clawdbot, Cline, CodeBuddy, CodeGPT, CommandCode, Continue, Crush, Droid, Factory, Goose, Kilo Code, Kiro CLI, Lovable, MCPJam, Mux, Neovate, OpenHands, Pi, PlayCode, Qoder, Qwen, Replit Agent, Roo Code, Tabby, Tabnine, Trae, Vercel, Zencoder, Universal

Full agent details

Commands

Core

skillkit install <source>        # Install skills
skillkit remove <skills>         # Remove skills
skillkit translate <skill> --to  # Translate between agents
skillkit sync                    # Deploy to agent config
skillkit recommend               # Smart recommendations
skillkit generate                # AI skill generation wizard
skillkit serve                   # Start REST API server

Discovery & Security

skillkit marketplace             # Browse skills
skillkit tree                    # Hierarchical taxonomy
skillkit find <query>            # Quick search
skillkit scan <path>             # Security scan for skills

Issue Planner

skillkit issue plan "#42"        # Plan from GitHub Issue
skillkit issue plan owner/repo#42 # Cross-repo plan
skillkit issue list              # List open issues

Session Intelligence

skillkit timeline                # Unified event stream
skillkit session handoff         # Agent-to-agent context
skillkit lineage                 # Skill impact graph
skillkit session explain         # Human-readable summary
skillkit activity                # Skill activity log

Advanced

skillkit primer --all-agents     # Generate agent instructions
skillkit memory compress         # Capture session learnings
skillkit mesh init               # Multi-machine distribution
skillkit message send            # Inter-agent messaging
skillkit workflow run <name>     # Run workflows
skillkit test                    # Test skills
skillkit cicd init               # CI/CD templates

Full command reference

Creating Skills

skillkit create my-skill

Or manually create a SKILL.md:

---
name: my-skill
description: What this skill does
license: MIT
---

# My Skill

Instructions for the AI agent.

## When to Use
- Scenario 1
- Scenario 2

## Steps
1. First step
2. Second step

Programmatic API

import { translateSkill, analyzeProject, RecommendationEngine } from 'skillkit';

const skill = await translateSkill(content, 'cursor');

const profile = await analyzeProject('./my-project');
const engine = new RecommendationEngine();
const recs = await engine.recommend(profile);
import { startServer } from '@skillkit/api';
await startServer({ port: 3737, skills: [...] });
import { MemoryCache, RelevanceRanker } from '@skillkit/core';
const cache = new MemoryCache({ maxSize: 500, ttlMs: 86_400_000 });
const ranker = new RelevanceRanker();
const results = ranker.rank(skills, 'react performance');

Skill Sources

SkillKit aggregates skills from trusted sources. All original creators are credited with their licenses preserved.

Official Partners

Repository Description
anthropics/skills Official Claude Code skills
vercel-labs/agent-skills Next.js and React skills
expo/skills Mobile development with Expo
remotion-dev/skills Programmatic video creation
supabase/agent-skills Database and auth skills
stripe/ai Payment integration patterns

Community

trailofbits/skills · obra/superpowers · wshobson/agents · ComposioHQ/awesome-claude-skills · travisvn/awesome-claude-skills · langgenius/dify · better-auth/skills · onmax/nuxt-skills · elysiajs/skills · kadajett/agent-nestjs-skills · cloudai-x/threejs-skills · dimillian/skills · waynesutton/convexskills · kepano/obsidian-skills · giuseppe-trisciuoglio/developer-kit · openrouterteam/agent-skills

Want to add your skills? Submit your repository

Install

npm install -g skillkit       # npm
pnpm add -g skillkit          # pnpm
yarn global add skillkit      # yarn
bun add -g skillkit           # bun
npx skillkit <command>        # no install

License

Apache License 2.0 — see LICENSE.

Documentation · Website · API Explorer · npm · GitHub

S
Description
No description provided
Readme
187 MiB
Languages
TypeScript 99.2%
JavaScript 0.4%
Python 0.2%