Files
Willie 94cf438066 fix(audit): match only uppercase role labels in source files (#923)
prompt-injection-1 kept reporting ordinary source lines as CRITICAL
prompt overrides: `root: Path = Path(".")` (#921), docstring arguments
such as `root: Repository root containing the manifests.`, and typed
parameters with a name default. #376 and #750 each added value shapes
to the exclude. Measured on 5,573 real Python files and 33,006 Markdown
and config files, widening the exclude again cleared no line, and a
docstring entry cannot be told from prose by any value pattern.

In a source file a lowercase label word is a parameter, field or
docstring entry, so the built-in rule now matches only uppercase labels
there. The file extension decides, never a shebang, so a text file
cannot switch the check off. Markdown, text, config and extensionless
files, and Markdown code fences, keep the case-insensitive match. A
custom regex under the same ID is used as written.

This reverses the earlier choice that a lowercase `system: "..."` in a
non-Markdown source file stays CRITICAL.

Closes #921
2026-11-16 22:13:30 +08:00

23 KiB
Raw Permalink Blame History

sidebar_position
sidebar_position
3

audit

Scan installed skills for security threats and malicious patterns.

skillshare audit                        # Scan all installed skills
skillshare audit <name>                 # Scan a specific installed skill
skillshare audit a b c                  # Scan multiple skills
skillshare audit --group frontend       # Scan all skills in a group
skillshare audit <path>                 # Scan a file/directory path
skillshare audit --threshold high       # Block on HIGH+ findings
skillshare audit -T h                   # Same as --threshold high
skillshare audit --format json           # JSON output
skillshare audit --format sarif         # SARIF 2.1.0 output (GitHub Code Scanning)
skillshare audit --format markdown      # Markdown report (for GitHub Issues/PRs)
skillshare audit --json                 # Same as --format json (deprecated)
skillshare audit -p                     # Scan project skills
skillshare audit --quiet                # Only show skills with findings
skillshare audit --yes                  # Skip large-scan confirmation
skillshare audit --no-tui               # Plain text output (no interactive TUI)
skillshare audit --profile strict       # Use strict profile (block on HIGH+)
skillshare audit --dedupe global        # Full composite-key deduplication
skillshare audit --analyzer static      # Run only the static analyzer
skillshare audit --analyzer static --analyzer dataflow  # Multiple analyzers

Display severity floor

skillshare audit --min-severity high
skillshare audit --format markdown --min-severity medium

--min-severity <level> shows findings at or above critical, high, medium, low, or info (case-insensitive; severity aliases such as h also work). The default shows all findings. It applies only to terminal findings, including the TUI, and Markdown tables/snippets. Human reports show visible and hidden counts even when every finding is hidden. Scores, maximum severity (summary.maxSeverity), block thresholds, exit codes, scan errors and incomplete-scan warnings still describe the full scan. JSON and SARIF ignore the display floor and keep their complete output unchanged; no display metadata or top-level severity alias is added.

In the dashboard, choose Minimum severity in the severity filter, then select a severity count to show that level and above. Exact severity preserves the previous selection behavior. Click the selected severity again to clear it. Without a selection, LOW/INFO findings retain their per-resource expander; visible/hidden counts include collapsed findings. Filtering never rescans or changes policy, scores or blocked status.

Accepted findings

skillshare audit --respect-accepted
skillshare audit --respect-accepted --threshold high --format json

Findings you accepted with --force during install or update are recorded in .metadata.json (see Accepted Findings). A plain audit still fails on them. With --respect-accepted, audit applies those acceptances the way the update gate does: the findings stay in the report, but they no longer fail it. A resource whose other findings are below the threshold counts as a warning, and the exit code is 0 unless another finding reaches the threshold. A changed finding needs a new acceptance.

Accepted findings are marked in every output format:

Format Mark
Text and TUI accepted earlier with --force under the finding; the text summary adds an Accepted row
JSON "acknowledged": true on the finding; summary.accepted counts them, and the severity counts such as summary.critical leave them out
Markdown (accepted earlier) after the message; the summary table adds an Accepted row
SARIF a suppressions entry with kind external

A skill inside a tracked repository also matches the acceptances recorded for the whole repository. Bundled hook and settings findings are the exception: their evidence names the file as the repository scan saw it, so they still fail a single-skill audit.

When to Use

  • Review security findings after installing a new skill
  • Scan all skills for prompt injection, data exfiltration, or credential access patterns
  • Customize audit rules for your organization's security policy
  • Generate audit reports for compliance (--format json), static analysis tools (--format sarif), or documentation (--format markdown)
  • Integrate into CI/CD pipelines to gate skill deployments
  • Upload SARIF results to GitHub Code Scanning for PR-level annotations

What It Detects

The audit engine scans every text-based file in a skill directory against 100+ built-in rules (regex patterns, table-driven credential detection, structural checks, content integrity verification, and supply-chain trust analysis), organized into 5 severity levels: CRITICAL, HIGH, MEDIUM, LOW, and INFO.

Markdown examples are still scanned. SDK-style system: / System: parameters in parsed fenced code blocks (prompt-injection-1) are reported as HIGH rather than CRITICAL when their value has a parameter-like shape. In non-Markdown source files, prompt-injection-1 matches only uppercase labels such as ROOT:. Generic disclosure restrictions such as "Do not tell the user they need to adopt an eval framework" use the HIGH rule prompt-injection-5 in all file types. Explicit concealment remains CRITICAL under prompt-injection-4. In .py files, built-in CRITICAL credential-access matches on comment-only lines and docstrings are reported as HIGH; see Credential Access. HIGH warns at the default CRITICAL threshold and blocks under strict; see Prompt Injection for limitations and rule-override compatibility.

For the full detection catalog, threat categories deep dive, risk scoring algorithm, command safety tiering, and cross-skill interaction analysis, see Audit Engine.

Example Output

skillshare audit
Audit  ~/.config/skillshare/skills
  global · blocks at CRITICAL · policy DEFAULT / dedupe:GLOBAL / analyzers:ALL

! ci-release-helper  HIGH · risk 25/100
✗ suspicious-skill   CRITICAL · risk 35/100

Summary
  Scanned      12 skills
✓ Passed       9
! Warning      2
✗ Failed       1
  Severity     1 critical, 2 high, 1 medium
  Threats      credential:1 exfiltration:1 injection:1 privilege:1
  Risk         HIGH 35/100 · 100% auditable

✗ Blocked 1 of 12 skills: findings at CRITICAL or above · 2.1s
  The risk score is informational; only the severity blocks

Next
  skillshare audit suspicious-skill  see its findings

Failed counts skills with findings at or above the active threshold (--threshold or config audit.block_threshold; default CRITICAL).

Threats shows a category breakdown of all findings using short names: inj (injection), exfil (exfiltration), cred (credential), obfusc (obfuscation), priv (privilege), integ (integrity), struct (structure), risk (risk). This line is omitted when there are no findings. In terminal output, each category is color-coded by threat type.

audit.block_threshold only controls the blocking threshold. It does not disable scanning.

Interactive TUI Mode

When scanning several skills in an interactive terminal, audit opens a full-screen view instead of printing results line by line: skills on the left with findings first, and the risk summary and findings of the selected skill on the right. Opening a skill shows its files at each finding, with the flagged line marked and its line number. The keys are listed at the bottom of the screen.

It opens only when the terminal is interactive, the output is not JSON, and there is more than one result. Use --no-tui to force plain text.

Large Scan Confirmation

When scanning more than 1,000 skills in an interactive terminal, the command prompts for confirmation before proceeding. Use --yes to skip this prompt in TTY environments (e.g., local automation scripts). In CI/CD pipelines (non-TTY), the prompt is automatically skipped.

Policy & Profiles

The audit command supports policy-driven configuration through profiles, deduplication modes, and analyzer selection. These can be set via CLI flags, project config, or global config.

Profiles

Profiles are presets that set sensible defaults for threshold and deduplication:

Profile Threshold Dedupe Use case
default CRITICAL global Standard behavior — block only critical threats
strict HIGH global Security-conscious teams — block high+ threats
permissive CRITICAL legacy Advisory-only — minimal blocking, no global dedup
skillshare audit --profile strict       # Block on HIGH+, global dedup
skillshare audit --profile permissive   # Advisory mode

Explicit flags always override profile defaults:

skillshare audit --profile strict --threshold medium  # strict profile but block on MEDIUM+

Deduplication

When the same finding is detected by multiple analyzers (e.g., both static and dataflow), deduplication removes redundant entries:

Mode Behavior
global Full composite-key dedup across all findings (default)
legacy Per-analyzer dedup only (pre-v0.16.9 behavior)

Analyzer Selection

By default all analyzers run. Use --analyzer to run only specific ones:

skillshare audit --analyzer static                    # Static pattern matching only
skillshare audit --analyzer static --analyzer dataflow # Multiple analyzers
Analyzer Scope Description
static Per-file Regex pattern matching and file-level whitespace-padding checks
dataflow Per-file Taint tracking for shell scripts and markdown code blocks
tier Per-skill Capability tier combination risk analysis
integrity Per-skill Content hash verification (file_hashes in SKILL.md)
metadata Per-skill Supply-chain trust verification (publisher mismatch, authority claims)
structure Per-skill Checks missing and out-of-root Markdown links/images and declared bundled paths; bundled hooks/settings
cross-skill Bundle Cross-skill exfiltration and privilege escalation analysis

You can also set this in config:

audit:
  enabled_analyzers: [static, dataflow]

Precedence

Settings resolve in this order (first non-empty wins):

  1. CLI flags (--profile, --threshold, --dedupe, --analyzer)
  2. Project config (.skillshare/config.yaml)
  3. Global config (~/.config/skillshare/config.yaml)
  4. Profile defaults

Automatic Scanning

Install-time

Skills are automatically scanned during installation. Findings at or above audit.block_threshold block installation (default: CRITICAL):

skillshare install /path/to/evil-skill
# Error: security audit failed: critical threats detected in skill

skillshare install /path/to/evil-skill --force
# Installs with warnings (use with caution)

skillshare install /path/to/skill --audit-threshold high
# Per-command block threshold override

skillshare install /path/to/skill -T h
# Same as --audit-threshold high

skillshare install /path/to/skill --skip-audit
# Bypasses scanning (use with caution)

--force overrides block decisions from findings; a scan that cannot run still blocks. --skip-audit disables scanning for that install command.

There is no config flag to globally disable install-time audit. Use --skip-audit only for commands where you intentionally want to bypass scanning.

Difference summary:

Install flag Audit runs? Findings available?
--force Yes Yes (installation still proceeds)
--skip-audit No No (scan is bypassed)

If both are provided, --skip-audit effectively wins because audit is not executed.

Update-time

skillshare update pulls tracked repos into an isolated candidate and audits it before publication. Findings at or above the active threshold (audit.block_threshold by default, or --audit-threshold / --threshold / -T override) block publication unless explicitly accepted; rejection leaves the active checkout unchanged. See update --skip-audit for details.

Findings accepted with --force are remembered for that skill. Acceptance keys ignore line numbers and include the rule, file, matched text, and any detector-specific evidence. For whitespace-padding, changed padded content produces a new key; density findings use all eligible file content as evidence. See Accepted Findings.

When updating tracked repos via install (skillshare install <repo> --track --update), the gate uses the same threshold policy (audit.block_threshold or --audit-threshold / --threshold / -T).

CI/CD Integration

The audit command is designed for pipeline automation. In non-TTY environments (CI runners, piped output), the interactive TUI and confirmation prompt are automatically disabled — no --yes or --no-tui needed.

For complete CI/CD workflows (GitHub Actions, GitLab CI, SARIF upload, output formats), see the CI/CD Skill Validation recipe.

Pre-commit Hook

Run skillshare audit automatically on every commit using the pre-commit framework. The hook scans files matching .skillshare/ or skills/ directories and blocks the commit if findings exceed your configured threshold.

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/runkids/skillshare
    rev: v0.16.11  # use latest release tag
    hooks:
      - id: skillshare-audit

See the Pre-commit Hook recipe for full setup instructions.

Best Practices

For Individual Developers

  • Audit before trusting — always run skillshare audit after installing skills from untrusted sources
  • Review findings, not just pass/fail — a "passed" skill may still have LOW/MEDIUM findings worth investigating
  • Read skill files — automated scanning catches known patterns, but novel attacks require human review

For Teams and Organizations

  • Set audit.block_threshold: HIGH — stricter than the default CRITICAL, catches obfuscation and destructive commands
  • Create organization-wide custom rules — add patterns for internal secret formats (e.g., corp-api-key-*)
  • Use project-mode rules for overrides — downgrade expected patterns per-project rather than globally
  1. Install: Skills are automatically scanned — blocked if threshold exceeded
  2. Periodic scan: Run skillshare audit regularly to catch rules updated after install
  3. Pre-commit hook: Catch issues before they're committed with the pre-commit framework
  4. CI gate: Add audit to your CI pipeline for shared skill repositories
  5. Custom rules: Tailor detection to your organization's threat model
  6. Review reports: Use --format json for compliance, --format sarif for GitHub Code Scanning, or --format markdown for GitHub Issues/PRs

Threshold Configuration

Set the blocking threshold in your config file:

# ~/.config/skillshare/config.yaml
audit:
  block_threshold: HIGH  # Block on HIGH or above (stricter than default CRITICAL)

Or per-command:

skillshare audit --threshold medium  # Block on MEDIUM or above

Full Audit Configuration

All audit settings can be persisted in config.yaml:

# ~/.config/skillshare/config.yaml (or .skillshare/config.yaml for project)
audit:
  block_threshold: HIGH                         # Blocking severity gate
  profile: strict                               # Profile preset (default/strict/permissive)
  dedupe_mode: global                           # Dedup mode (global/legacy)
  enabled_analyzers: [static, dataflow, tier]   # Limit to specific analyzers

CLI flags override config values. See Precedence for full resolution order.

The skillshare status command displays the resolved audit policy, showing the effective profile, threshold, dedupe mode, and analyzer list after applying all precedence layers.

Web UI

The audit feature is also available in the web dashboard at /audit:

skillshare ui
# Navigate to Audit page → Click "Run Audit"

Security Audit page in web dashboard

The Dashboard page includes a Security Audit section with a quick-scan summary.

The Audit page always applies accepted findings: it shows them muted with the note accepted earlier with --force, counts them under Accepted instead of their severity, and does not count them as blocked. Clear Show accepted to hide them. A skill's Audit tab collapses them behind their own expander, and the Dashboard's attention card leaves them out of the high-risk count.

Custom Rules Editor

The web dashboard includes a dedicated Audit Rules page at /audit/rules for creating and editing custom rules directly in the browser:

  • Create: If no audit-rules.yaml exists, click "Create Rules File" to scaffold one
  • Edit: YAML editor with syntax highlighting and validation
  • Save: Validates YAML format and regex patterns before saving

Access it from the Audit page via the "Custom Rules" button.

Exit Codes

Code Meaning
0 No scan errors and no findings at or above active threshold
1 One or more findings at or above active threshold, with no scan errors
2 Audit execution, usage, or configuration error, including a failed scan

With --respect-accepted, findings accepted earlier with --force do not count toward 1. Declining the large-scan confirmation exits 0 without scanning. Audits launched from list use the same exit codes.

Scan errors take precedence over findings. Partial structured reports are still emitted when available.

Scanned Files

The audit scans text-based files in skill directories:

  • .md, .txt, .yaml, .yml, .json, .toml
  • .sh, .bash, .zsh, .fish
  • .py, .js, .ts, .rb, .php, .php3, .php4, .php5, .phtml, .go, .rs, .swift
  • Files without extensions (e.g., Makefile, Dockerfile)

Text scanning skips hidden directories and is recursive within the remaining skill directory, so SKILL.md, nested references/*.md, and scripts/*.sh are all inspected when they match supported text file types. A whole-repository scan, such as the one install --track and tracked updates run, measures hidden directories inside each skill it contains, so a skill in .agents/skills/ or .curated/ gets the full text scan while target copies such as .claude/skills/ stay header-only.

Before text filtering, audit checks the first 512 bytes of each regular file (and reads a bounded PE header at its declared offset when needed) to classify executables, CPython bytecode, ZIP/JAR archives, and disguised shebang scripts. Hidden directories inside a skill receive header classification only; __pycache__ is also included. Symlinks to non-special files retain their existing text scanning; special targets are skipped and direct audits of them return an error; .git, .skillshare, internal metadata, OS metadata, symlinks and special files are excluded from header reads. Binary and oversized files receive no full text analysis. Summaries report binaryFiles, skippedFiles and oversizedFiles; these counts overlap. Header classification does not analyze runtime behavior or archive contents. See Executable and archive headers for formats, severity and limits. The structure analyzer also reads the skill’s .claude/settings.json and .claude/settings.local.json for bundled hooks and settings.

Options

Flag Description
-G, --group <name> Scan all skills in a group (repeatable)
-p, --project Scan project-level skills
-g, --global Scan global skills
--threshold <t>, -T <t> Block threshold: critical|high|medium|low|info (shorthand: c|h|m|l|i, plus crit, med)
--min-severity <level> Display findings at/above a severity; scores, gates and machine output stay unchanged
--profile <p> Audit profile preset: default, strict, permissive
--dedupe <mode> Dedup mode: legacy, global (default)
--analyzer <id> Only run specified analyzer (repeatable). IDs: static, dataflow, tier, integrity, metadata, structure, cross-skill
--format <f> Output format: text (default), json, sarif, markdown. JSON adds warnings and incomplete: true when a followed source link could not be read, so a partial scan is not mistaken for a full one; Markdown lists those links under Source links not followed, and the TUI summary line counts them
--json Output JSON (deprecated: use --format json)
--yes, -y Skip large-scan confirmation prompt (auto-confirms)
--quiet, -q Only show skills with findings + summary (suppress clean ✓ lines)
--no-tui Disable interactive TUI, print plain text output
--respect-accepted Do not fail on findings accepted earlier with --force; they are marked as accepted. See Accepted findings
--init-rules Create a starter audit-rules.yaml (respects -p/-g)
-h, --help Show help

Subcommands

Subcommand Description
rules Browse, enable, and disable audit rules (see audit rules)

Agent Support

skillshare audit agents scopes the security scan to agents only, scanning .md files in the agents source directory:

skillshare audit agents                    # Scan all agents
skillshare audit agents --threshold high   # Block on HIGH+ for agents
skillshare audit agents --format sarif     # SARIF output for agents
skillshare audit agents -p                 # Scan project agents

Agents are subject to the same audit rules, severity levels, and threshold gating as skills. Without the agents argument, audit scans skills only (default behavior). See Agents for background.

See Also