Files
skillshare/internal/install/install_discovery.go
T
a352b8d243 feat(install): pin skills to a tag or commit from their web URL (#293)
* feat(install): pin GitHub tree/blob URLs to their ref

Hub entries only carry a source URL, so a team hub had no way to pin a
third-party skill to a reviewed tag or commit. The parser already read
tree/<ref>/ and blob/<ref>/ URLs but discarded the ref, installing from
the default branch instead of the version the URL names.

Keep that ref as Source.Branch. An explicit --branch (CLI) or branch
(API body) still overrides it; the server handlers now only override
when a branch is given. Batch installs group by repo and ref so two
pinned versions of one repo get separate clones.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs(install): describe pinning hub sources to a tag or commit

Explain that a GitHub tree/<ref>/ source URL pins the install, in both
the hub index guide and the install reference.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat(install): pin GitLab/Bitbucket URLs and resolve refs containing "/"

Pinning should not depend on the host, so GitLab (-/tree/<ref>/) and
Bitbucket (src/<ref>/) web URLs now keep their ref too.

Taking the first path segment as the ref broke branch names containing
"/": tree/feature/x/skills/foo cloned branch "feature" and failed. When
the first segment is not a branch or tag on the remote, the longest
matching one is used instead. A ref the remote does not have fails
with a hint to use --branch, rather than silently installing the
default branch. A saved branch that covers more of the path (from
--branch or config) moves the subdir without asking the remote.

The grouped-install round-trip test pinned -/tree/main/ against a repo
on master; the ref used to be ignored, so its branch is now renamed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs(install): cover GitLab, Bitbucket and "/" branches in URL pinning

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(install): keep web URL refs correct on reinstall, --branch and grouping

Review of the slash-branch resolution found four gaps:

- A saved branch equal to the whole URL tail (tree/feature/x) did not
  move the subdir back, so every reinstall or update looked for "x".
- An unrelated --branch with a slash-branch URL kept the parse-time
  subdir (x/skills/foo) instead of the URL's real one.
- Grouped installs copy the source to a repo root; resolution then put
  the subdir back and discovery looked under a doubled path. Grouping
  also keyed on the unresolved first segment, so tree/feature/x and
  tree/feature/y could share one clone. Web URLs with a ref now install
  on their own, and resolution leaves a repo-root copy at its root.
- A skill named after the parse-time subdir kept that name once the
  path turned out to be all ref (tree/feature/x named the skill "x").

The unknown-ref error now reads as a missing ref first, since a stale
ref (tree/master after a rename) is more common than a slash branch.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(update): keep legacy installs on the default branch

Skills installed before URL refs were honoured got the default branch
and recorded no branch. Single-skill update re-parses the recorded
source, so it moved them to the URL's ref (or failed on a stale one,
with a hint to pass --branch, which update does not accept), while
update --all rebuilds from RepoURL and kept the default branch.

ApplyRecordedBranch treats an empty recorded branch as the remote
default and drops the URL's ref. It replaces the branch handling at
every site that reinstalls from recorded metadata: CLI and server
update for skills and agents, and tracked-repo rehydration.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* perf(search): share one clone across pinned hub skills from a repo

Any web URL with a path after its ref installed on its own, so a hub
listing many tree/<tag>/ skills from one repo cloned it once per skill.

groupByRepo now resolves each URL's ref before grouping, with a
RemoteRefs cache that lists each remote's refs once, and groups by
repo and resolved ref. Only refs containing "/" (whose subdir the
discovery match cannot re-derive) and refs that fail to resolve still
install alone. A resolved source is marked settled so the grouped
clone does not resolve it again, and ApplyRecordedBranch settles the
subdir from a recorded branch without asking the remote.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Willie <egg358358@gmail.com>
2026-09-28 22:08:16 +08:00

529 lines
15 KiB
Go

package install
import (
"fmt"
"os"
"path/filepath"
"strings"
"skillshare/internal/skillignore"
"skillshare/internal/utils"
)
// TargetDotDirs is the set of hidden directory names (e.g. ".claude", ".cursor")
// to skip during skill discovery. Set by the CLI entrypoint from config data
// to avoid a circular import between install and config packages.
var TargetDotDirs map[string]bool
func discoverFromGitWithProgressImpl(source *Source, onProgress ProgressCallback) (*DiscoveryResult, error) {
if !isGitInstalled() {
return nil, fmt.Errorf("git is not installed or not in PATH")
}
if err := resolveWebRef(source); err != nil {
return nil, err
}
// Clone to temp directory
tempDir, err := os.MkdirTemp("", "skillshare-discover-*")
if err != nil {
return nil, fmt.Errorf("failed to create temp directory: %w", err)
}
repoPath := filepath.Join(tempDir, "repo")
if err := cloneRepoForSource(source, repoPath, source.Branch, true, onProgress); err != nil {
cleanupTempRepo(tempDir)
return nil, fmt.Errorf("failed to clone repository: %w", err)
}
// Discover skills (include root to support single-skill-at-root repos)
skills := discoverSkills(repoPath, true)
// Fix root skill name: temp dir gives random name, use source.Name instead
for i := range skills {
if skills[i].Path == "." {
skills[i].Name = source.Name
break
}
}
// Discover agents (agents/ dir or pure agent repo fallback)
agents := discoverAgents(repoPath, len(skills) > 0)
skills, agents, err = constrainDiscoveryToExplicitSkill(source, skills, agents)
if err != nil {
cleanupTempRepo(tempDir)
return nil, err
}
commitHash, _ := getGitCommit(repoPath)
return &DiscoveryResult{
RepoPath: tempDir,
Skills: skills,
Agents: agents,
Source: source,
CommitHash: commitHash,
}, nil
}
// discoverFromGitImpl is the non-progress variant used by the public facade.
func discoverFromGitImpl(source *Source) (*DiscoveryResult, error) {
return discoverFromGitWithProgressImpl(source, nil)
}
func discoverLocalImpl(source *Source) (*DiscoveryResult, error) {
if source == nil {
return nil, fmt.Errorf("source is required")
}
if source.Type != SourceTypeLocalPath {
return nil, fmt.Errorf("source is not a local path")
}
info, err := os.Stat(source.Path)
if err != nil {
return nil, fmt.Errorf("cannot access source path: %w", err)
}
if !info.IsDir() {
return nil, fmt.Errorf("source path is not a directory: %s", source.Path)
}
skills := discoverSkills(source.Path, true)
for i := range skills {
if skills[i].Path == "." {
skills[i].Name = source.Name
break
}
}
agents := discoverAgents(source.Path, len(skills) > 0)
skills, agents, err = constrainDiscoveryToExplicitSkill(source, skills, agents)
if err != nil {
return nil, err
}
return &DiscoveryResult{
RepoPath: source.Path,
Skills: skills,
Agents: agents,
Source: source,
}, nil
}
// resolveSubdir resolves a subdirectory path within a cloned repo.
// It first checks for an exact match. If not found, it scans the repo for
// SKILL.md files and looks for a skill whose name matches filepath.Base(subdir).
// Returns the resolved subdir path (may differ from input) or an error.
func resolveSubdir(repoPath, subdir string) (string, error) {
// 1. Exact match — fast path
exact := filepath.Join(repoPath, subdir)
info, err := os.Stat(exact)
if err == nil {
if !info.IsDir() {
return "", fmt.Errorf("'%s' is not a directory", subdir)
}
return subdir, nil
}
if !os.IsNotExist(err) {
return "", fmt.Errorf("cannot access subdirectory: %w", err)
}
// 2. Fuzzy match — scan for SKILL.md files whose directory basename matches
baseName := filepath.Base(subdir)
skills := discoverSkills(repoPath, false) // exclude root
var candidates []string
for _, sk := range skills {
if sk.Name == baseName {
candidates = append(candidates, sk.Path)
}
}
switch len(candidates) {
case 0:
return "", fmt.Errorf("subdirectory '%s' does not exist in repository", subdir)
case 1:
return candidates[0], nil
default:
return "", fmt.Errorf("subdirectory '%s' is ambiguous — multiple matches found:\n %s",
subdir, strings.Join(candidates, "\n "))
}
}
func constrainDiscoveryToExplicitSkill(source *Source, skills []SkillInfo, agents []AgentInfo) ([]SkillInfo, []AgentInfo, error) {
if source == nil || !source.TargetsExplicitSkill() {
return skills, agents, nil
}
for _, skill := range skills {
if skill.Path == "." {
return []SkillInfo{skill}, nil, nil
}
}
return nil, nil, fmt.Errorf("explicit skill target does not resolve to a root SKILL.md: %s", source.Raw)
}
// discoverSkills finds directories containing SKILL.md
// If includeRoot is true, root-level SKILL.md is also included (with Path=".")
func discoverSkills(repoPath string, includeRoot bool) []SkillInfo {
var skills []SkillInfo
filepath.Walk(repoPath, func(path string, info os.FileInfo, err error) error {
if err != nil {
return nil
}
// Skip .git and known target dotdirs (.claude, .cursor, .skillshare, etc.)
// to avoid counting target-synced copies as source skills. The walk root
// is exempt: it was requested explicitly (e.g. `install repo/.claude`),
// so filtering it would discover nothing at all.
if info.IsDir() {
name := info.Name()
if name == ".git" || (path != repoPath && TargetDotDirs[name]) {
return filepath.SkipDir
}
}
// Check if this is a SKILL.md file
if !info.IsDir() && info.Name() == "SKILL.md" {
skillDir := filepath.Dir(path)
relPath, _ := filepath.Rel(repoPath, skillDir)
fm := utils.ParseFrontmatterFields(path, []string{"description", "license"})
// Handle root level SKILL.md
if relPath == "." {
if includeRoot {
skills = append(skills, SkillInfo{
Name: filepath.Base(repoPath),
Path: ".",
License: fm["license"],
Description: fm["description"],
})
}
} else {
skills = append(skills, SkillInfo{
Name: filepath.Base(skillDir),
Path: strings.ReplaceAll(relPath, "\\", "/"),
License: fm["license"],
Description: fm["description"],
})
}
}
return nil
})
// Apply .skillignore filtering.
// Skill paths are directories (they contain SKILL.md), so pass isDir=true.
matcher := skillignore.ReadMatcher(repoPath)
if matcher.HasRules() {
filtered := skills[:0]
for _, s := range skills {
if !matcher.Match(s.Path, true) {
filtered = append(filtered, s)
}
}
skills = filtered
}
return skills
}
// discoverAgents finds .md files in an agents/ convention directory.
// Also detects "pure agent repos" — repos with no SKILL.md and no agents/ dir
// but with .md files at root (per D5 rule 4).
func discoverAgents(repoPath string, hasSkills bool) []AgentInfo {
var agents []AgentInfo
// Rule 2: Check agents/ convention directory
agentsDir := filepath.Join(repoPath, "agents")
if info, err := os.Stat(agentsDir); err == nil && info.IsDir() {
agents = append(agents, scanAgentDir(repoPath, agentsDir)...)
return agents
}
// Rule 4: Pure agent repo fallback — no skills, no agents/ dir, root has .md files
if !hasSkills {
agents = append(agents, scanAgentDir(repoPath, repoPath)...)
}
return agents
}
// scanAgentDir scans a directory for .md agent files, excluding conventional files.
func scanAgentDir(repoRoot, dir string) []AgentInfo {
var agents []AgentInfo
filepath.Walk(dir, func(path string, info os.FileInfo, err error) error {
if err != nil {
return nil
}
if info.IsDir() {
name := info.Name()
// Same walk-root exemption as discoverSkills: an explicitly
// requested dot-dir must still be scanned.
if name == ".git" || (path != dir && TargetDotDirs[name]) {
return filepath.SkipDir
}
return nil
}
if !strings.HasSuffix(strings.ToLower(info.Name()), ".md") {
return nil
}
// Skip conventional excludes
if conventionalAgentExcludes[info.Name()] {
return nil
}
// Skip hidden files
if strings.HasPrefix(info.Name(), ".") {
return nil
}
relPath, relErr := filepath.Rel(repoRoot, path)
if relErr != nil {
return nil
}
relPath = strings.ReplaceAll(relPath, "\\", "/")
name := strings.TrimSuffix(info.Name(), ".md")
agents = append(agents, AgentInfo{
Name: name,
Path: relPath,
FileName: info.Name(),
})
return nil
})
return agents
}
var conventionalAgentExcludes = map[string]bool{
"README.md": true,
"CHANGELOG.md": true,
"LICENSE.md": true,
"HISTORY.md": true,
"SECURITY.md": true,
"SKILL.md": true,
}
// DiscoverFromGitSubdir clones a repo and discovers skills within a subdirectory
// Unlike DiscoverFromGit, this includes root-level SKILL.md of the subdir
func discoverFromGitSubdirWithProgressImpl(source *Source, onProgress ProgressCallback) (*DiscoveryResult, error) {
if !isGitInstalled() {
return nil, fmt.Errorf("git is not installed or not in PATH")
}
if !source.HasSubdir() {
return nil, fmt.Errorf("source has no subdirectory specified")
}
if err := resolveWebRef(source); err != nil {
return nil, err
}
if !source.HasSubdir() {
// The whole path was the ref, e.g. tree/feature/x.
return discoverFromGitWithProgressImpl(source, onProgress)
}
// Prepare temporary repo directory
tempDir, err := os.MkdirTemp("", "skillshare-discover-*")
if err != nil {
return nil, fmt.Errorf("failed to create temp directory: %w", err)
}
repoPath := filepath.Join(tempDir, "repo")
var warnings []string
var commitHash string
var subdirPath string
// The Contents API paths below ignore any branch, tag or commit.
hasRef := source.ref() != ""
// Fast path 1: sparse checkout (preferred for speed if git is modern)
// Works for GitHub and non-GitHub hosts.
if gitSupportsSparseCheckout() {
if err := sparseCloneSubdir(source.CloneURL, source.Subdir, repoPath, source.ref(), source.authEnv(), onProgress); err == nil {
subdirPath = filepath.Join(repoPath, source.Subdir)
if info, statErr := os.Stat(subdirPath); statErr == nil && info.IsDir() {
source.recordCommit(repoPath)
if hash, hashErr := getGitCommit(repoPath); hashErr == nil {
commitHash = hash
}
skills := discoverSkills(subdirPath, true)
agents := discoverAgents(subdirPath, len(skills) > 0)
skills, agents, err = constrainDiscoveryToExplicitSkill(source, skills, agents)
if err != nil {
cleanupTempRepo(tempDir)
return nil, err
}
return &DiscoveryResult{
RepoPath: tempDir,
Skills: skills,
Agents: agents,
Source: source,
CommitHash: commitHash,
Warnings: warnings,
}, nil
}
warnings = append(warnings, "sparse checkout discovery fallback: subdirectory missing after checkout")
cleanupTempRepo(repoPath)
subdirPath = ""
} else {
warnings = append(warnings, fmt.Sprintf("sparse checkout discovery fallback: %v", err))
cleanupTempRepo(repoPath)
subdirPath = ""
}
}
// Fast path 2: GitHub/GHE Contents API
// Fallback for when sparse checkout is unavailable or fails.
if subdirPath == "" && !hasRef && isGitHubAPISource(source) {
owner, repo := source.GitHubOwner(), source.GitHubRepo()
subdirPath = filepath.Join(repoPath, source.Subdir)
hash, dlErr := downloadGitHubDir(owner, repo, source.Subdir, subdirPath, source, onProgress)
if dlErr == nil {
commitHash = hash
skills := discoverSkills(subdirPath, true)
agents := discoverAgents(subdirPath, len(skills) > 0)
skills, agents, err = constrainDiscoveryToExplicitSkill(source, skills, agents)
if err != nil {
cleanupTempRepo(tempDir)
return nil, err
}
return &DiscoveryResult{
RepoPath: tempDir,
Skills: skills,
Agents: agents,
Source: source,
CommitHash: commitHash,
}, nil
}
warnings = append(warnings, fmt.Sprintf("GitHub API discovery fallback: %v", dlErr))
cleanupTempRepo(repoPath)
subdirPath = ""
}
// Fast path 2b: CNB Contents API
if subdirPath == "" && !hasRef && isCNBAPISource(source) {
repo := cnbRepoPath(source.CloneURL)
subdirPath = filepath.Join(repoPath, source.Subdir)
hash, dlErr := downloadCNBDir(repo, source.Subdir, subdirPath, source, onProgress)
if dlErr == nil {
commitHash = hash
skills := discoverSkills(subdirPath, true)
agents := discoverAgents(subdirPath, len(skills) > 0)
skills, agents, err = constrainDiscoveryToExplicitSkill(source, skills, agents)
if err != nil {
_ = os.RemoveAll(tempDir)
return nil, err
}
return &DiscoveryResult{
RepoPath: tempDir,
Skills: skills,
Agents: agents,
Source: source,
CommitHash: commitHash,
Warnings: warnings,
}, nil
}
warnings = append(warnings, fmt.Sprintf("CNB API discovery fallback: %v", dlErr))
_ = os.RemoveAll(repoPath)
subdirPath = ""
}
// Fast path 2b: Gitea Contents API
if subdirPath == "" && !hasRef && isGiteaAPISource(source) {
owner, repo := giteaOwnerRepo(source.CloneURL)
subdirPath = filepath.Join(repoPath, source.Subdir)
hash, dlErr := downloadGiteaDir(owner, repo, source.Subdir, subdirPath, source, onProgress)
if dlErr == nil {
commitHash = hash
skills := discoverSkills(subdirPath, true)
agents := discoverAgents(subdirPath, len(skills) > 0)
skills, agents, err = constrainDiscoveryToExplicitSkill(source, skills, agents)
if err != nil {
_ = os.RemoveAll(tempDir)
return nil, err
}
return &DiscoveryResult{
RepoPath: tempDir,
Skills: skills,
Agents: agents,
Source: source,
CommitHash: commitHash,
Warnings: warnings,
}, nil
}
warnings = append(warnings, fmt.Sprintf("Gitea API discovery fallback: %v", dlErr))
_ = os.RemoveAll(repoPath)
subdirPath = ""
}
// Fallback: full clone + fuzzy subdir resolution
cleanupTempRepo(repoPath)
if onProgress != nil {
onProgress("Cloning repository...")
}
if err := cloneRepoForSource(source, repoPath, source.Branch, true, onProgress); err != nil {
cleanupTempRepo(tempDir)
return nil, fmt.Errorf("failed to clone repository: %w", err)
}
resolved, err := resolveSubdir(repoPath, source.Subdir)
if err != nil {
cleanupTempRepo(tempDir)
return nil, err
}
if resolved != source.Subdir {
source.Subdir = resolved
source.Name = filepath.Base(resolved)
}
subdirPath = filepath.Join(repoPath, resolved)
if hash, hashErr := getGitCommit(repoPath); hashErr == nil {
commitHash = hash
}
skills := discoverSkills(subdirPath, true)
agents := discoverAgents(subdirPath, len(skills) > 0)
skills, agents, err = constrainDiscoveryToExplicitSkill(source, skills, agents)
if err != nil {
cleanupTempRepo(tempDir)
return nil, err
}
return &DiscoveryResult{
RepoPath: tempDir,
Skills: skills,
Agents: agents,
Source: source,
CommitHash: commitHash,
Warnings: warnings,
}, nil
}
// discoverFromGitSubdirImpl is the non-progress variant used by the public facade.
func discoverFromGitSubdirImpl(source *Source) (*DiscoveryResult, error) {
return discoverFromGitSubdirWithProgressImpl(source, nil)
}
// CleanupDiscovery removes the temporary directory created for git-based
// discovery (clone under RepoPath). It only runs when Source is known to be
// a git remote; local paths, missing source metadata, and other types are
// skipped so RepoPath is never treated as disposable unless it came from a
// git clone.
func cleanupDiscoveryImpl(result *DiscoveryResult) {
if result == nil || result.RepoPath == "" || result.Source == nil {
return
}
if !result.Source.IsGit() {
return
}
cleanupTempRepo(result.RepoPath)
}
// InstallFromDiscovery installs a skill from a discovered repository