Files
skillshare/internal/install/install_tracked.go
T
a352b8d243 feat(install): pin skills to a tag or commit from their web URL (#293)
* feat(install): pin GitHub tree/blob URLs to their ref

Hub entries only carry a source URL, so a team hub had no way to pin a
third-party skill to a reviewed tag or commit. The parser already read
tree/<ref>/ and blob/<ref>/ URLs but discarded the ref, installing from
the default branch instead of the version the URL names.

Keep that ref as Source.Branch. An explicit --branch (CLI) or branch
(API body) still overrides it; the server handlers now only override
when a branch is given. Batch installs group by repo and ref so two
pinned versions of one repo get separate clones.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs(install): describe pinning hub sources to a tag or commit

Explain that a GitHub tree/<ref>/ source URL pins the install, in both
the hub index guide and the install reference.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat(install): pin GitLab/Bitbucket URLs and resolve refs containing "/"

Pinning should not depend on the host, so GitLab (-/tree/<ref>/) and
Bitbucket (src/<ref>/) web URLs now keep their ref too.

Taking the first path segment as the ref broke branch names containing
"/": tree/feature/x/skills/foo cloned branch "feature" and failed. When
the first segment is not a branch or tag on the remote, the longest
matching one is used instead. A ref the remote does not have fails
with a hint to use --branch, rather than silently installing the
default branch. A saved branch that covers more of the path (from
--branch or config) moves the subdir without asking the remote.

The grouped-install round-trip test pinned -/tree/main/ against a repo
on master; the ref used to be ignored, so its branch is now renamed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs(install): cover GitLab, Bitbucket and "/" branches in URL pinning

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(install): keep web URL refs correct on reinstall, --branch and grouping

Review of the slash-branch resolution found four gaps:

- A saved branch equal to the whole URL tail (tree/feature/x) did not
  move the subdir back, so every reinstall or update looked for "x".
- An unrelated --branch with a slash-branch URL kept the parse-time
  subdir (x/skills/foo) instead of the URL's real one.
- Grouped installs copy the source to a repo root; resolution then put
  the subdir back and discovery looked under a doubled path. Grouping
  also keyed on the unresolved first segment, so tree/feature/x and
  tree/feature/y could share one clone. Web URLs with a ref now install
  on their own, and resolution leaves a repo-root copy at its root.
- A skill named after the parse-time subdir kept that name once the
  path turned out to be all ref (tree/feature/x named the skill "x").

The unknown-ref error now reads as a missing ref first, since a stale
ref (tree/master after a rename) is more common than a slash branch.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(update): keep legacy installs on the default branch

Skills installed before URL refs were honoured got the default branch
and recorded no branch. Single-skill update re-parses the recorded
source, so it moved them to the URL's ref (or failed on a stale one,
with a hint to pass --branch, which update does not accept), while
update --all rebuilds from RepoURL and kept the default branch.

ApplyRecordedBranch treats an empty recorded branch as the remote
default and drops the URL's ref. It replaces the branch handling at
every site that reinstalls from recorded metadata: CLI and server
update for skills and agents, and tracked-repo rehydration.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* perf(search): share one clone across pinned hub skills from a repo

Any web URL with a path after its ref installed on its own, so a hub
listing many tree/<tag>/ skills from one repo cloned it once per skill.

groupByRepo now resolves each URL's ref before grouping, with a
RemoteRefs cache that lists each remote's refs once, and groups by
repo and resolved ref. Only refs containing "/" (whose subdir the
discovery match cannot re-derive) and refs that fail to resolve still
install alone. A resolved source is marked settled so the grouped
clone does not resolve it again, and ApplyRecordedBranch settles the
subdir from a recorded branch without asking the remote.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Willie <egg358358@gmail.com>
2026-09-28 22:08:16 +08:00

389 lines
12 KiB
Go

package install
import (
"fmt"
"os"
"path/filepath"
"strings"
)
func installTrackedRepoImpl(source *Source, sourceDir string, opts InstallOptions) (*TrackedRepoResult, error) {
if !source.IsGit() {
return nil, fmt.Errorf("--track requires a git repository source")
}
if err := resolveWebRef(source); err != nil {
return nil, err
}
// Determine repo name: opts.Name > source.Name (from config) > TrackName (derived from URL)
repoName := opts.Name
if repoName == "" {
repoName = source.Name
}
if repoName == "" {
repoName = source.TrackName()
}
// Prefix with _ to indicate tracked repo (avoid double prefix if user already added _)
trackedName := repoName
if !strings.HasPrefix(repoName, "_") {
trackedName = "_" + repoName
}
if err := validateTrackedRepoDirName(trackedName); err != nil {
return nil, fmt.Errorf("invalid tracked repo name %q: %w", trackedName, err)
}
destBase := sourceDir
if opts.Into != "" {
destBase = filepath.Join(sourceDir, opts.Into)
}
if err := os.MkdirAll(destBase, 0755); err != nil {
if opts.Into != "" {
return nil, fmt.Errorf("failed to create --into directory: %w", err)
}
return nil, fmt.Errorf("failed to create source directory: %w", err)
}
destPath := filepath.Join(destBase, trackedName)
result := &TrackedRepoResult{
RepoName: trackedName,
RepoPath: destPath,
}
// Check if already exists
if _, err := os.Stat(destPath); err == nil {
if opts.Update {
return updateTrackedRepo(destPath, result, opts)
}
if !opts.Force {
hint := buildForceHint(source.Raw, opts.Into) // includes --into if set
hint = strings.Replace(hint, " --force", " --track --force", 1)
// Tracked repos store a git remote; read it for comparison.
existingURL := getRemoteURL(destPath)
if existingURL != "" && repoURLsMatch(existingURL, source.CloneURL) {
return nil, fmt.Errorf("%w: tracked repo '%s'. Use 'skillshare update' or --force to overwrite", ErrSkipSameRepo, trackedName)
}
if existingURL != "" {
return nil, fmt.Errorf("tracked repo '%s' already exists (installed from %s). To overwrite: %s", trackedName, existingURL, hint)
}
return nil, fmt.Errorf("tracked repo '%s' already exists. To overwrite: %s", trackedName, hint)
}
// Force mode - remove existing
if !opts.DryRun {
if err := os.RemoveAll(destPath); err != nil {
return nil, fmt.Errorf("failed to remove existing repo: %w", err)
}
}
}
if opts.DryRun {
result.Action = "would clone"
return result, nil
}
// Clone tracked repos with a download-optimized strategy first, then
// fallback to the legacy full clone for compatibility.
// Prefer CLI --branch over source.Branch (from config); both beat empty.
cloneBranch := opts.Branch
if cloneBranch == "" {
cloneBranch = source.Branch
}
// A tag or commit SHA has no branch for `git pull` to follow. A SHA fails
// at clone (--branch rejects it); a tag clones but leaves HEAD detached.
if err := cloneTrackedRepoForSource(source, destPath, cloneBranch, opts.OnProgress); err != nil {
if IsCommitSHA(cloneBranch) {
return nil, errTrackedNeedsBranch(cloneBranch)
}
return nil, fmt.Errorf("failed to clone repository: %w", err)
}
if isDetachedHead(destPath) {
_ = os.RemoveAll(destPath)
return nil, errTrackedNeedsBranch(cloneBranch)
}
if source.Commit != "" {
if err := resetTrackedToCommit(destPath, source.Commit, source.authEnv()); err != nil {
_ = os.RemoveAll(destPath)
return nil, err
}
}
// Discover skills in the cloned repo. Include root SKILL.md so the count
// matches what `skillshare sync` will see: every SKILL.md inside a tracked
// repo (root and nested alike) becomes an independent skill on sync.
skills := discoverSkills(destPath, true)
result.SkillCount = len(skills)
for _, skill := range skills {
result.Skills = append(result.Skills, skill.Name)
}
// Also discover agents in the tracked repo
agents := discoverAgents(destPath, len(skills) > 0)
result.AgentCount = len(agents)
if len(agents) > 0 {
for _, agent := range agents {
result.Agents = append(result.Agents, agent.Name)
}
result.Warnings = append(result.Warnings, fmt.Sprintf("%d agent(s) found in tracked repo", len(agents)))
}
if len(skills) == 0 && len(agents) == 0 {
result.Warnings = append(result.Warnings, "no SKILL.md files or agents found in repository")
} else if len(skills) == 0 {
// Only agents found — not a warning, just informational
}
// Security audit on the entire tracked repo
if err := auditTrackedRepo(destPath, result, opts); err != nil {
return nil, err
}
// Auto-add to .gitignore to prevent committing tracked repo contents
gitignoreEntry := trackedName
if opts.Into != "" {
gitignoreEntry = filepath.Join(opts.Into, trackedName)
}
if err := UpdateGitIgnore(sourceDir, gitignoreEntry); err != nil {
result.Warnings = append(result.Warnings, fmt.Sprintf("failed to update .gitignore: %v", err))
}
result.Action = "cloned"
return result, nil
}
func errTrackedNeedsBranch(ref string) error {
return fmt.Errorf("tracked repos must follow a branch; %q is a tag or commit — install without --track to pin it", ref)
}
// updateTrackedRepo performs git pull on an existing tracked repo
func updateTrackedRepo(repoPath string, result *TrackedRepoResult, opts InstallOptions) (*TrackedRepoResult, error) {
if !IsGitRepo(repoPath) {
return nil, fmt.Errorf("'%s' is not a git repository", repoPath)
}
if opts.DryRun {
result.Action = "would update (git pull)"
return result, nil
}
// Record hash before pull for rollback on audit failure
beforeHash, err := getGitFullHash(repoPath)
if err != nil {
return nil, fmt.Errorf("failed to determine rollback commit before update (aborting for safety): %w", err)
}
if beforeHash == "" {
return nil, fmt.Errorf("failed to determine rollback commit before update (aborting for safety): empty commit hash")
}
if err := gitPull(repoPath, opts.OnProgress); err != nil {
return nil, fmt.Errorf("failed to update: %w", err)
}
// Post-pull audit: rollback via git reset (not os.RemoveAll) to preserve repo.
if err := auditTrackedRepoUpdate(repoPath, beforeHash, result, opts); err != nil {
return nil, err
}
// Re-discover skills (include root so count matches `sync` view).
skills := discoverSkills(repoPath, true)
result.SkillCount = len(skills)
for _, skill := range skills {
result.Skills = append(result.Skills, skill.Name)
}
// Also discover agents in the tracked repo
agents := discoverAgents(repoPath, len(skills) > 0)
result.AgentCount = len(agents)
if len(agents) > 0 {
for _, agent := range agents {
result.Agents = append(result.Agents, agent.Name)
}
result.Warnings = append(result.Warnings, fmt.Sprintf("%d agent(s) found in tracked repo", len(agents)))
}
result.Action = "updated"
return result, nil
}
// cloneRepoFull performs a full git clone (quiet mode for cleaner output)
func cloneRepoFull(url, destPath, branch string, onProgress ProgressCallback) error {
args := []string{"clone"}
if onProgress != nil {
args = append(args, "--progress")
} else {
args = append(args, "--quiet")
}
if branch != "" {
args = append(args, "--branch", branch)
}
args = append(args, url, destPath)
return runGitCommandWithProgress(args, "", authEnv(url), onProgress)
}
func cloneTrackedRepoForSource(source *Source, destPath, branch string, onProgress ProgressCallback) error {
err := cloneTrackedRepoForParsedSource(source, destPath, branch, onProgress)
if err == nil {
return nil
}
if !shouldRetryNestedGitLabURL(err) {
return err
}
var lastErr error
for _, fallback := range cloneFallbackSourcesForNestedGitLabURL(source) {
if cleanupErr := removeAll(destPath); cleanupErr != nil && !os.IsNotExist(cleanupErr) {
return fmt.Errorf("clone failed (%v), and cleanup before fallback failed: %w", err, cleanupErr)
}
if onProgress != nil {
onProgress("Clone failed; retrying as a nested GitLab repository...")
}
fallbackErr := cloneTrackedRepoForParsedSource(&fallback, destPath, branch, onProgress)
if fallbackErr == nil {
*source = fallback
return nil
}
lastErr = fallbackErr
}
if lastErr != nil {
return fmt.Errorf("%w (nested GitLab fallback also failed: %v)", err, lastErr)
}
return err
}
// cloneTrackedRepo clones a tracked repository with an optimized payload first
// and falls back to full clone when the remote does not support partial/shallow
// capabilities.
//
// When subdir is provided, sparse checkout is attempted first to reduce payload
// while preserving .git for future tracked updates.
func cloneTrackedRepo(url, subdir, destPath, branch string, onProgress ProgressCallback) error {
return cloneTrackedRepoWithEnv(url, subdir, destPath, branch, authEnv(url), onProgress)
}
func cloneTrackedRepoForParsedSource(source *Source, destPath, branch string, onProgress ProgressCallback) error {
if source == nil {
return fmt.Errorf("nil source")
}
return cloneTrackedRepoWithEnv(source.CloneURL, source.Subdir, destPath, branch, source.authEnv(), onProgress)
}
func cloneTrackedRepoWithEnv(url, subdir, destPath, branch string, extraEnv []string, onProgress ProgressCallback) error {
subdir = strings.TrimSpace(subdir)
if subdir != "" && gitSupportsSparseCheckout() {
if onProgress != nil {
onProgress("Preparing sparse checkout...")
}
if err := sparseCloneSubdir(url, subdir, destPath, branch, extraEnv, onProgress); err == nil {
checkedSubdir := strings.TrimLeft(subdir, "/")
if checkedSubdir == "" {
return nil
}
if info, statErr := os.Stat(filepath.Join(destPath, checkedSubdir)); statErr == nil && info.IsDir() {
return nil
}
if cleanupErr := removeAll(destPath); cleanupErr != nil {
return fmt.Errorf("sparse checkout produced no subdirectory %q, and cleanup failed: %w", subdir, cleanupErr)
}
if onProgress != nil {
onProgress("Sparse checkout path missing; retrying standard clone...")
}
} else if shouldFallbackSparseTrackedClone(err) {
// sparseCloneSubdir may have already created destPath. Clean it before
// falling back to a standard clone strategy.
if cleanupErr := removeAll(destPath); cleanupErr != nil {
return fmt.Errorf("sparse checkout failed (%v), and cleanup failed: %w", err, cleanupErr)
}
if onProgress != nil {
onProgress("Sparse checkout unavailable; retrying standard clone...")
}
} else {
return err
}
}
args := []string{
"clone",
"--filter=blob:none",
"--depth", "1",
"--single-branch",
}
if branch != "" {
args = append(args, "--branch", branch)
}
if onProgress != nil {
args = append(args, "--progress")
} else {
args = append(args, "--quiet")
}
args = append(args, url, destPath)
err := runGitCommandWithProgress(args, "", extraEnv, onProgress)
if err == nil {
return nil
}
if !shouldFallbackTrackedClone(err) {
return err
}
if onProgress != nil {
onProgress("Remote lacks partial clone support; retrying standard clone...")
}
return cloneRepoWithEnv(url, destPath, branch, false, extraEnv, onProgress)
}
// isAuthOrAccessError returns true for auth failures and access denials that
// should NOT trigger a fallback clone strategy (retrying won't help).
func isAuthOrAccessError(err error) bool {
if err == nil {
return false
}
s := err.Error()
if IsAuthError(s) {
return true
}
low := strings.ToLower(s)
return strings.Contains(low, "permission denied") ||
strings.Contains(low, "repository not found")
}
func shouldFallbackSparseTrackedClone(err error) bool {
if err == nil {
return false
}
if isAuthOrAccessError(err) {
return false
}
// For compatibility, fallback to the legacy tracked clone flow for
// capability, sparse-path, and server-specific sparse checkout errors.
return true
}
func shouldFallbackTrackedClone(err error) bool {
if err == nil {
return false
}
if isAuthOrAccessError(err) {
return false
}
low := strings.ToLower(err.Error())
capabilityHints := []string{
"does not support",
"not support",
"filter",
"shallow",
"depth",
"single-branch",
"partial clone",
"dumb http",
}
for _, hint := range capabilityHints {
if strings.Contains(low, hint) {
return true
}
}
return false
}
// GetUpdatableSkills returns skill names that have metadata with a remote source.
// It walks subdirectories recursively so nested skills are found.