Files
skillshare/internal/install/install_git.go
T
Willie 2ce46834e2 fix(collect,install): collect only real skills; let --track take a local git path (#508)
* fix(collect,install): collect only real skills; let --track take a local git path

collect treated every directory in a target as a local skill, so a scratch
folder was listed and `collect --force` copied it into the source. A folder
now counts only when it holds SKILL.md; the dashboard scan shares
FindLocalSkills and the dashboard collect request rejects such a folder.

`install /path/to/repo --track` failed with "requires a git repository
source" while file:///path worked. A local path that is a git repository
root is now normalized to the same file:// clone URL for --track (also past
the --branch guard); other local folders get an error that names file://.
Help text, docs (English and i18n) and the built-in skill reference follow.

* fix(install): keep the drive letter in file:// clone URLs for local --track paths

A Windows path such as C:/repo produced file://C:/repo, which makes C: the
URI authority instead of part of the path, so a tracked install from a local
Windows repository could clone the wrong location. Prefix the missing slash
so the URL is file:///C:/repo; POSIX paths are unchanged. Also cover the
dashboard install endpoint with a local-path --track test.

* fix(collect,install): keep UNC file URLs intact; require SKILL.md to be a file

The drive-letter fix lived in the helper shared with explicit file:// URLs,
which rewrote file://server/share/repo into a local path. Keep the slash
logic in the local-path --track case only and leave explicit URLs as parsed.

A subdirectory named SKILL.md passed the existence check, so a scratch folder
holding one was still collected. Both collect entry points (scan and the
dashboard request) now share sync.HasSkillFile, which rejects directories.

* fix(install): escape the local path in the file:// URL used by --track

Git decodes a file:// URL, so a repository path containing %20, %2F, a space
or # was cloned from a different path than the one IsGitRepo accepted. Build
the URL with net/url so those characters are escaped; drive-letter and POSIX
paths keep their form. The integration test now installs from a directory
whose name contains a space and a literal %20.

* fix(install): accept a local bare repository path for --track

A bare repository has HEAD and objects at its root but no .git entry, so the
local-path check rejected it while file:///same/path cloned fine. Add
IsLocalGitRepo (working tree or bare) and use it for both the --track
normalization and the --branch guard.

* test(install): build the bare repo path from a Windows file URL correctly

makeRemote returns file:///C:/dir on Windows; trimming only file:// left
/C:/dir, which filepath.Abs turned into D:\C:\dir, so the bare-repo test
failed on the Windows runner. Drop the slash before the drive letter.

* fix(install): refuse a --track local source that is the install destination

--force removes the tracked destination before cloning. When the local path
given to --track is that same directory, the removal deleted the only clone
source (and any unpushed work) before the clone failed. Compare the two with
os.SameFile, which also covers symlinked paths, and stop before touching the
destination.

* fix(install,collect): keep a tracked source inside its destination; reject special SKILL.md files

--track --force removed the destination before cloning, so a local source
nested below it (or given as an explicit file:// URL) was deleted along with
any unpushed work, and the clone then failed. Resolve the source and refuse
when it is the destination or lies under it, for dry runs too.

HasSkillFile used a not-a-directory test, so a FIFO named SKILL.md counted as
a skill and the collect copy blocked on it. Require a regular file; a
symlinked SKILL.md still resolves through os.Stat.

* fix(install): treat file://localhost URLs as local clone sources

Git clones file://localhost/path from the local filesystem, but the
containment guard only recognised an empty authority, so --track --force
could still delete a source given that way. Accept localhost (any case) as
local.

* test(install): build the localhost file URL from a slash path

On Windows the temp path starts with a drive letter, so prefixing it onto
file://localhost produced file://localhostC:\... instead of a valid URL and
the case failed on the Windows runner. Convert to slashes and add the
separator explicitly.

* fix(install): clone --track --force into a staging dir and swap it in last

--force removed the existing tracked repo before cloning, so any source that
lived under it (plain path, file:///, file://localhost, UNC, symlink or 8.3
alias) was destroyed before git could read it, and a failed clone also lost
the old repo. Clone beside the destination, run the checks and the audit on
the staging copy, then move the old repo aside, rename the new one in and
drop the backup; on any failure the staging copy is removed and the old repo
stays. The pre-removal containment refusal now covers plain local paths only
(the url.Parse special-casing is gone).

* fix(install): refuse a file:// --track source inside the forced destination

The staged swap in 7375400 still lost data for a file:///, file://localhost
or UNC source nested under the destination: the clone succeeded, the old repo
(with the source inside it) was renamed to the backup, and removing the backup
deleted the source's uncommitted work, which a clone cannot carry.

Restore the file URL to local path mapping (now including UNC hosts) so the
containment refusal covers every local spelling, not only plain paths. The
test now keeps an uncommitted file in the source and requires it to survive.

* fix(install): apply the source-inside-destination refusal only to --force

The refusal ran before the --update branch, so `install <tracked repo path>
--track --update` was rejected even though an update pulls the existing
checkout in place and removes nothing. Only the forced replacement can
destroy a source that lives under the destination, so check there.

* fix(install): report a forced clone's root skill under its tracked name

With --force the repo is discovered in the .skillshare-clone-<stamp> staging
directory, and discoverSkills names a root SKILL.md after the directory, so
the CLI summary, dashboard response and oplog listed a random staging name
instead of _foo. Use the tracked name for the root entry.

* fix(install): stage fresh tracked clones so cleanup never hits another installer

A fresh tracked install cloned straight into the destination, and the
deferred cleanup removed it on any failure. If another process created the
same tracked repo between the existence check and the clone, the failing
install deleted that process's checkout. Clone every install into its own
staging directory and rename it in last; a claimed destination makes the
rename fail and only the staging copy is removed.
2026-10-10 01:06:23 +08:00

541 lines
17 KiB
Go

package install
import (
"context"
"errors"
"fmt"
"net/url"
"os"
"os/exec"
"path/filepath"
"regexp"
"strings"
"time"
"skillshare/internal/utils"
)
var commitSHAPattern = regexp.MustCompile(`^[0-9a-f]{7,40}$`)
// IsCommitSHA reports whether ref looks like an abbreviated or full commit hash
// rather than a branch or tag name.
func IsCommitSHA(ref string) bool {
return commitSHAPattern.MatchString(ref)
}
func isGitInstalled() bool {
_, err := exec.LookPath("git")
return err == nil
}
// IsGitRepo recognizes .git directories and gitdir files used by worktrees/submodules.
func IsGitRepo(path string) bool {
gitDir := filepath.Join(path, ".git")
info, err := os.Stat(gitDir)
return err == nil && (info.IsDir() || info.Mode().IsRegular())
}
// IsLocalGitRepo reports whether path is the root of a git repository, either a
// working tree (.git) or a bare repository (HEAD and objects at the root).
func IsLocalGitRepo(path string) bool {
if IsGitRepo(path) {
return true
}
head, headErr := os.Stat(filepath.Join(path, "HEAD"))
objects, objErr := os.Stat(filepath.Join(path, "objects"))
return headErr == nil && head.Mode().IsRegular() && objErr == nil && objects.IsDir()
}
// IsTrackedCheckout reports whether path is a tracked repo checkout: a
// _-prefixed directory that is a git repo.
func IsTrackedCheckout(path string) bool {
return utils.IsTrackedRepoDir(filepath.Base(path)) && IsGitRepo(path)
}
// gitCommandTimeout is the maximum time for a git network operation.
// Remote tracked-repo clones can take longer in constrained CI/Docker networks.
const gitCommandTimeout = 180 * time.Second
// gitCommand creates an exec.Cmd for git with GIT_TERMINAL_PROMPT=0
// to prevent interactive credential prompts that hang CLI spinners and web UI.
func gitCommand(ctx context.Context, args ...string) *exec.Cmd {
cmd := exec.CommandContext(ctx, "git", args...)
cmd.Env = append(os.Environ(),
"GIT_TERMINAL_PROMPT=0",
"GIT_ASKPASS=",
"SSH_ASKPASS=",
)
return cmd
}
// runGitCommand runs a git command with timeout, captures stderr for error messages.
func runGitCommand(args []string, dir string) error {
return runGitCommandEnv(args, dir, nil)
}
// runGitCommandEnv is like runGitCommand but accepts extra environment variables.
func runGitCommandEnv(args []string, dir string, extraEnv []string) error {
return runGitCommandWithProgress(args, dir, extraEnv, nil)
}
// UsedTokenAuth reports whether extraEnv contains token-based auth overrides.
func UsedTokenAuth(extraEnv []string) bool {
for _, env := range extraEnv {
if strings.HasPrefix(env, "GIT_CONFIG_KEY_") && strings.Contains(env, ".insteadOf") {
return true
}
}
return false
}
// WrapGitError inspects stderr output to produce actionable error messages.
func WrapGitError(stderr string, err error, tokenAuthAttempted bool) error {
s := sanitizeTokens(strings.TrimSpace(stderr))
if IsSSLError(s) {
return fmt.Errorf("SSL certificate verification failed — options:\n"+
" 1. Custom CA bundle: GIT_SSL_CAINFO=/path/to/ca-bundle.crt skillshare install <url>\n"+
" 2. Skip verification: GIT_SSL_NO_VERIFY=true skillshare install <url>\n"+
" 3. Use SSH instead: git@<host>:<owner>/<repo>.git\n %s", s)
}
if IsAuthError(s) {
if tokenAuthAttempted {
return fmt.Errorf("authentication failed — token rejected, check permissions and expiry\n %s", s)
}
return fmt.Errorf("authentication required — options:\n"+
" 1. SSH URL: git@<host>:<owner>/<repo>.git\n"+
" 2. Token env var: GITHUB_TOKEN, GITLAB_TOKEN, BITBUCKET_TOKEN, AZURE_DEVOPS_TOKEN, CNB_TOKEN, GITEA_TOKEN, or SKILLSHARE_GIT_TOKEN\n"+
" 3. Git credential helper: gh auth login\n %s", s)
}
if s != "" {
// Extract the fatal/error line from verbose git output, strip hint lines
msg := extractGitFatal(s)
if login := sshPublickeyLogin(msg); login != "" {
msg += "\n check your SSH key: ssh -T " + login
}
return fmt.Errorf("%s", msg)
}
// When stderr is empty, extract exit code for a more actionable message
var exitErr *exec.ExitError
if errors.As(err, &exitErr) {
code := exitErr.ExitCode()
switch code {
case 128:
if tokenAuthAttempted {
return fmt.Errorf("git failed (exit 128): authentication token was rejected — check permissions and expiry")
}
return fmt.Errorf("git failed (exit 128): repository not found or authentication required")
case 1:
return fmt.Errorf("git failed (exit 1): command error — check network connectivity and repository URL")
default:
return fmt.Errorf("git failed (exit %d)", code)
}
}
return err
}
// gitRemoteReadFatal is git's generic message when the transport (usually ssh)
// fails; the reason is on the line ssh printed before it.
const gitRemoteReadFatal = "Could not read from remote repository."
// sshPublickeyLogin returns "user@host" when msg carries an ssh
// "user@host: Permission denied (publickey)." reason, otherwise "".
func sshPublickeyLogin(msg string) string {
i := strings.Index(msg, ": Permission denied (publickey")
if i < 0 {
return ""
}
login := msg[:i]
if j := strings.LastIndex(login, " "); j >= 0 {
login = login[j+1:]
}
if !strings.Contains(login, "@") {
return ""
}
return login
}
// extractGitFatal extracts the "fatal:" or "error:" line from git stderr,
// stripping verbose "hint:" lines. If no fatal/error line is found, returns
// the original text with hint lines removed. For the generic remote-read
// failure it keeps the preceding transport line that explains it.
func extractGitFatal(stderr string) string {
var fatal, reason string
var nonHint []string
for _, line := range strings.Split(stderr, "\n") {
trimmed := strings.TrimSpace(line)
if trimmed == "" {
continue
}
if strings.HasPrefix(trimmed, "hint:") {
continue
}
if strings.HasPrefix(trimmed, "fatal: ") {
fatal = strings.TrimPrefix(trimmed, "fatal: ")
if fatal == gitRemoteReadFatal && reason == "" {
reason = lastTransportLine(nonHint)
}
} else if strings.HasPrefix(trimmed, "error: ") {
fatal = strings.TrimPrefix(trimmed, "error: ")
}
nonHint = append(nonHint, trimmed)
}
if fatal != "" {
if strings.Contains(fatal, "failed to push some refs") && len(nonHint) > 1 {
return strings.Join(nonHint, "; ")
}
if fatal == gitRemoteReadFatal && reason != "" {
return strings.TrimSuffix(fatal, ".") + " — " + reason
}
return fatal
}
if len(nonHint) > 0 {
return strings.Join(nonHint, "; ")
}
return stderr
}
// lastTransportLine returns the last line before a remote-read failure that
// explains it, skipping git's own progress and warning chatter.
func lastTransportLine(lines []string) string {
for i := len(lines) - 1; i >= 0; i-- {
l := lines[i]
if strings.HasPrefix(l, "Cloning into") || strings.HasPrefix(l, "warning:") ||
strings.HasPrefix(l, "fatal:") || strings.HasPrefix(l, "error:") {
continue
}
return l
}
return ""
}
// cloneRepo performs a git clone (quiet mode for cleaner output).
// If a token is available in env vars, it injects authentication via
// GIT_CONFIG env vars without modifying the stored remote URL.
// branch specifies the branch to clone; empty string uses the remote default.
func cloneRepo(url, destPath, branch string, shallow bool, onProgress ProgressCallback) error {
return cloneRepoWithEnv(url, destPath, branch, shallow, authEnv(url), onProgress)
}
func cloneRepoWithEnv(url, destPath, branch string, shallow bool, extraEnv []string, onProgress ProgressCallback) error {
// git clone --branch accepts branches and tags but not commit SHAs:
// clone the default branch, then detach HEAD at the pinned commit.
if IsCommitSHA(branch) {
if err := cloneRepoWithEnv(url, destPath, "", shallow, extraEnv, onProgress); err != nil {
return err
}
return checkoutCommit(destPath, branch, shallow, extraEnv, onProgress)
}
args := []string{"clone"}
if onProgress != nil {
args = append(args, "--progress")
} else {
args = append(args, "--quiet")
}
if shallow {
args = append(args, "--depth", "1")
}
if branch != "" {
args = append(args, "--branch", branch)
}
args = append(args, url, destPath)
return runGitCommandWithProgress(args, "", extraEnv, onProgress)
}
// checkoutCommit detaches HEAD at sha. A targeted fetch works for full SHAs on
// GitHub/GitLab; when the server refuses or the SHA is abbreviated, it deepens
// the clone and resolves the commit locally.
func checkoutCommit(repoPath, sha string, shallow bool, extraEnv []string, onProgress ProgressCallback) error {
fetch := []string{"fetch", "--quiet", "origin", sha}
if shallow {
fetch = []string{"fetch", "--quiet", "--depth", "1", "origin", sha}
}
if err := runGitCommandWithProgress(fetch, repoPath, extraEnv, onProgress); err == nil {
return runGitCommandWithProgress([]string{"checkout", "--quiet", "--detach", "FETCH_HEAD"}, repoPath, extraEnv, nil)
}
// clone --depth 1 implies --single-branch, so fetch every branch and tag
// explicitly: the pinned commit may live off the default branch.
full := []string{"fetch", "--quiet", "--tags", "origin", "+refs/heads/*:refs/remotes/origin/*"}
if isShallowRepo(repoPath) {
full = append(full[:2], append([]string{"--unshallow"}, full[2:]...)...)
}
if err := runGitCommandWithProgress(full, repoPath, extraEnv, onProgress); err != nil {
return fmt.Errorf("resolve commit %s: %w", sha, err)
}
return runGitCommandWithProgress([]string{"checkout", "--quiet", "--detach", sha}, repoPath, extraEnv, nil)
}
func isShallowRepo(repoPath string) bool {
cmd := exec.Command("git", "rev-parse", "--is-shallow-repository")
cmd.Dir = repoPath
out, err := cmd.Output()
return err == nil && strings.TrimSpace(string(out)) == "true"
}
// isDetachedHead reports whether HEAD points at a commit rather than a branch.
func isDetachedHead(repoPath string) bool {
cmd := exec.Command("git", "symbolic-ref", "-q", "HEAD")
cmd.Dir = repoPath
return cmd.Run() != nil
}
// ref is what to clone: the locked commit when there is one, else the branch.
func (s *Source) ref() string {
if s.Commit != "" {
return s.Commit
}
return s.Branch
}
// recordCommit notes the full SHA a clone landed on, for the lockfile.
func (s *Source) recordCommit(repoPath string) {
if hash, err := getGitFullHash(repoPath); err == nil {
s.Commit = hash
}
}
func cloneRepoForSource(source *Source, destPath, branch string, shallow bool, onProgress ProgressCallback) error {
if source.Commit != "" {
branch = source.Commit
}
err := cloneSourceWithFallback(source, destPath, branch, shallow, onProgress)
if err == nil {
source.recordCommit(destPath)
}
return err
}
func cloneSourceWithFallback(source *Source, destPath, branch string, shallow bool, onProgress ProgressCallback) error {
err := cloneRepoWithEnv(source.CloneURL, destPath, branch, shallow, source.authEnv(), onProgress)
if err == nil {
return nil
}
if !shouldRetryNestedGitLabURL(err) {
return err
}
var lastErr error
for _, fallback := range cloneFallbackSourcesForNestedGitLabURL(source) {
if cleanupErr := removeAll(destPath); cleanupErr != nil && !os.IsNotExist(cleanupErr) {
return fmt.Errorf("clone failed (%v), and cleanup before fallback failed: %w", err, cleanupErr)
}
if onProgress != nil {
onProgress("Clone failed; retrying as a nested GitLab repository...")
}
fallbackErr := cloneRepoWithEnv(fallback.CloneURL, destPath, branch, shallow, fallback.authEnv(), onProgress)
if fallbackErr == nil {
*source = fallback
return nil
}
lastErr = fallbackErr
}
if lastErr != nil {
return fmt.Errorf("%w (nested GitLab fallback also failed: %v)", err, lastErr)
}
return err
}
func shouldRetryNestedGitLabURL(err error) bool {
if err == nil {
return false
}
if IsAuthError(err.Error()) || IsSSLError(err.Error()) {
return false
}
low := strings.ToLower(err.Error())
blockedHints := []string{
"remote branch",
"could not resolve host",
"failed to connect",
"connection timed out",
"operation timed out",
"network is unreachable",
"permission denied",
}
for _, hint := range blockedHints {
if strings.Contains(low, hint) {
return false
}
}
retryHints := []string{
"repository not found",
"not a git repository",
"not a valid git repo",
"does not appear to be a git repository",
"requested url returned error: 404",
}
for _, hint := range retryHints {
if strings.Contains(low, hint) {
return true
}
}
if strings.Contains(low, "repository") && strings.Contains(low, "not found") {
return true
}
return false
}
func cloneFallbackSourcesForNestedGitLabURL(source *Source) []Source {
if source == nil || source.Type != SourceTypeGitHTTPS || source.Subdir == "" {
return nil
}
u, err := url.Parse(source.CloneURL)
if err != nil || u.Scheme == "" || u.Host == "" {
return nil
}
repoPath := strings.Trim(strings.TrimSuffix(u.Path, ".git"), "/")
if repoPath == "" || strings.Contains(repoPath, "/-/") {
return nil
}
subdirParts := strings.Split(strings.Trim(source.Subdir, "/"), "/")
if len(subdirParts) == 0 {
return nil
}
fallbacks := make([]Source, 0, len(subdirParts))
for i := 1; i <= len(subdirParts); i++ {
candidate := *source
candidateRepoPath := repoPath + "/" + strings.Join(subdirParts[:i], "/")
remaining := strings.Join(subdirParts[i:], "/")
candidateURL := *u
candidateURL.Path = "/" + candidateRepoPath + ".git"
candidate.CloneURL = candidateURL.String()
candidate.Subdir = remaining
if remaining == "" {
candidate.Name = filepath.Base(candidateRepoPath)
} else {
candidate.Name = filepath.Base(remaining)
}
fallbacks = append(fallbacks, candidate)
}
return fallbacks
}
// ShallowCloneToTemp clones cloneURL into a fresh temporary directory using a
// shallow (depth 1) clone, injecting auth env for the URL. The caller owns the
// returned directory and must remove it. branch may be empty to use the remote
// default. On error the temp directory is removed and an empty path returned.
func ShallowCloneToTemp(cloneURL, branch string) (string, error) {
dir, err := os.MkdirTemp("", "skillshare-clone-*")
if err != nil {
return "", err
}
if err := cloneRepo(cloneURL, dir, branch, true, nil); err != nil {
cleanupTempRepo(dir)
return "", err
}
return dir, nil
}
// gitPull performs a git pull (quiet mode).
// If the remote uses HTTPS and a token is available, it injects
// authentication via GIT_CONFIG env vars (same mechanism as cloneRepo).
func gitPull(repoPath string, onProgress ProgressCallback) error {
remoteURL := getRemoteURL(repoPath)
args := []string{"pull", "--quiet"}
if onProgress != nil {
args = []string{"pull", "--progress"}
}
return runGitCommandWithProgress(args, repoPath, authEnv(remoteURL), onProgress)
}
// getRemoteURL returns the fetch URL for the "origin" remote, or "".
func getRemoteURL(repoPath string) string {
cmd := exec.Command("git", "remote", "get-url", "origin")
cmd.Dir = repoPath
out, err := cmd.Output()
if err != nil {
return ""
}
return strings.TrimSpace(string(out))
}
// getGitCommit returns the current HEAD commit hash
func getGitCommit(repoPath string) (string, error) {
cmd := exec.Command("git", "rev-parse", "--short", "HEAD")
cmd.Dir = repoPath
output, err := cmd.Output()
if err != nil {
return "", err
}
return string(output[:len(output)-1]), nil // Remove trailing newline
}
// getGitFullHash returns the full HEAD commit hash for reliable rollback.
// NOTE: duplicates git.GetCurrentFullHash — kept here because internal/git
// imports internal/install (for AuthEnvForURL), so the reverse import would
// create a cycle.
func getGitFullHash(repoPath string) (string, error) {
cmd := exec.Command("git", "rev-parse", "HEAD")
cmd.Dir = repoPath
out, err := cmd.Output()
if err != nil {
return "", err
}
return strings.TrimSpace(string(out)), nil
}
func shortHash(hash string) string {
if len(hash) <= 12 {
return hash
}
return hash[:12]
}
func validateTrackedRepoDirName(name string) error {
if strings.TrimSpace(name) == "" {
return fmt.Errorf("tracked repo name cannot be empty")
}
if !strings.HasPrefix(name, "_") || len(name) < 2 {
return fmt.Errorf("tracked repo name must start with '_' and include at least one additional character")
}
if strings.Contains(name, "..") {
return fmt.Errorf("tracked repo name cannot contain '..'")
}
if strings.Contains(name, "/") || strings.Contains(name, "\\") {
return fmt.Errorf("tracked repo name cannot contain path separators")
}
if filepath.Clean(name) != name {
return fmt.Errorf("tracked repo name contains invalid path segments")
}
return nil
}
// gitResetHard resets the working tree to the given revision.
// NOTE: duplicates git.ResetHard — kept here to avoid import cycle
// (internal/git imports internal/install).
func gitResetHard(repoPath, rev string) error {
cmd := exec.Command("git", "reset", "--hard", rev)
cmd.Dir = repoPath
return cmd.Run()
}
// getSubdirTreeHash returns the git tree hash for a subdirectory within a repo.
// This is the SHA of the tree object for that directory — it changes only when
// files inside the subdir change, even if the repo HEAD moves.
// Returns "" on any error (no .git, subdir not found, etc.).
func getSubdirTreeHash(repoPath, subdir string) string {
if subdir == "" {
return ""
}
// Strip leading "/" from subdir — git rev-parse doesn't accept absolute paths
subdir = strings.TrimPrefix(subdir, "/")
cmd := exec.Command("git", "rev-parse", "HEAD:"+subdir)
cmd.Dir = repoPath
out, err := cmd.Output()
if err != nil {
return ""
}
return strings.TrimSpace(string(out))
}
// copyDir copies a directory recursively