Files
skillshare/internal/install/source.go
T
a352b8d243 feat(install): pin skills to a tag or commit from their web URL (#293)
* feat(install): pin GitHub tree/blob URLs to their ref

Hub entries only carry a source URL, so a team hub had no way to pin a
third-party skill to a reviewed tag or commit. The parser already read
tree/<ref>/ and blob/<ref>/ URLs but discarded the ref, installing from
the default branch instead of the version the URL names.

Keep that ref as Source.Branch. An explicit --branch (CLI) or branch
(API body) still overrides it; the server handlers now only override
when a branch is given. Batch installs group by repo and ref so two
pinned versions of one repo get separate clones.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs(install): describe pinning hub sources to a tag or commit

Explain that a GitHub tree/<ref>/ source URL pins the install, in both
the hub index guide and the install reference.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat(install): pin GitLab/Bitbucket URLs and resolve refs containing "/"

Pinning should not depend on the host, so GitLab (-/tree/<ref>/) and
Bitbucket (src/<ref>/) web URLs now keep their ref too.

Taking the first path segment as the ref broke branch names containing
"/": tree/feature/x/skills/foo cloned branch "feature" and failed. When
the first segment is not a branch or tag on the remote, the longest
matching one is used instead. A ref the remote does not have fails
with a hint to use --branch, rather than silently installing the
default branch. A saved branch that covers more of the path (from
--branch or config) moves the subdir without asking the remote.

The grouped-install round-trip test pinned -/tree/main/ against a repo
on master; the ref used to be ignored, so its branch is now renamed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs(install): cover GitLab, Bitbucket and "/" branches in URL pinning

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(install): keep web URL refs correct on reinstall, --branch and grouping

Review of the slash-branch resolution found four gaps:

- A saved branch equal to the whole URL tail (tree/feature/x) did not
  move the subdir back, so every reinstall or update looked for "x".
- An unrelated --branch with a slash-branch URL kept the parse-time
  subdir (x/skills/foo) instead of the URL's real one.
- Grouped installs copy the source to a repo root; resolution then put
  the subdir back and discovery looked under a doubled path. Grouping
  also keyed on the unresolved first segment, so tree/feature/x and
  tree/feature/y could share one clone. Web URLs with a ref now install
  on their own, and resolution leaves a repo-root copy at its root.
- A skill named after the parse-time subdir kept that name once the
  path turned out to be all ref (tree/feature/x named the skill "x").

The unknown-ref error now reads as a missing ref first, since a stale
ref (tree/master after a rename) is more common than a slash branch.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(update): keep legacy installs on the default branch

Skills installed before URL refs were honoured got the default branch
and recorded no branch. Single-skill update re-parses the recorded
source, so it moved them to the URL's ref (or failed on a stale one,
with a hint to pass --branch, which update does not accept), while
update --all rebuilds from RepoURL and kept the default branch.

ApplyRecordedBranch treats an empty recorded branch as the remote
default and drops the URL's ref. It replaces the branch handling at
every site that reinstalls from recorded metadata: CLI and server
update for skills and agents, and tracked-repo rehydration.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* perf(search): share one clone across pinned hub skills from a repo

Any web URL with a path after its ref installed on its own, so a hub
listing many tree/<tag>/ skills from one repo cloned it once per skill.

groupByRepo now resolves each URL's ref before grouping, with a
RemoteRefs cache that lists each remote's refs once, and groups by
repo and resolved ref. Only refs containing "/" (whose subdir the
discovery match cannot re-derive) and refs that fail to resolve still
install alone. A resolved source is marked settled so the grouped
clone does not resolve it again, and ApplyRecordedBranch settles the
subdir from a recorded branch without asking the remote.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Willie <egg358358@gmail.com>
2026-09-28 22:08:16 +08:00

978 lines
30 KiB
Go

package install
import (
"fmt"
"net/url"
"os"
"path/filepath"
"regexp"
"strings"
"unicode"
)
const maxURLDecodeRounds = 3
// validateRepoSubdir checks that a repo subdir is a safe relative path.
// It rejects: NUL, control chars, backslashes, absolute paths, traversal (.)
// and (..) segments. It iteratively URL-decodes up to maxURLDecodeRounds to
// catch encoded traversal like %2e%2e or ..%2F... Every observable form —
// before and after each decode — must pass checkSubdirSafety. If the string
// is still encodable after exhausting all rounds, it is rejected.
func validateRepoSubdir(subdir string) error {
if subdir == "" {
return nil
}
current := subdir
for round := 0; round < maxURLDecodeRounds; round++ {
if err := checkSubdirSafety(current); err != nil {
return fmt.Errorf("unsafe subdir %q: %w", subdir, err)
}
decoded, err := url.PathUnescape(current)
if err != nil {
return fmt.Errorf("unsafe subdir %q: invalid URL encoding: %w", subdir, err)
}
if decoded == current {
return nil // stable — no more encoding to strip
}
current = decoded
}
// Final round: validate the fully-decoded result.
if err := checkSubdirSafety(current); err != nil {
return fmt.Errorf("unsafe subdir %q: %w", subdir, err)
}
// Check if it can still be decoded (deeper than maxURLDecodeRounds).
if decoded, err := url.PathUnescape(current); err == nil && decoded != current {
return fmt.Errorf("unsafe subdir %q: too deeply URL-encoded", subdir)
}
return nil
}
func checkSubdirSafety(s string) error {
for i, r := range s {
if r == 0 {
return fmt.Errorf("contains NUL at byte %d", i)
}
if r < 0x20 || r == 0x7f || (r >= 0x80 && r < 0xa0) {
return fmt.Errorf("contains control character U+%04X at byte %d", r, i)
}
if r == '\\' {
return fmt.Errorf("contains backslash at byte %d", i)
}
}
if filepath.IsAbs(s) {
return fmt.Errorf("is absolute path")
}
for _, seg := range strings.Split(s, "/") {
if seg == "." || seg == ".." {
return fmt.Errorf("contains traversal segment %q", seg)
}
}
return nil
}
// validateSourceName checks that a derived source name is safe.
// It rejects: empty, ".", "..", NUL, control chars, slashes, and backslashes.
// Dots within the name (e.g. "team.json") are allowed.
func validateSourceName(name string) error {
if name == "" || name == "." || name == ".." {
return fmt.Errorf("name is empty, dot, or dot-dot")
}
for i, r := range name {
if r == 0 {
return fmt.Errorf("name contains NUL at byte %d", i)
}
if unicode.IsControl(r) {
return fmt.Errorf("name contains control character U+%04X at byte %d", r, i)
}
if r == '/' || r == '\\' {
return fmt.Errorf("name contains path separator at byte %d", i)
}
}
return nil
}
// SourceType represents the type of installation source
type SourceType int
const (
SourceTypeUnknown SourceType = iota
SourceTypeLocalPath
SourceTypeGitHub
SourceTypeGitHTTPS
SourceTypeGitSSH
)
func (t SourceType) String() string {
switch t {
case SourceTypeLocalPath:
return "local"
case SourceTypeGitHub:
return "github"
case SourceTypeGitHTTPS:
return "git-https"
case SourceTypeGitSSH:
return "git-ssh"
default:
return "unknown"
}
}
// Source represents a parsed installation source
type Source struct {
Type SourceType
Raw string // Original user input
CloneURL string // Git clone URL (empty for local)
Subdir string // Subdirectory path for monorepo
Path string // Local path (empty for git)
Name string // Derived skill name
Branch string // Git branch to clone from (empty = remote default)
// Commit is the exact commit to check out. A lockfile sets it before the
// install; a clone fills it with the full SHA it ended up on. Unlike
// Branch it is never written back to the config as the skill's ref.
Commit string
// CNBHosts contains configured CNB hostnames for platform-specific auth/API behavior.
CNBHosts []string
// GiteaHosts contains configured Gitea hostnames for platform-specific auth/API behavior.
GiteaHosts []string
// ExplicitSkill is true when the user pointed directly at a SKILL.md file.
// That intent should resolve to exactly one skill, not a pack/discovery view.
ExplicitSkill bool
// webRef is the {ref}/{path} tail of a tree/, blob/ or src/ web URL,
// kept so a ref containing "/" can be resolved at install time.
webRef webRef
}
// GitHub URL pattern: github.com/owner/repo[/path/to/subdir]
var githubPattern = regexp.MustCompile(`^(?:https?://)?github\.com/([^/]+)/([^/]+)(?:/(.+))?$`)
// Git SSH pattern: user@host:owner/repo[.git][//subdir]
var gitSSHPattern = regexp.MustCompile(`^([^@:\s]+)@([^:\s]+):([^/]+)/(.+?)(?:\.git)?(?://(.+))?$`)
// Git SSH URL with scheme: ssh://[user@]host[:port]/path[.git][//subdir]
var sshURLPattern = regexp.MustCompile(`^ssh://(?:([^@/]+)@)?([^/:]+)(?::(\d+))?/(.+?)(?:\.git)?(?://(.+))?$`)
// Git HTTPS pattern: https://host/path (flexible path for GitLab subgroups)
var gitHTTPSPattern = regexp.MustCompile(`^(https?)://([^/]+)/(.+)$`)
// File URL pattern: file:///path/to/repo[//subdir]
var fileURLPattern = regexp.MustCompile(`^file://(.+?)(?://(.+))?$`)
// Azure DevOps HTTPS: https://dev.azure.com/{org}/{project}/_git/{repo}[/subdir]
var azureDevOpsPattern = regexp.MustCompile(
`^https?://dev\.azure\.com/([^/]+)/([^/]+)/_git/([^/?]+?)(?:\.git)?(?:/(.+))?$`)
// Azure DevOps legacy: https://{org}.visualstudio.com/{project}/_git/{repo}[/subdir]
var azureVSPattern = regexp.MustCompile(
`^https?://([^.]+)\.visualstudio\.com/([^/]+)/_git/([^/?]+?)(?:\.git)?(?:/(.+))?$`)
// Azure DevOps SSH: git@ssh.dev.azure.com:v3/{org}/{project}/{repo}[//subdir]
var azureSSHPattern = regexp.MustCompile(
`^git@ssh\.dev\.azure\.com:v3/([^/]+)/([^/]+)/(.+?)(?:\.git)?(?://(.+))?$`)
// Azure DevOps on-premises: https://{custom-host}/{org}/{project}/_git/{repo}[/subdir]
var azureOnPremPattern = regexp.MustCompile(
`^https?://([^/]+)/([^/]+)/([^/]+)/_git/([^/?]+?)(?:\.git)?(?:/(.+))?$`)
// ParseOptions holds optional configuration that affects source parsing.
type ParseOptions struct {
GitLabHosts []string // extra hostnames to treat as GitLab (nested subgroup support)
AzureHosts []string // extra hostnames to treat as Azure DevOps on-premises
CNBHosts []string // extra hostnames to treat as CNB instances
GiteaHosts []string // extra hostnames to treat as Gitea instances
}
// IsSSHURL reports whether input is an SSH URL — either scp-style
// (git@host:owner/repo.git) or scheme-style (ssh://[user@]host[:port]/path),
// optionally with a //subdir suffix. Such sources must be resolved by cloning
// rather than a direct HTTP fetch or local read.
func IsSSHURL(input string) bool {
s := strings.TrimSpace(input)
return strings.HasPrefix(s, "ssh://") || gitSSHPattern.MatchString(s)
}
// SSHIdentity extracts the SSH username and hostname from scp-style or
// scheme-style SSH sources. It returns ok=false when either part is absent.
func SSHIdentity(input string) (user, host string, ok bool) {
s := strings.TrimSpace(input)
if matches := gitSSHPattern.FindStringSubmatch(s); matches != nil {
user = strings.TrimSpace(matches[1])
host = strings.ToLower(strings.TrimSpace(matches[2]))
return user, host, user != "" && host != ""
}
if matches := sshURLPattern.FindStringSubmatch(s); matches != nil {
user = strings.TrimSpace(matches[1])
host = strings.ToLower(strings.TrimSpace(matches[2]))
return user, host, user != "" && host != ""
}
return "", "", false
}
// ParseSource analyzes the input string and returns a Source struct.
// Uses default (zero) ParseOptions.
func ParseSource(input string) (*Source, error) {
return ParseSourceWithOptions(input, ParseOptions{})
}
// ParseSourceWithOptions analyzes the input string with the given options
// and returns a Source struct.
func ParseSourceWithOptions(input string, opts ParseOptions) (*Source, error) {
input = strings.TrimSpace(input)
if input == "" {
return nil, fmt.Errorf("source cannot be empty")
}
// Expand GitHub shorthand: owner/repo -> github.com/owner/repo
input = expandGitHubShorthand(input)
source := &Source{Raw: input, CNBHosts: opts.CNBHosts, GiteaHosts: opts.GiteaHosts}
// Check for file:// URL (for testing with local git repos)
if matches := fileURLPattern.FindStringSubmatch(input); matches != nil {
return parseFileURL(matches, source)
}
// Check for local path first (starts with /, ~, or .)
if isLocalPath(input) {
return parseLocalPath(input, source)
}
// Try Azure DevOps patterns (before generic HTTPS to avoid misparse)
if matches := azureDevOpsPattern.FindStringSubmatch(input); matches != nil {
return parseAzureDevOps(matches[1], matches[2], matches[3], matches[4], source)
}
if matches := azureVSPattern.FindStringSubmatch(input); matches != nil {
return parseAzureDevOps(matches[1], matches[2], matches[3], matches[4], source)
}
if matches := azureSSHPattern.FindStringSubmatch(input); matches != nil {
return parseAzureSSH(matches[1], matches[2], matches[3], matches[4], source)
}
// Try Azure DevOps on-premises (custom host with /_git/ marker)
if len(opts.AzureHosts) > 0 {
if matches := azureOnPremPattern.FindStringSubmatch(input); matches != nil {
if isAzureHost(matches[1], opts.AzureHosts) {
return parseAzureOnPrem(matches[1], matches[2], matches[3], matches[4], matches[5], source)
}
}
}
// Try GitHub shorthand pattern
if matches := githubPattern.FindStringSubmatch(input); matches != nil {
return parseGitHub(matches, source)
}
// Try Git SSH URL with scheme (ssh://...)
if matches := sshURLPattern.FindStringSubmatch(input); matches != nil {
return parseSSHURL(matches, source)
}
// Try Git SSH pattern
if matches := gitSSHPattern.FindStringSubmatch(input); matches != nil {
return parseGitSSH(matches, source)
}
// Try Git HTTPS pattern (non-GitHub)
if matches := gitHTTPSPattern.FindStringSubmatch(input); matches != nil {
return parseGitHTTPS(matches, source, opts)
}
return nil, fmt.Errorf("unrecognized source format: %s", input)
}
// validateCloneURL checks that a CloneURL contains no NUL or control characters.
func validateCloneURL(cloneURL string) error {
for i, r := range cloneURL {
if r == 0 {
return fmt.Errorf("clone URL contains NUL at byte %d", i)
}
if r < 0x20 || r == 0x7f || (r >= 0x80 && r < 0xa0) {
return fmt.Errorf("clone URL contains control character at byte %d", i)
}
}
return nil
}
// windowsLocalPattern matches Windows drive-letter paths (C:\..., D:/...) and
// backslash-relative paths (.\..., ..\...).
var windowsLocalPattern = regexp.MustCompile(`^(?:[a-zA-Z]:[\\/]|\.{1,2}\\)`)
func isLocalPath(input string) bool {
return strings.HasPrefix(input, "/") ||
strings.HasPrefix(input, "~") ||
strings.HasPrefix(input, "./") ||
strings.HasPrefix(input, "../") ||
windowsLocalPattern.MatchString(input)
}
// expandGitHubShorthand expands owner/repo to github.com/owner/repo
// Examples:
// - anthropics/skills -> github.com/anthropics/skills
// - anthropics/skills/skills/pdf -> github.com/anthropics/skills/skills/pdf
// - ado:org/project/repo -> https://dev.azure.com/org/project/_git/repo
func expandGitHubShorthand(input string) string {
// Azure DevOps shorthand: ado:org/project/repo[/subdir]
if strings.HasPrefix(input, "ado:") {
parts := strings.SplitN(input[4:], "/", 4) // org/project/repo[/subdir]
if len(parts) >= 3 {
base := fmt.Sprintf("https://dev.azure.com/%s/%s/_git/%s", parts[0], parts[1], parts[2])
if len(parts) == 4 {
return base + "/" + parts[3]
}
return base
}
}
// Skip if already has a known prefix
if strings.HasPrefix(input, "github.com/") ||
strings.HasPrefix(input, "http://") ||
strings.HasPrefix(input, "https://") ||
strings.HasPrefix(input, "ssh://") ||
gitSSHPattern.MatchString(input) ||
strings.HasPrefix(input, "file://") ||
isLocalPath(input) {
return input
}
// Check if it looks like owner/repo (at least one slash)
if strings.Contains(input, "/") {
// If the first segment contains ".", it's a domain (e.g., gitlab.com/user/repo)
// not a GitHub owner — prepend https:// so gitHTTPSPattern can match it
firstSlash := strings.Index(input, "/")
firstSegment := input[:firstSlash]
if strings.Contains(firstSegment, ".") {
return "https://" + input
}
return "github.com/" + input
}
return input
}
func parseLocalPath(input string, source *Source) (*Source, error) {
source.Type = SourceTypeLocalPath
// Expand ~ to home directory
path := input
if strings.HasPrefix(path, "~") {
home, err := os.UserHomeDir()
if err != nil {
return nil, fmt.Errorf("cannot expand home directory: %w", err)
}
path = filepath.Join(home, path[1:])
}
// Convert to absolute path
absPath, err := filepath.Abs(path)
if err != nil {
return nil, fmt.Errorf("invalid path: %w", err)
}
source.Path = absPath
source.Name = filepath.Base(absPath)
return source, nil
}
func parseGitHub(matches []string, source *Source) (*Source, error) {
// matches: [full, owner, repo, subdir]
owner := matches[1]
if err := validateSourceName(owner); err != nil {
return nil, fmt.Errorf("invalid owner name: %w", err)
}
repo := strings.TrimSuffix(matches[2], ".git")
if err := validateSourceName(repo); err != nil {
return nil, fmt.Errorf("invalid repo name: %w", err)
}
subdir := ""
if len(matches) > 3 {
subdir = matches[3]
}
// Handle GitHub web URL format: /tree/{ref}/path or /blob/{ref}/path.
if w, ok := splitGitHubWebRef(subdir); ok {
subdir = source.applyWebRef(w)
}
// Normalize "." subdir (explicit root) to empty string
if subdir == "." {
subdir = ""
}
source.Type = SourceTypeGitHub
source.CloneURL = fmt.Sprintf("https://github.com/%s/%s.git", owner, repo)
if err := validateCloneURL(source.CloneURL); err != nil {
return nil, err
}
if subdir != "" {
if err := validateRepoSubdir(subdir); err != nil {
return nil, err
}
source.Subdir = subdir
name := filepath.Base(subdir)
if err := validateSourceName(name); err != nil {
return nil, err
}
source.Name = name
} else {
source.Name = repo
}
return source, nil
}
// splitGitHubWebRef recognises the tree/{ref}/{path} and blob/{ref}/{path}
// parts of a GitHub web URL.
func splitGitHubWebRef(subdir string) (webRef, bool) {
kind, tail, _ := strings.Cut(subdir, "/")
if (kind == "tree" || kind == "blob") && tail != "" {
return webRef{tail: tail, blob: kind == "blob"}, true
}
return webRef{}, false
}
// trimSkillFileSuffix strips a trailing SKILL.md segment from a blob URL path so
// the resulting subdir is the containing directory of the skill. Non-blob URLs
// and paths that do not end in SKILL.md are returned unchanged.
func trimSkillFileSuffix(path string, isBlob bool) (string, bool) {
if !isBlob {
return path, false
}
lastSlash := strings.LastIndex(path, "/")
name := path
if lastSlash >= 0 {
name = path[lastSlash+1:]
}
if !strings.EqualFold(name, "SKILL.md") {
return path, false
}
if lastSlash < 0 {
return "", true
}
// Keep the parent path raw so later subdir validation can reject traversal
// instead of filepath.Dir cleaning it into a seemingly safe path.
return path[:lastSlash], true
}
func parseGitSSH(matches []string, source *Source) (*Source, error) {
// matches: [full, user, host, owner, repo, subdir]
user := matches[1]
host := matches[2]
owner := matches[3]
if err := validateSourceName(owner); err != nil {
return nil, fmt.Errorf("invalid owner name: %w", err)
}
repo := strings.TrimSuffix(matches[4], ".git")
subdir := ""
if len(matches) > 5 {
subdir = matches[5]
}
source.Type = SourceTypeGitSSH
source.CloneURL = fmt.Sprintf("%s@%s:%s/%s.git", user, host, owner, repo)
if err := validateCloneURL(source.CloneURL); err != nil {
return nil, err
}
if subdir != "" {
if err := validateRepoSubdir(subdir); err != nil {
return nil, err
}
source.Subdir = subdir
name := filepath.Base(subdir)
if err := validateSourceName(name); err != nil {
return nil, err
}
source.Name = name
} else {
// Validate the final segment (actual repo name), not the full path
// which may contain subgroup separators (e.g. org/subgroup/repo).
name := filepath.Base(repo)
if err := validateSourceName(name); err != nil {
return nil, err
}
source.Name = name
}
return source, nil
}
func parseSSHURL(matches []string, source *Source) (*Source, error) {
// matches: [full, user, host, port, repoPath, subdir]
user := matches[1]
host := matches[2]
port := matches[3]
repoPath := strings.TrimSuffix(matches[4], ".git")
subdir := ""
if len(matches) > 5 {
subdir = matches[5]
}
source.Type = SourceTypeGitSSH
hostPart := host
if port != "" {
hostPart = host + ":" + port
}
userPart := ""
if user != "" {
userPart = user + "@"
}
source.CloneURL = fmt.Sprintf("ssh://%s%s/%s.git", userPart, hostPart, repoPath)
if err := validateCloneURL(source.CloneURL); err != nil {
return nil, err
}
if subdir != "" {
if err := validateRepoSubdir(subdir); err != nil {
return nil, err
}
source.Subdir = subdir
name := filepath.Base(subdir)
if err := validateSourceName(name); err != nil {
return nil, err
}
source.Name = name
} else {
// Validate the final segment (actual repo name), not the full path.
name := filepath.Base(repoPath)
if err := validateSourceName(name); err != nil {
return nil, err
}
source.Name = name
}
return source, nil
}
func parseFileURL(matches []string, source *Source) (*Source, error) {
// matches: [full, path, subdir]
path := filepath.Clean(matches[1])
subdir := ""
if len(matches) > 2 && matches[2] != "" {
subdir = strings.TrimRight(matches[2], "/")
}
source.Type = SourceTypeGitHTTPS // Treat as git for cloning
source.CloneURL = "file://" + path
if err := validateCloneURL(source.CloneURL); err != nil {
return nil, err
}
if subdir != "" {
if err := validateRepoSubdir(subdir); err != nil {
return nil, err
}
source.Subdir = subdir
name := filepath.Base(subdir)
if err := validateSourceName(name); err != nil {
return nil, err
}
source.Name = name
} else {
source.Name = filepath.Base(path)
}
return source, nil
}
func parseAzureDevOps(org, project, repo, subdir string, source *Source) (*Source, error) {
return parseAzureOnPrem("dev.azure.com", org, project, repo, subdir, source)
}
func parseAzureOnPrem(host, org, project, repo, subdir string, source *Source) (*Source, error) {
repo = strings.TrimSuffix(repo, ".git")
source.Type = SourceTypeGitHTTPS
source.CloneURL = fmt.Sprintf("https://%s/%s/%s/_git/%s", host, org, project, repo)
if err := validateCloneURL(source.CloneURL); err != nil {
return nil, err
}
if subdir != "" {
if err := validateRepoSubdir(subdir); err != nil {
return nil, err
}
source.Subdir = subdir
name := filepath.Base(subdir)
if err := validateSourceName(name); err != nil {
return nil, err
}
source.Name = name
} else {
name := filepath.Base(repo)
if err := validateSourceName(name); err != nil {
return nil, err
}
source.Name = name
}
return source, nil
}
func parseAzureSSH(org, project, repo, subdir string, source *Source) (*Source, error) {
repo = strings.TrimSuffix(repo, ".git")
source.Type = SourceTypeGitSSH
source.CloneURL = fmt.Sprintf("git@ssh.dev.azure.com:v3/%s/%s/%s", org, project, repo)
if err := validateCloneURL(source.CloneURL); err != nil {
return nil, err
}
if subdir != "" {
if err := validateRepoSubdir(subdir); err != nil {
return nil, err
}
source.Subdir = subdir
name := filepath.Base(subdir)
if err := validateSourceName(name); err != nil {
return nil, err
}
source.Name = name
} else {
name := filepath.Base(repo)
if err := validateSourceName(name); err != nil {
return nil, err
}
source.Name = name
}
return source, nil
}
func parseGitHTTPS(matches []string, source *Source, opts ParseOptions) (*Source, error) {
// matches: [full, schema, host, path]
schema := matches[1]
host := matches[2]
// Trim trailing slashes first, then /. — order matters:
// "foo/.//" → "foo/." → "foo"
path := strings.TrimRight(matches[3], "/")
path = strings.TrimSuffix(path, "/.")
var repoPath, subdir string
if strings.HasSuffix(path, ".git") {
// Explicit .git suffix marks end of repo path, no subdir
repoPath = strings.TrimSuffix(path, ".git")
} else if idx := strings.Index(path, ".git/"); idx >= 0 {
// .git/ in the middle splits repo from subdir
repoPath = path[:idx]
subdir = path[idx+len(".git/"):]
} else if idx := strings.Index(path, "/-/"); idx >= 0 {
// GitLab web URL marker: /-/tree/branch/path or /-/blob/branch/path
repoPath = path[:idx]
subdir = "-/" + path[idx+len("/-/"):]
} else if strings.Contains(host, "bitbucket") {
if idx := strings.Index(path, "/src/"); idx >= 0 {
// Bitbucket web URL: owner/repo/src/branch/path
repoPath = path[:idx]
subdir = path[idx+1:] // keep "src/..." for stripGitBranchPrefix
} else {
repoPath = path
}
} else if isGitLabHost(host, opts.GitLabHosts) {
// GitLab hosts (including on-prem like onprem.gitlab.internal)
// may have nested subgroups up to 20 levels deep.
// Without .git, treat entire path as repo.
repoPath = path
} else if isCNBHost(host, opts.CNBHosts) || isGiteaHost(host, opts.GiteaHosts) {
// CNB and Gitea Contents APIs address repositories as {owner}/{repo};
// keep the existing owner/repo + subdir split but make configured hosts
// explicit so platform detection and future URL handling stay aligned.
parts := strings.SplitN(path, "/", 3)
if len(parts) >= 2 {
repoPath = parts[0] + "/" + parts[1]
if len(parts) == 3 {
subdir = parts[2]
}
} else {
repoPath = path
}
} else {
// Default for GHE, Gitea, Gogs, and other platforms:
// assume owner/repo (2 segments), remainder is subdir.
parts := strings.SplitN(path, "/", 3)
if len(parts) >= 2 {
repoPath = parts[0] + "/" + parts[1]
if len(parts) == 3 {
subdir = parts[2]
}
} else {
repoPath = path
}
}
// Handle GitLab and Bitbucket web URLs: -/tree/{ref}/path, src/{ref}/path.
if w, ok := splitGitWebRef(host, subdir); ok {
subdir = source.applyWebRef(w)
}
// Normalize "." subdir (explicit root) to empty string
if subdir == "." {
subdir = ""
}
repoName := filepath.Base(repoPath)
if err := validateSourceName(repoName); err != nil {
return nil, fmt.Errorf("invalid repo name: %w", err)
}
source.Type = SourceTypeGitHTTPS
source.CloneURL = fmt.Sprintf("%s://%s/%s.git", schema, host, repoPath)
if err := validateCloneURL(source.CloneURL); err != nil {
return nil, err
}
if subdir != "" {
if err := validateRepoSubdir(subdir); err != nil {
return nil, err
}
source.Subdir = subdir
name := filepath.Base(subdir)
if err := validateSourceName(name); err != nil {
return nil, err
}
source.Name = name
} else {
source.Name = repoName
}
return source, nil
}
func hostMatchesAny(host string, list []string) bool {
for _, h := range list {
if strings.EqualFold(h, host) {
return true
}
}
return false
}
// isAzureHost returns true if the host is an Azure DevOps on-premises instance.
func isAzureHost(host string, extraHosts []string) bool {
return hostMatchesAny(host, extraHosts)
}
// isGitLabHost returns true if the host should be treated as a GitLab instance.
func isGitLabHost(host string, extraHosts []string) bool {
return strings.Contains(host, "gitlab") || strings.Contains(host, "jihulab") ||
hostMatchesAny(host, extraHosts)
}
// isCNBHost returns true if the host should be treated as a CNB instance.
func isCNBHost(host string, extraHosts []string) bool {
return strings.Contains(host, "cnb.cool") || hostMatchesAny(host, extraHosts)
}
// isGiteaHost returns true if the host should be treated as a Gitea instance.
func isGiteaHost(host string, extraHosts []string) bool {
return strings.Contains(host, "gitea") || hostMatchesAny(host, extraHosts)
}
// detectPlatformFromHost returns the platform for a given hostname, using
// configured extra hosts for CNB and Gitea self-hosted instances.
func detectPlatformFromHost(host string, cnbHosts, giteaHosts []string) Platform {
host = strings.ToLower(host)
if host == "" {
return PlatformUnknown
}
if strings.Contains(host, "github") {
return PlatformGitHub
}
if strings.Contains(host, "gitlab") {
return PlatformGitLab
}
if strings.Contains(host, "bitbucket") {
return PlatformBitbucket
}
if host == "dev.azure.com" || host == "ssh.dev.azure.com" || strings.HasSuffix(host, ".visualstudio.com") {
return PlatformAzureDevOps
}
if isCNBHost(host, cnbHosts) {
return PlatformCNB
}
if isGiteaHost(host, giteaHosts) {
return PlatformGitea
}
return PlatformUnknown
}
// splitGitWebRef recognises the {ref}/{path} part of Bitbucket (src/{ref}/path)
// and GitLab (-/tree/{ref}/path, -/blob/{ref}/path) web URLs.
func splitGitWebRef(host, subdir string) (webRef, bool) {
subdir = strings.TrimRight(subdir, "/")
// Bitbucket has no separate blob marker, so a trailing SKILL.md is treated
// the same as a blob URL.
if strings.Contains(host, "bitbucket") {
if tail, ok := strings.CutPrefix(subdir, "src/"); ok && tail != "" {
return webRef{tail: tail, blob: true}, true
}
}
if rest, ok := strings.CutPrefix(subdir, "-/"); ok {
kind, tail, _ := strings.Cut(rest, "/")
if kind == "tree" || kind == "blob" {
return webRef{tail: tail, blob: kind == "blob"}, true
}
}
return webRef{}, false
}
// HasSubdir returns true if this source requires subdirectory extraction
func (s *Source) HasSubdir() bool {
return s.Subdir != ""
}
// TargetsExplicitSkill reports whether the user pointed directly at a SKILL.md file.
func (s *Source) TargetsExplicitSkill() bool {
return s != nil && s.ExplicitSkill
}
// IsGit returns true if this source requires git clone
func (s *Source) IsGit() bool {
return s.Type == SourceTypeGitHub ||
s.Type == SourceTypeGitHTTPS ||
s.Type == SourceTypeGitSSH
}
// GitHubOwner returns the repository owner for GitHub/GHE sources.
// Returns empty string for non-GitHub hosts or unparsable URLs.
func (s *Source) GitHubOwner() string {
owner, _ := s.gitHubOwnerRepo()
return owner
}
// GitHubRepo returns the repository name for GitHub/GHE sources.
// Returns empty string for non-GitHub hosts or unparsable URLs.
func (s *Source) GitHubRepo() string {
_, repo := s.gitHubOwnerRepo()
return repo
}
func (s *Source) gitHubOwnerRepo() (owner, repo string) {
_, owner, repo = s.gitHubHostOwnerRepo()
return owner, repo
}
// GitHubAPIBase returns the REST API base URL for GitHub-family sources
// (https://api.github.com for github.com, https://api.<host>.ghe.com for
// GHE Cloud/Data Residency, or https://<host>/api/v3 for GHE Server).
// Returns "" for non-GitHub hosts or unparsable URLs.
func (s *Source) GitHubAPIBase() string {
host, _, _ := s.gitHubHostOwnerRepo()
if host == "" {
return ""
}
return gitHubAPIBaseForHost(host)
}
// gitHubHostOwnerRepo extracts the host, owner, and repo from a GitHub-family
// clone URL (HTTPS or SSH). Returns empty strings for non-GitHub hosts.
func (s *Source) gitHubHostOwnerRepo() (host, owner, repo string) {
cloneURL := strings.TrimSpace(s.CloneURL)
if cloneURL == "" {
return "", "", ""
}
// SSH clone URL: user@host:owner/repo.git
if sshMatches := gitSSHPattern.FindStringSubmatch(cloneURL); sshMatches != nil {
host = strings.ToLower(strings.TrimSpace(sshMatches[2]))
if !isGitHubLikeHost(host) {
return "", "", ""
}
return host, sshMatches[3], strings.TrimSuffix(sshMatches[4], ".git")
}
u, err := url.Parse(cloneURL)
if err != nil {
return "", "", ""
}
host = strings.ToLower(u.Hostname())
if !isGitHubLikeHost(host) {
return "", "", ""
}
parts := strings.Split(strings.Trim(u.Path, "/"), "/")
if len(parts) < 2 {
return "", "", ""
}
owner = parts[0]
repo = strings.TrimSuffix(parts[1], ".git")
if owner == "" || repo == "" {
return "", "", ""
}
return host, owner, repo
}
// TrackName returns a unique name for --track mode by joining path segments with "-".
// For GitHub: https://github.com/openai/skills.git → "openai-skills"
// For SSH: git@github.com:openai/skills.git → "openai-skills"
// For HTTPS: https://gitlab.com/team/repo.git → "team-repo"
// For subgroups: https://gitlab.com/group/subgroup/project.git → "group-subgroup-project"
// For Azure SSH: git@ssh.dev.azure.com:v3/org/proj/repo → "org-proj-repo"
// For Azure: https://dev.azure.com/org/proj/_git/repo → "org-proj-repo"
// Falls back to source.Name if path cannot be extracted.
func (s *Source) TrackName() string {
cloneURL := s.CloneURL
if cloneURL == "" {
return s.Name
}
// Azure DevOps SSH: git@ssh.dev.azure.com:v3/org/project/repo
if sshMatches := azureSSHPattern.FindStringSubmatch(s.Raw); sshMatches != nil {
return sshMatches[1] + "-" + sshMatches[2] + "-" + strings.TrimSuffix(sshMatches[3], ".git")
}
// Azure DevOps HTTPS: dev.azure.com/org/project/_git/repo
if strings.Contains(cloneURL, "dev.azure.com") || strings.Contains(cloneURL, "visualstudio.com") {
u, err := url.Parse(cloneURL)
if err == nil {
parts := strings.Split(strings.Trim(u.Path, "/"), "/")
// parts: [org, project, _git, repo]
if len(parts) >= 4 && parts[2] == "_git" {
return parts[0] + "-" + parts[1] + "-" + parts[3]
}
}
}
// Azure DevOps on-premises: https://custom-host/org/project/_git/repo
if strings.Contains(cloneURL, "/_git/") {
u, err := url.Parse(cloneURL)
if err == nil {
parts := strings.Split(strings.Trim(u.Path, "/"), "/")
if len(parts) >= 4 && parts[len(parts)-2] == "_git" {
return parts[len(parts)-4] + "-" + parts[len(parts)-3] + "-" + parts[len(parts)-1]
}
}
}
// Try SSH format: user@host:owner/repo.git
if sshMatches := gitSSHPattern.FindStringSubmatch(s.Raw); sshMatches != nil {
owner := sshMatches[3]
repo := strings.TrimSuffix(sshMatches[4], ".git")
// Replace / with - to handle subgroup paths (e.g., group/subgroup/repo)
return owner + "-" + strings.ReplaceAll(repo, "/", "-")
}
// Try extracting full path from HTTPS clone URL
cloneURL = strings.TrimSuffix(cloneURL, ".git")
if u, err := url.Parse(cloneURL); err == nil {
pathStr := strings.Trim(u.Path, "/")
if pathStr != "" {
return strings.ReplaceAll(pathStr, "/", "-")
}
}
return s.Name
}
// MetaType returns the type string for metadata
func (s *Source) MetaType() string {
if s.HasSubdir() {
return s.Type.String() + "-subdir"
}
return s.Type.String()
}