Use one skills traversal policy for dashboard update discovery, post-pull audits, and staged regular-skill writes so linked repositories are audited and child updates preserve their source links. Keep agent updates and disabled follow behavior on the nil-policy path.
Cover global and project modes through HTTP and SSE handlers, including audit rollback and disabled-follow parity.
Users link a checkout directly under the skills source (e.g.
skills/_dev-skills -> ~/code/dev-skills) and expect it to behave like any
other first-level entry. sourcewalk resolved only the root, so these links
were invisible to discovery, sync, update, and the dashboard.
follow_source_links (global and project config, default false) enables a
per-operation sourcewalk.Follow policy. Walk, WalkDir, and ReadDir report a
followed link under its own name as a directory and descend into its
target, so names, RelPath, FlatName, SourcePath, tracked-repo detection, and
metadata keys stay logical. Only entries directly under the source root are
followed. The policy is built from config and passed explicitly to every
discovery caller in the CLI, server, and reconcile, so counts agree; the
server keeps a per-instance snapshot refreshed on config load and save
instead of a process-global flag.
A link is skipped, with one warning per operation, when its target is
missing or unreadable, is the source or a parent of it, or overlaps an
active skills target. A missing or unreadable target marks the operation
incomplete: sync then skips pruning and copy-orphan deletion, and reconcile
keeps metadata, so an unmounted drive or a re-clone in progress cannot
erase target entries or install records. A link to an existing file is not
a directory link and stays an ordinary entry; treating it as incomplete
would disable pruning for anyone sharing a file such as .skillignore.
doctor keeps raw link identity: with the key off it still says the link is
not followed and names follow_source_links: true; with it on it reports
followed links and skipped links with the reason.
The raw-walk allowlist gains entries for the policy path in Walk and
WalkDir: the source root walk with a policy, and the walk of a validated
link target whose paths are remapped to the logical link path. Both live
in sourcewalk itself, the package that owns source traversal.
Refs #274
GET /api/audit/{name} joined the client-supplied name to the source
without checking it, so a name such as ../../x scanned a directory
outside the skills source and returned a clean result for it. The
single-skill update, the update stream, and the batch uninstall built
paths the same way; the tracked-repo uninstall route already refused
such names inline.
One helper, validSourceName, now holds that rule: non-empty, relative,
not starting with "..", and no Windows separators or volume prefixes on
any host, as sourcewalk does for declared names. The audit and update
routes answer 400, the update stream reports the name as an error
result, the batch uninstall fails that item, and the repo uninstall
route uses the same helper. Nested names such as group/skill still
resolve.
Closes#402
Every API request reloads the skills and agents metadata stores and
swaps the server's pointers under s.mu, but several handlers read or
wrote the stores outside that lock: listing, skill detail, check, the
update stream, and the hash refresh after setting targets. Concurrent
requests could race on the pointer or on the stores' maps, which the
race detector reported hundreds of times under load and which can
crash the server with a concurrent map access.
Store access now always happens under s.mu. Reads return a copy of
the entry so it can be used after unlocking, and target handlers hash
files outside the lock and only write the result under it.
InstallOptions.Force carried two meanings: "overwrite what is already
there" and "proceed despite audit findings". Update paths set Force
unconditionally to overwrite a staged temp dir, so they had to leave it
off to keep the audit gate alive -- which swallowed the caller's --force
and made the gate impossible to override from either the CLI or the web
UI.
Split the audit intent into InstallOptions.AuditOverride and route every
gate through it alone. Entry points translate --force explicitly, so a
missed one degrades to "force has no effect" rather than "audit is
bypassed". Scan failures stay fail-closed regardless: accepting known
findings is not the same as being unable to learn any.
This also closes two paths that bypassed the gate outright, since both
inherited Force for overwrite reasons alone: install --json and grouped
batch updates.
In the web UI, Force Retry now appears only on failures force can
actually resolve -- a permission-denied unlink previously offered a
button that could never succeed -- and CLI-only flag hints are stripped
from messages rendered there.
Behaviour change: a --force at install time no longer exempts later
updates from the audit gate; it is a one-off decision per command.
When tracked repo metadata is committed but the gitignored clone dirs are
absent on a fresh machine, `update --all` silently reported zero work: the
batch path skipped missing repos with no message, and the web UI's
update-all paths only listed repos present on disk.
Missing tracked repos are now detected and surfaced everywhere:
- CLI: a warning block (human output) plus an aggregated
`missing_tracked_repos` summary (names + one-shot rehydrate hint) in JSON;
the per-item error is concise. Fixed for both global and project modes,
and batch and single-target paths.
- Server: GET /api/update/missing-tracked-repos and the update-all/stream
summaries report them; POST /api/update/rehydrate re-clones them from
metadata (mirrors bare `skillshare install`).
- UI: the Update page shows a warning banner listing missing repos with a
Rehydrate button; strings added to all 11 locales.
Closes#212
Make the v0.19.16 sources map actually do something: route all path
reads through Config.EffectiveSkillsSource() / EffectiveAgentsSource() /
EffectiveExtrasSource() so users who set sources.skills / sources.agents /
sources.extras get the configured paths everywhere.
Mechanical changes (~50 files):
- cmd/skillshare/**: replace cfg.Source / cfg.ExtrasSource reads with
cfg.EffectiveSkillsSource() / cfg.EffectiveExtrasSource()
- internal/server/**: replace s.cfg.Source / s.cfg.ExtrasSource reads
with the same helpers; server.skillsSource() / gitignoreDir() use the
global-mode helper
- internal/config/config_validate.go: validate using EffectiveSkillsSource
while still rejecting configs that set neither field explicitly
- internal/config/reconcile.go, registry.go: read via helpers
Behavioral fixes (paired with the migration):
- extras init / extras source / POST /api/extras no longer silently
backfill the legacy extras_source field — the runtime fallback inside
EffectiveExtrasSource() handles it. This stops surprise diffs in users'
config.yaml after running extras commands.
- extras source command writes back to whichever field the user already
configured (sources.extras if set, otherwise legacy extras_source) so
the update is not shadowed.
- source_type in /api/extras and extras list correctly reports "default"
vs "extras_source" by inspecting the raw (unresolved) configured value;
sources.extras counts as configured for this purpose.
UI:
- ui/src/lib/fieldDocs.ts: neutralize sources / sources.* descriptions
so they describe both project and global modes (Phase 5).
Preserved (intentional):
- Direct writes to cfg.Source / cfg.AgentsSource / cfg.ExtrasSource in
init, migrate.go, cloneForSave, and server.reloadConfig — these are
raw-field writes, not path reads.
- Updated test expectations for the removed backfills.
Addresses P1 and P2 from code review on the RWMutex commit:
P1 — Stream handlers now snapshot config under RLock before streaming:
- handleDiffStream: clone Targets map to prevent concurrent map panic
- handleCheckStream, handleAuditStream: snapshot source/projectRoot
- handleUpdateStream: snapshot source (per-item Lock already correct)
P2 — All read handlers now use narrow snapshot instead of defer RUnlock:
- RLock → copy needed fields to locals → RUnlock → do work with locals
- Prevents slow I/O (git fetch, version check, audit scan) from blocking
the auto-reload middleware's exclusive Lock on concurrent requests
Other changes:
- computeTargetDiff takes source parameter (no more s.cfg.Source access)
- Add cloneTargets() helper for safe map copying
- Add writeOpsLogTo/writeAuditLogTo for lock-free oplog writes
Replace sequential POST loops and blocking useQuery calls with
Server-Sent Events for real-time progress feedback on Update, Check,
and Diff operations. Extract shared helpers to reduce duplication:
- initSSE() in middleware.go consolidates SSE preamble across 4 handlers
- createSSEStream() in client.ts unifies EventSource lifecycle management
- StreamProgressBar component replaces 3 near-identical progress bars
- handleDiff now delegates to computeTargetDiff (eliminates ~150 lines)
- Paginate audit results (20 cards + 10 findings per card) to prevent
DOM overload with large skill counts