Files
ProxyPilot/docs/memory.md
T

179 lines
7.0 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Long-Session Memory (ProxyPilot / CLIProxyAPI)
CLIProxyAPI can keep long agentic CLI sessions running without manual `/compact` by:
- Persisting trimmed/dropped history to disk (external memory)
- Retrieving relevant snippets and injecting them back into the prompt when the request must be trimmed
This is **not** a 7M-token context window. It is a store + retrieval loop.
## When it activates
The memory flow runs inside `internal/api/middleware/codex_prompt_budget.go` for agentic CLIs:
- `OpenAI Codex`
- `factory-cli` / `droid`
- `warp`
- Stainless SDK clients (`X-Stainless-*` headers)
It only injects memory when a request is oversized and must be trimmed.
## Where memory is stored
Default directory:
- If `util.WritablePath()` is set: `<WritablePath>/.proxypilot/memory`
- Otherwise: `./.proxypilot/memory`
Per-session logs:
- `.proxypilot/memory/sessions/<sessionKey>/events.jsonl`
The session key is chosen in this order:
1. `X-CLIProxyAPI-Session` header (preferred)
2. `X-Session-Id` header
3. JSON `prompt_cache_key`
4. JSON `metadata.session_id` / `session_id`
5. Fallback hash of `Authorization + User-Agent` (hashed; raw values are not stored)
## Retrieval / injection
- Dropped items are appended as JSONL events (`dropped_chat` / `dropped_responses`)
- Retrieval is a lightweight keyword scorer over the last ~2MB of stored events
- Up to 8 snippets (~6k chars) are injected by appending to the **last user message** (best-effort).
## Anchored summaries (heuristic)
When history is trimmed, ProxyPilot writes a lightweight anchor summary based on **heuristic extraction**
(latest user intent + file/command hits). It does not use an LLM to summarize old turns.
Key behaviors:
- No LLM-based summarization of old turns. The anchored summary is generated by heuristic extraction (latest user intent + file/command hits) from dropped messages, not a model-produced summary.
- No explicit “~140k sweet spot” threshold or scheduled compression cycle. The memory flow only kicks in when a request exceeds budget and must be trimmed.
- No explicit cache-aware anchoring beyond append-only scaffold (we preserve append-only, but we don’t do Droid’s full anchored compression pipeline).
Environment variables:
- `CLIPROXY_ANCHOR_SUMMARY_MAX_CHARS` (default: `14000`)
- `CLIPROXY_ANCHOR_APPEND_ONLY` (default: enabled)
- `CLIPROXY_MEMORY_MAX_WRITES_PER_MIN` (default: `120`)
- `CLIPROXY_MEMORY_EXPORT_MAX_BYTES` (default: `262144000` / 250MB)
When append-only anchoring is on, the new anchor is appended once and then preserved in subsequent turns by the client.
If disabled, the anchor is included in the scaffold on every request (legacy behavior).
## Environment variables
- `CLIPROXY_MEMORY_ENABLED`:
- default: enabled
- set to `0`, `false`, `off`, or `no` to disable persistence + retrieval
- `CLIPROXY_MEMORY_DIR`: override the base directory used to store memory
## Persistent TODO (never dropped)
CLIProxyAPI maintains an optional per-session TODO file:
- `.proxypilot/memory/sessions/<sessionKey>/todo.md`
If present, it is injected on **every** agentic request (before any trimming). This keeps the agent aligned even across many compression cycles.
Ways to set it:
- Write the file directly
- Or send a request header `X-CLIProxyAPI-Todo` (the proxy stores it to `todo.md` and strips the header before forwarding upstream)
Environment variables:
- `CLIPROXY_TODO_ENABLED` (default: enabled)
- `CLIPROXY_TODO_MAX_CHARS` (default: `4000`)
## Prompt-cache friendly scaffolding (append-only)
To preserve stable prompt prefixes for long sessions, CLIProxyAPI injects session state as an **append-only scaffold** by default (it appends a new system message/input entry instead of mutating prior content).
Packed block format:
- `<proxypilot_state>`
- `<pinned>` from `pinned.md`
- `<anchor>` from `summary.md`
- `<todo>` from `todo.md`
- `<spec>` when Spec Mode is enabled
## Text normalization
Some clients (notably PowerShell `ConvertTo-Json`) can store TODO/pinned state with escaped sequences like `\\n` / `\\u2019`.
CLIProxyAPI normalizes common escape sequences when reading/writing `todo.md`, `pinned.md`, and `summary.md` so the injected state stays readable.
## AGENTS.md pinning
If `harness-root-dir` is set (or provided to the middleware), CLIProxyAPI will read `AGENTS.md` from that directory and merge it into pinned context.
## Spec Mode (optional)
Spec Mode can be enabled to require a spec/plan before any code changes:
- Header: `X-CLIProxyAPI-Spec-Mode: true`
- Env: `CLIPROXY_SPEC_MODE=1`
When enabled and not yet approved, the scaffold injects a spec prompt that asks the agent to produce a reviewable specification and wait for approval.
## Semantic memory (local embeddings)
ProxyPilot can optionally embed and retrieve semantic memory using a local Ollama embed model.
This runs fully local and is **always-on** when enabled.
Embedding of dropped history is queued in the background to keep request latency low.
Environment variables:
- `CLIPROXY_SEMANTIC_ENABLED` (default: enabled)
- `CLIPROXY_SEMANTIC_MODEL` (default: `embeddinggemma`; also supports `nomic-embed-text`)
- `CLIPROXY_SEMANTIC_BASE_URL` (default: `http://127.0.0.1:11434`)
- `CLIPROXY_SEMANTIC_MAX_SNIPS` (default: `4`)
- `CLIPROXY_SEMANTIC_MAX_CHARS` (default: `3000`)
- `CLIPROXY_SEMANTIC_RERANK` (default: enabled)
- `CLIPROXY_SEMANTIC_KEYWORD_BOOST` (default: `0.25`)
- `CLIPROXY_SEMANTIC_RECENCY_BOOST` (default: `0.15`)
- `CLIPROXY_SEMANTIC_MAX_AGE_DAYS` (default: disabled)
- `CLIPROXY_SEMANTIC_MAX_NAMESPACES` (default: disabled)
- `CLIPROXY_SEMANTIC_MAX_BYTES_PER_NAMESPACE` (default: disabled)
- `CLIPROXY_SEMANTIC_MAX_WRITES_PER_MIN` (default: `120`)
- `CLIPROXY_SEMANTIC_QUERY_MAX_CHARS` (default: `512`)
Namespace:
- `X-CLIProxyAPI-Repo` / `X-Repo-Path` / `X-Workspace-Root` / `X-Project-Root`
- or request `metadata.repo`
- fallback: session key
## Memory hygiene (prune / size caps)
You can set guardrails to keep memory from growing without bound. When configured,
ProxyPilot will prune on a best-effort interval (every ~10 minutes after writes).
- `CLIPROXY_MEMORY_MAX_AGE_DAYS` (default: disabled)
- `CLIPROXY_MEMORY_MAX_SESSIONS` (default: disabled)
- `CLIPROXY_MEMORY_MAX_BYTES_PER_SESSION` (default: disabled)
The management UI also exposes manual prune + import/export tools.
## Per-session semantic toggle
You can disable semantic memory per session (stored in `semantic_disabled` in the session folder).
This stops embeddings/retrieval for that session without disabling global semantic memory.
Management endpoints (local only):
- `GET /v0/management/memory/sessions`
- `GET /v0/management/memory/session?session=...`
- `GET /v0/management/memory/events?session=...`
- `GET /v0/management/memory/anchors?session=...`
- `POST /v0/management/memory/todo|pinned|summary`
- `POST /v0/management/memory/semantic` (enable/disable per-session semantic)
- `POST /v0/management/memory/prune`
- `GET /v0/management/memory/export?session=...`
- `GET /v0/management/memory/export-all`
- `POST /v0/management/memory/import?session=...`
- `POST /v0/management/memory/delete-all`