mirror of
https://github.com/Finesssee/ProxyPilot.git
synced 2026-11-18 18:04:20 +00:00
Update docs to reflect token-aware compression and LLM summarization features. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
187 lines
7.4 KiB
Markdown
187 lines
7.4 KiB
Markdown
# Long-Session Memory (ProxyPilot / CLIProxyAPI)
|
||
|
||
CLIProxyAPI can keep long agentic CLI sessions running without manual `/compact` by:
|
||
|
||
- Persisting trimmed/dropped history to disk (external memory)
|
||
- Retrieving relevant snippets and injecting them back into the prompt when the request must be trimmed
|
||
|
||
This is **not** a 7M-token context window. It is a store + retrieval loop.
|
||
|
||
## When it activates
|
||
|
||
The memory flow runs inside `internal/api/middleware/codex_prompt_budget.go` for agentic CLIs:
|
||
|
||
- `OpenAI Codex`
|
||
- `factory-cli` / `droid`
|
||
- `warp`
|
||
- Stainless SDK clients (`X-Stainless-*` headers)
|
||
|
||
It only injects memory when a request is oversized and must be trimmed.
|
||
|
||
## Where memory is stored
|
||
|
||
Default directory:
|
||
|
||
- If `util.WritablePath()` is set: `<WritablePath>/.proxypilot/memory`
|
||
- Otherwise: `./.proxypilot/memory`
|
||
|
||
Per-session logs:
|
||
|
||
- `.proxypilot/memory/sessions/<sessionKey>/events.jsonl`
|
||
|
||
The session key is chosen in this order:
|
||
|
||
1. `X-CLIProxyAPI-Session` header (preferred)
|
||
2. `X-Session-Id` header
|
||
3. JSON `prompt_cache_key`
|
||
4. JSON `metadata.session_id` / `session_id`
|
||
5. Fallback hash of `Authorization + User-Agent` (hashed; raw values are not stored)
|
||
|
||
## Retrieval / injection
|
||
|
||
- Dropped items are appended as JSONL events (`dropped_chat` / `dropped_responses`)
|
||
- Retrieval is a lightweight keyword scorer over the last ~2MB of stored events
|
||
- Up to 8 snippets (~6k chars) are injected by appending to the **last user message** (best-effort).
|
||
|
||
## Anchored summaries (heuristic)
|
||
|
||
When history is trimmed, ProxyPilot writes a lightweight anchor summary based on **heuristic extraction**
|
||
(latest user intent + file/command hits). It does not use an LLM to summarize old turns.
|
||
|
||
Key behaviors:
|
||
|
||
- No LLM-based summarization of old turns. The anchored summary is generated by heuristic extraction (latest user intent + file/command hits) from dropped messages, not a model-produced summary.
|
||
- No explicit “~140k sweet spot” threshold or scheduled compression cycle. The memory flow only kicks in when a request exceeds budget and must be trimmed.
|
||
- No explicit cache-aware anchoring beyond append-only scaffold (we preserve append-only, but we don’t do Droid’s full anchored compression pipeline).
|
||
|
||
Environment variables:
|
||
|
||
- `CLIPROXY_ANCHOR_SUMMARY_MAX_CHARS` (default: `14000`)
|
||
- `CLIPROXY_ANCHOR_APPEND_ONLY` (default: enabled)
|
||
- `CLIPROXY_MEMORY_MAX_WRITES_PER_MIN` (default: `120`)
|
||
- `CLIPROXY_MEMORY_EXPORT_MAX_BYTES` (default: `262144000` / 250MB)
|
||
|
||
When append-only anchoring is on, the new anchor is appended once and then preserved in subsequent turns by the client.
|
||
If disabled, the anchor is included in the scaffold on every request (legacy behavior).
|
||
|
||
## Environment variables
|
||
|
||
- `CLIPROXY_MEMORY_ENABLED`:
|
||
- default: enabled
|
||
- set to `0`, `false`, `off`, or `no` to disable persistence + retrieval
|
||
- `CLIPROXY_MEMORY_DIR`: override the base directory used to store memory
|
||
|
||
## Persistent TODO (never dropped)
|
||
|
||
CLIProxyAPI maintains an optional per-session TODO file:
|
||
|
||
- `.proxypilot/memory/sessions/<sessionKey>/todo.md`
|
||
|
||
If present, it is injected on **every** agentic request (before any trimming). This keeps the agent aligned even across many compression cycles.
|
||
|
||
Ways to set it:
|
||
|
||
- Write the file directly
|
||
- Or send a request header `X-CLIProxyAPI-Todo` (the proxy stores it to `todo.md` and strips the header before forwarding upstream)
|
||
|
||
Environment variables:
|
||
|
||
- `CLIPROXY_TODO_ENABLED` (default: enabled)
|
||
- `CLIPROXY_TODO_MAX_CHARS` (default: `4000`)
|
||
|
||
## Prompt-cache friendly scaffolding (append-only)
|
||
|
||
To preserve stable prompt prefixes for long sessions, CLIProxyAPI injects session state as an **append-only scaffold** by default (it appends a new system message/input entry instead of mutating prior content).
|
||
|
||
Packed block format:
|
||
|
||
- `<proxypilot_state>`
|
||
- `<pinned>` from `pinned.md`
|
||
- `<anchor>` from `summary.md`
|
||
- `<todo>` from `todo.md`
|
||
- `<spec>` when Spec Mode is enabled
|
||
|
||
## Text normalization
|
||
|
||
Some clients (notably PowerShell `ConvertTo-Json`) can store TODO/pinned state with escaped sequences like `\\n` / `\\u2019`.
|
||
CLIProxyAPI normalizes common escape sequences when reading/writing `todo.md`, `pinned.md`, and `summary.md` so the injected state stays readable.
|
||
|
||
## AGENTS.md pinning
|
||
|
||
If `harness-root-dir` is set (or provided to the middleware), CLIProxyAPI will read `AGENTS.md` from that directory and merge it into pinned context.
|
||
|
||
## Spec Mode (optional)
|
||
|
||
Spec Mode can be enabled to require a spec/plan before any code changes:
|
||
|
||
- Header: `X-CLIProxyAPI-Spec-Mode: true`
|
||
- Env: `CLIPROXY_SPEC_MODE=1`
|
||
|
||
When enabled and not yet approved, the scaffold injects a spec prompt that asks the agent to produce a reviewable specification and wait for approval.
|
||
|
||
## Semantic memory (local embeddings)
|
||
|
||
ProxyPilot can optionally embed and retrieve semantic memory using a local Ollama embed model.
|
||
This runs fully local and is **always-on** when enabled.
|
||
Embedding of dropped history is queued in the background to keep request latency low.
|
||
|
||
Environment variables:
|
||
|
||
- `CLIPROXY_SEMANTIC_ENABLED` (default: enabled)
|
||
- `CLIPROXY_SEMANTIC_MODEL` (default: `embeddinggemma`; also supports `nomic-embed-text`)
|
||
- `CLIPROXY_SEMANTIC_BASE_URL` (default: `http://127.0.0.1:11434`)
|
||
- `CLIPROXY_SEMANTIC_MAX_SNIPS` (default: `4`)
|
||
- `CLIPROXY_SEMANTIC_MAX_CHARS` (default: `3000`)
|
||
- `CLIPROXY_SEMANTIC_RERANK` (default: enabled)
|
||
- `CLIPROXY_SEMANTIC_KEYWORD_BOOST` (default: `0.25`)
|
||
- `CLIPROXY_SEMANTIC_RECENCY_BOOST` (default: `0.15`)
|
||
- `CLIPROXY_SEMANTIC_MAX_AGE_DAYS` (default: disabled)
|
||
- `CLIPROXY_SEMANTIC_MAX_NAMESPACES` (default: disabled)
|
||
- `CLIPROXY_SEMANTIC_MAX_BYTES_PER_NAMESPACE` (default: disabled)
|
||
- `CLIPROXY_SEMANTIC_MAX_WRITES_PER_MIN` (default: `120`)
|
||
- `CLIPROXY_SEMANTIC_QUERY_MAX_CHARS` (default: `512`)
|
||
|
||
Namespace:
|
||
|
||
- `X-CLIProxyAPI-Repo` / `X-Repo-Path` / `X-Workspace-Root` / `X-Project-Root`
|
||
- or request `metadata.repo`
|
||
- fallback: session key
|
||
|
||
## Memory hygiene (prune / size caps)
|
||
|
||
You can set guardrails to keep memory from growing without bound. When configured,
|
||
ProxyPilot will prune on a best-effort interval (every ~10 minutes after writes).
|
||
|
||
- `CLIPROXY_MEMORY_MAX_AGE_DAYS` (default: disabled)
|
||
- `CLIPROXY_MEMORY_MAX_SESSIONS` (default: disabled)
|
||
- `CLIPROXY_MEMORY_MAX_BYTES_PER_SESSION` (default: disabled)
|
||
|
||
The management UI also exposes manual prune + import/export tools.
|
||
|
||
## Per-session semantic toggle
|
||
|
||
You can disable semantic memory per session (stored in `semantic_disabled` in the session folder).
|
||
This stops embeddings/retrieval for that session without disabling global semantic memory.
|
||
|
||
Management endpoints (local only):
|
||
|
||
- `GET /v0/management/memory/sessions`
|
||
- `GET /v0/management/memory/session?session=...`
|
||
- `GET /v0/management/memory/events?session=...`
|
||
- `GET /v0/management/memory/anchors?session=...`
|
||
- `POST /v0/management/memory/todo|pinned|summary`
|
||
- `POST /v0/management/memory/semantic` (enable/disable per-session semantic)
|
||
- `POST /v0/management/memory/prune`
|
||
- `GET /v0/management/memory/export?session=...`
|
||
- `GET /v0/management/memory/export-all`
|
||
- `POST /v0/management/memory/import?session=...`
|
||
- `POST /v0/management/memory/delete-all`
|
||
|
||
## LLM-Based Context Compression
|
||
|
||
For advanced context compression using LLM-generated structured summaries and token-aware proactive trimming, see [Context Compression](context-compression.md).
|
||
|
||
## Agentic Harness
|
||
|
||
For structured incremental development workflows (INITIALIZER/CODING modes, feature tracking, progress logging), see [Agentic Harness](agentic-harness.md).
|