7.0 KiB
Long-Session Memory (ProxyPilot / CLIProxyAPI)
CLIProxyAPI can keep long agentic CLI sessions running without manual /compact by:
- Persisting trimmed/dropped history to disk (external memory)
- Retrieving relevant snippets and injecting them back into the prompt when the request must be trimmed
This is not a 7M-token context window. It is a store + retrieval loop.
When it activates
The memory flow runs inside internal/api/middleware/codex_prompt_budget.go for agentic CLIs:
OpenAI Codexfactory-cli/droidwarp- Stainless SDK clients (
X-Stainless-*headers)
It only injects memory when a request is oversized and must be trimmed.
Where memory is stored
Default directory:
- If
util.WritablePath()is set:<WritablePath>/.proxypilot/memory - Otherwise:
./.proxypilot/memory
Per-session logs:
.proxypilot/memory/sessions/<sessionKey>/events.jsonl
The session key is chosen in this order:
X-CLIProxyAPI-Sessionheader (preferred)X-Session-Idheader- JSON
prompt_cache_key - JSON
metadata.session_id/session_id - Fallback hash of
Authorization + User-Agent(hashed; raw values are not stored)
Retrieval / injection
- Dropped items are appended as JSONL events (
dropped_chat/dropped_responses) - Retrieval is a lightweight keyword scorer over the last ~2MB of stored events
- Up to 8 snippets (~6k chars) are injected by appending to the last user message (best-effort).
Anchored summaries (heuristic)
When history is trimmed, ProxyPilot writes a lightweight anchor summary based on heuristic extraction (latest user intent + file/command hits). It does not use an LLM to summarize old turns.
Key behaviors:
- No LLM-based summarization of old turns. The anchored summary is generated by heuristic extraction (latest user intent + file/command hits) from dropped messages, not a model-produced summary.
- No explicit “~140k sweet spot” threshold or scheduled compression cycle. The memory flow only kicks in when a request exceeds budget and must be trimmed.
- No explicit cache-aware anchoring beyond append-only scaffold (we preserve append-only, but we don’t do Droid’s full anchored compression pipeline).
Environment variables:
CLIPROXY_ANCHOR_SUMMARY_MAX_CHARS(default:14000)CLIPROXY_ANCHOR_APPEND_ONLY(default: enabled)CLIPROXY_MEMORY_MAX_WRITES_PER_MIN(default:120)CLIPROXY_MEMORY_EXPORT_MAX_BYTES(default:262144000/ 250MB)
When append-only anchoring is on, the new anchor is appended once and then preserved in subsequent turns by the client. If disabled, the anchor is included in the scaffold on every request (legacy behavior).
Environment variables
CLIPROXY_MEMORY_ENABLED:- default: enabled
- set to
0,false,off, ornoto disable persistence + retrieval
CLIPROXY_MEMORY_DIR: override the base directory used to store memory
Persistent TODO (never dropped)
CLIProxyAPI maintains an optional per-session TODO file:
.proxypilot/memory/sessions/<sessionKey>/todo.md
If present, it is injected on every agentic request (before any trimming). This keeps the agent aligned even across many compression cycles.
Ways to set it:
- Write the file directly
- Or send a request header
X-CLIProxyAPI-Todo(the proxy stores it totodo.mdand strips the header before forwarding upstream)
Environment variables:
CLIPROXY_TODO_ENABLED(default: enabled)CLIPROXY_TODO_MAX_CHARS(default:4000)
Prompt-cache friendly scaffolding (append-only)
To preserve stable prompt prefixes for long sessions, CLIProxyAPI injects session state as an append-only scaffold by default (it appends a new system message/input entry instead of mutating prior content).
Packed block format:
<proxypilot_state><pinned>frompinned.md<anchor>fromsummary.md<todo>fromtodo.md<spec>when Spec Mode is enabled
Text normalization
Some clients (notably PowerShell ConvertTo-Json) can store TODO/pinned state with escaped sequences like \\n / \\u2019.
CLIProxyAPI normalizes common escape sequences when reading/writing todo.md, pinned.md, and summary.md so the injected state stays readable.
AGENTS.md pinning
If harness-root-dir is set (or provided to the middleware), CLIProxyAPI will read AGENTS.md from that directory and merge it into pinned context.
Spec Mode (optional)
Spec Mode can be enabled to require a spec/plan before any code changes:
- Header:
X-CLIProxyAPI-Spec-Mode: true - Env:
CLIPROXY_SPEC_MODE=1
When enabled and not yet approved, the scaffold injects a spec prompt that asks the agent to produce a reviewable specification and wait for approval.
Semantic memory (local embeddings)
ProxyPilot can optionally embed and retrieve semantic memory using a local Ollama embed model. This runs fully local and is always-on when enabled. Embedding of dropped history is queued in the background to keep request latency low.
Environment variables:
CLIPROXY_SEMANTIC_ENABLED(default: enabled)CLIPROXY_SEMANTIC_MODEL(default:embeddinggemma; also supportsnomic-embed-text)CLIPROXY_SEMANTIC_BASE_URL(default:http://127.0.0.1:11434)CLIPROXY_SEMANTIC_MAX_SNIPS(default:4)CLIPROXY_SEMANTIC_MAX_CHARS(default:3000)CLIPROXY_SEMANTIC_RERANK(default: enabled)CLIPROXY_SEMANTIC_KEYWORD_BOOST(default:0.25)CLIPROXY_SEMANTIC_RECENCY_BOOST(default:0.15)CLIPROXY_SEMANTIC_MAX_AGE_DAYS(default: disabled)CLIPROXY_SEMANTIC_MAX_NAMESPACES(default: disabled)CLIPROXY_SEMANTIC_MAX_BYTES_PER_NAMESPACE(default: disabled)CLIPROXY_SEMANTIC_MAX_WRITES_PER_MIN(default:120)CLIPROXY_SEMANTIC_QUERY_MAX_CHARS(default:512)
Namespace:
X-CLIProxyAPI-Repo/X-Repo-Path/X-Workspace-Root/X-Project-Root- or request
metadata.repo - fallback: session key
Memory hygiene (prune / size caps)
You can set guardrails to keep memory from growing without bound. When configured, ProxyPilot will prune on a best-effort interval (every ~10 minutes after writes).
CLIPROXY_MEMORY_MAX_AGE_DAYS(default: disabled)CLIPROXY_MEMORY_MAX_SESSIONS(default: disabled)CLIPROXY_MEMORY_MAX_BYTES_PER_SESSION(default: disabled)
The management UI also exposes manual prune + import/export tools.
Per-session semantic toggle
You can disable semantic memory per session (stored in semantic_disabled in the session folder).
This stops embeddings/retrieval for that session without disabling global semantic memory.
Management endpoints (local only):
GET /v0/management/memory/sessionsGET /v0/management/memory/session?session=...GET /v0/management/memory/events?session=...GET /v0/management/memory/anchors?session=...POST /v0/management/memory/todo|pinned|summaryPOST /v0/management/memory/semantic(enable/disable per-session semantic)POST /v0/management/memory/pruneGET /v0/management/memory/export?session=...GET /v0/management/memory/export-allPOST /v0/management/memory/import?session=...POST /v0/management/memory/delete-all