1
Locate
Use a known source ID, time range, or curated-session link to select one or a very small number of raw records.
Reference architecture · sanitized edition
Durable memory without turning context into a junk drawer.
A provenance-first pattern for preserving source history, rebuilding derived indexes, retrieving the smallest useful context, and reviving raw evidence only when curated memory is not enough.
Share the method. Protect the memory.
DurableKeep source identity, timestamps, fidelity, and provenance.
RebuildableTreat chunks, projections, rankings, and caches as disposable.
ScopedLoad the smallest context that can support the answer.
ExplainableEvery recalled claim retains a path back to evidence.
The default retrieval path
Normal recall runs on reviewed, curated material. Raw source storage stays outside the everyday index.
① Curated durable sources
- canonical sessions
- reviewed knowledge
- dated current state
- decision provenance
→build
② Rebuildable retrieval data
- label validation
- source catalog
- bounded chunks
- state projections
- decision projections
→rank
③ Context resolver
- RETRIEVE / ASK / SKIP
- topic and entity hints
- fidelity and stale-state checks
- context budget
→packet
④ Current interaction
- minimal relevant context
- visible uncertainty
- source-aware response
- no summary-as-quote
Cold lane · closed by defaultEnter only after curated evidence proves insufficient and the target can be narrowed.
Metadata-only raw catalogLocate by source ID, date, curated link, path, size, and hash—never by conversation text.
→
Verified raw sourcePreserve an exact, read-only copy with structure and integrity checks.
→
Targeted revivalParse one known source, recover only the required turns, then derive a new projection if useful.
Cold retrieval is an evidence escalation, not a second default search
The raw catalog is a warehouse card: it identifies the box and verifies the seal. It does not expose the contents to ordinary recall.
2
Verify
Check path containment, file mode, byte size, cryptographic hash, record count, source identity, and timestamps.
3
Rehydrate minimally
Separate visible dialogue from system context, reasoning, and tool records. Recover only what the current question requires.
Clear ownership prevents memory corruption
Each layer has a narrow job. A convenient derived layer never becomes more authoritative than its source.
Operating rules
Current safety, retrieval, privacy, and collaboration rules. Historical content cannot promote itself into this layer.
Curated archive
Canonical sessions, reviewed facts, dated state, decision history, and source links used for normal evidence-backed recall.
Raw cold storage
Exact source preservation plus a metadata-only locator. Private, default-off, and read only during targeted recovery.
Retrieval and context
Chunks, aliases, state projections, scores, traces, and context packets. Observable, replaceable, and safe to rebuild.
Time is evidence, not a fixed TTL
Facts change at different rates. Passing an expected review point does not prove that the state reversed.
observed_at
When the observation was recorded. It answers “when did we see this?”, not “when must it expire?”
valid_from / valid_to
Store only a genuinely known applicability interval. Leave an unknown end open instead of inventing a TTL.
last_confirmed_at
The latest explicit confirmation of the same state. Collection reviewed_at is not a substitute.
expected_resolution_at
An expected review or outcome date. Passing it adds a visible note; it does not silently rewrite state.
Testable rule: inject as_of. Keep base ranking stable and add a temporal label only when the checkpoint passed without a later confirmation.
Evidence escalation workflow
Escalate only as far as the question requires.
| Signal | First action | Escalate when | Guardrail |
|---|---|---|---|
| Current context is sufficient | Answer without retrieval | Never | Do not search private history by habit. |
| Prior context is clearly relevant | Run curated retrieval | The packet lacks exact evidence or provenance. | Use a concise intent, not the entire user message. |
| Exact wording, chronology, or rationale is required | Search the private curated archive | The best surviving curated source is still incomplete. | Do not present a summary as verbatim dialogue. |
| A known source must be recovered | Use the raw catalog, then verify | Only after the target has been narrowed. | Never scan or load the whole raw store by default. |
| The question lacks a usable anchor | Ask one minimal clarifying question | The answer supplies a source, event, entity, or time anchor. | Do not guess which private history the user meant. |
Separate must-pass behavior from probes that expose problems
Failure must have an explicit meaning. Raw scores from heterogeneous rankers must not pretend to be one probability scale.
| Lane | Purpose | Run semantics | Promotion rule |
|---|---|---|---|
| Regression | Protect behavior already promised | Every normal run must pass | Accept only fixed, stable cases |
| Challenge / probe | Preserve real misses and hand-written contrasts | Known failures may be reported explicitly | Promote after the fix is stable |
provenance_supported
The candidate has a canonical source link or explicit provenance. This labels evidence shape, not probability.
structured_context
Topic, entity, state, or decision fields explain why the candidate was recalled.
lexical_only
Wording overlaps without stronger structure. Measure forbidden hits and packet pollution.
SKIP miss ownership
Start at semantic triage. Use a few human contrasts and real misses, not a large generated context-free query set.
Reference repository layout
This is a portable pattern, not a copy of any live personal-memory workspace.
Durable and private layers
memory-system/
├── policy/ current operating rules
├── archive/
│ ├── snapshots/ immutable curated history
│ ├── active/ append-only local overlay
│ └── raw-sources/
│ ├── catalog.jsonl metadata only
│ └── source-type/YYYY/MM/DD/
├── knowledge/ stable reviewed facts
└── private-state/ dated sensitive current state
├── policy/ current operating rules
├── archive/
│ ├── snapshots/ immutable curated history
│ ├── active/ append-only local overlay
│ └── raw-sources/
│ ├── catalog.jsonl metadata only
│ └── source-type/YYYY/MM/DD/
├── knowledge/ stable reviewed facts
└── private-state/ dated sensitive current state
Derived and replaceable layers
memory-system/
├── retrieval/ planner, ranking, context builder
├── derived/
│ ├── sources.jsonl
│ ├── chunks.jsonl
│ ├── current-states.json
│ ├── decisions.json
│ └── traces/
├── scripts/ build, search, verify
└── tests/
├── regression/ must pass
└── challenge/ explicit known failures
├── retrieval/ planner, ranking, context builder
├── derived/
│ ├── sources.jsonl
│ ├── chunks.jsonl
│ ├── current-states.json
│ ├── decisions.json
│ └── traces/
├── scripts/ build, search, verify
└── tests/
├── regression/ must pass
└── challenge/ explicit known failures
Fidelity is part of the answer
Use the strongest surviving evidence, but label transformed material honestly.
Visible transcriptdirect dialogue
Transcript snapshotpreserved export
Hybrid recordsource + curation
Structured summaryrecovery aid
Derived projectionlocator, not source
Compact model: the archive remembers what happened; retrieval decides what matters now; the raw catalog knows where the evidence box lives; policy defines what the assistant may do.
Reject bad labels. Keep useful history.
Two concrete mechanisms from the reviewed implementation, illustrated with a fictional expedition.
Canonical labels before publication
Validate available state and decision records against config keys, including inactive history. An unknown label stops the build before index writes and identifies the file, record, field, and label.
A typo is a warning
An unknown user hint warns and querying continues. The runtime recognition set cannot validate stored data: a repeated typo must not make itself legal.
“Observatory captain”, not every captain
Prefer supported, specific aliases. Short common names can assign unrelated content to one entity. Canonical IDs need not have invented aliases.
New choice first, old reason second
The expedition now uses an offline tablet. Its earlier paper-log decision remains useful as historical rationale, attached to its selected direct successor rather than competing as current advice.
| Synthetic input | Boundary | Expected observation |
|---|---|---|
| stored topic: feild_test | Config defines field_test | Build fails; previous index remains |
| user hint: feild_test | Query input | Warning; query continues |
| tablet → paper rationale | Final context packet | Direct successor first; predecessor marked historical |
CHECK
Decision attachment happens before packing. Evaluate successor adjacency again on the final packet, especially with tight budgets. Label membership is not full schema or file-presence validation; absent projection files are currently skipped.
Make the remaining boundaries visible
A reference pattern should distinguish working mechanisms from open experiments.
Observable, limited guarantees
Build-time label checks, warning-only hints, direct-successor attachment, whole-label fidelity matching, writer-enforced modes, rule-based control-plane exclusion, contiguous-phrase ranking, and expiring self-explaining traces have implementation counterparts. Pinecone itself publishes documentation, not an engine.
Permissions do not travel in Git
Owner-only modes need writer enforcement and checks after checkout or restore. Frozen-input advisories remain visible but do not fail the check or authorize automatic repair. Directory-symlink containment remains a known limitation.
A backup you have never restored is a guess
The trees that exist in only one place are compared against an external copy file by file, and a restore drill has been run. Refresh stays manual, so the honest claim is “verified as of the last check” — which is what the check reports.
Do not turn a proposal into a feature claim
Separate challenge runners and calibrated evidence labels remain follow-up work. Retrieval-intent parsing is measured against a ceiling it does not yet meet, and saying so is the point. Temporal checkpoint logic has synthetic coverage, not proof of a live checkpoint.
Read the workflow details and synthetic evaluation cases.