I wanted permanent memory for the assistants I use while building things. Just something that could keep facts, project decisions and old conversations, then bring back the relevant part when I needed it. I didn’t want another chat export or a folder I had to search myself.

The existing memory products probably work fine. My issue is that memories are personal, and even a local product can change in a way I don’t like. I could also miss an important update and leak something sensitive because of it. I know this is slightly paranoid and every one of those problems has a separate solution, but I still want to completely understand a system that stores this much about me.

The main constraint was resource usage. It had to run on lower-end hardware, or at least stay out of the way on a normal machine. I didn’t want Ollama keeping a multi-gigabyte model in memory just so an assistant could recall what I worked on last week. So I put a hard 1 GB limit on any model running locally. I would only replace the small stack if the benchmarks showed there was no realistic way to get close to the better memory systems.

That constraint was reasonable, the first architecture I built around it wasn’t.

The first version was too eager to be clever

The first pipeline made a language model read each conversation and extract small facts from it. This was supposed to keep the vault clean, one note for a preference, another for a project decision, then an update when something changed.

The extractor was actually a lossy gate in front of storage.

• If it missed a detail, the detail was gone.
• If the small model got an update wrong, the system saved that mistake with a lot of confidence. It behaved like a very confident kid.

A better model helped, but then I needed Ollama and a much larger memory footprint again.

I had five things to fix:

1. Ingestion had to happen automatically without dropping facts.
2. Dates and updates needed proper version semantics.
3. Search needed a real hybrid ranker instead of a loose mix of scores.
4. Every result needed enough information for me, or another agent, to reproduce it.
5. The benchmark had to run against the complete datasets.

I tried finding a better extractor first. One of the options was a local Gemma3 1B model, the installed artifact was 815 MB so it fit under the limit. It still failed the representative extraction and reader checks. Being small enough to download doesn’t make a model good enough to decide what my future self is allowed to remember. Honestly I expected this one to work.

So extraction stopped being responsible for memory.

Decision 1: store the conversation before interpreting it

The production path now stores timestamped raw turns automatically and without changing them. Each turn gets a stable identity derived from its content, so replaying a conversation doesn’t create duplicates. Long turns are split into overlapping chunks for search, the original text still stays intact.

Semantic extraction is optional enrichment now. If someone wants compact notes, an installed Ollama model below 1 GB can generate them, but enrichment can’t block or replace raw storage.

This is less sophisticated than having an AI maintain a personal knowledge graph. That is fine, I wanted the storage path to be simple so it would be harder to lose information.

Dates follow the same rule. Every turn keeps its observation timestamp, and the existing ADD, UPDATE and DELETE operations keep dated history. Relative phrases are resolved from an explicit as_of value instead of asking an embedding model to infer time.

Deterministic code handles anything the system can calculate, models only handle the fuzzy part. Honestly I should have started with this rule.

Decision 2: let different search methods disagree

Semantic search helps when the question and the memory use different words. Keyword search is better for names, products and exact phrases. I had already learned from Yank that making one retrieval method do everything doesn’t work very well.

I kept all-MiniLM-L6-v2 for embeddings and added BM25 over the same chunks. They each produce a ranked candidate list. Reciprocal Rank Fusion combines the positions instead of pretending a cosine score and a BM25 score are directly comparable, then cross-encoder/ms-marco-MiniLM-L-6-v2 reranks the bounded pool against the complete question. The two model artifacts are 91.6 MB and 91.8 MB.

The process used around 690 MB of resident memory when I sampled it during the full LongMemEval retrieval run. That is still a lot, but better than keeping a 4B chat model loaded all day.

I tested BGE-small locally too. It improved Hit@20 by about two points on one LoCoMo conversation, but the Recall@15 gain wasn’t enough to justify making the default larger and slower. A small improvement on one slice wasn’t enough reason to switch.

Then the benchmark told me exactly what I wanted to hear

I ran two long-term memory benchmarks:

• LongMemEval-S, all 500 questions, including 30 that should be rejected as unanswerable.
• LoCoMo, all 1,540 non-adversarial questions, with evidence labels for 1,536 of them.

I generated the complete run dataset with gpt-5.6-sol btw.

The retrieval numbers looked excellent.

Supermemory reports 95% Recall@15 on LongMemEval. My result was 98.20% with a much smaller local model. It made a good chart and told me what I wanted to hear.

The protocols were different. My number only covered the 470 answerable questions and excluded all 30 abstention cases. Supermemory describes session aggregation across all 500 questions. The comparison wasn’t fair and it didn’t show that my system had beaten theirs.

I had also missed the more important issue: retrieval and answering are separate tests. The system could find a relevant record, but I didn’t know if the returned context was complete enough to answer the actual question. I plan to put these memories into my regular assistant prompts, so that gap can directly affect both the answer and how much time the assistant saves me.

Decision 3: finish the comparison, but label the protocol

I had already saved the retrieved memories for every question, so I built the next benchmark to resume after interruptions. GPT-5.4 answered each question from those memories and GPT-5.4-mini graded the answer. Both models ran remotely, which kept the local ai-memory setup unchanged.

This isn’t an official evaluation protocol. I just wanted to know whether the local system retrieves enough context for a smarter LLM to work with.

I recorded the models, reasoning settings, dataset and retrieval hashes, exact question order, source hashes, package versions, Git state and every per-question answer. Writes are atomic. If the configuration changes, resume fails instead of quietly combining two different runs.

Then I ran all 2,040 questions.

• LongMemEval-S: ai-memory scored 85.40% (427/500), against Supermemory’s published 95.00%. The gap is 9.60 percentage points.
• LoCoMo: ai-memory scored 85.58% (1,318/1,540), against Zep’s published 94.70%. The gap is 9.12 points.

This still isn’t a controlled comparison, and honestly it may not be fully accurate. The published systems use different readers, judges, context budgets and retrieval depths. Mem0 reports 94.8% on LongMemEval and 92.5% on LoCoMo, Hindsight reports 94.6% and 92.0%, and Zep reports 90.2% and 94.7%. I treat these as product-level reference points, not a clean ranking of retrievers.

Still, a nine point gap is too large to explain away with protocol differences.

The useful result was inside the failures

LongMemEval single-session user questions scored 95.71%, and knowledge updates reached 96.15%. The system can retrieve a specific personal fact and usually finds the current version of it.

Multi-session questions dropped to 77.44%. These often ask for a total assembled from separate events, and the failures were very literal: four model kits instead of five, three festivals instead of four, one subscription instead of two. The reader could do the arithmetic. It just never received every item it needed to count. These are all examples from the dataset.

LoCoMo had the same kind of split. Single-hop questions scored 89.30% and temporal questions scored 87.54%, while open-domain questions fell to 54.17%. The overall evidence Recall@20 was already 78.59%, so I probably shouldn’t have been surprised by the end-to-end ceiling.

This answered the model question too. GPT-5.4 was already reading the retrieved memories in this run, so replacing the 92 MB embedder with a slightly larger local model can’t create facts that never reach the final context.

The next work is session aggregation, neighbouring-turn aggregation and multi-evidence coverage. The retrieval benchmark also needs to reward finding every required fact, not just any one of them.

I will reconsider the model after those changes plateau, because switching now would cost more and keep the same blind spot.

What I am keeping

The local stack stays:

• Lossless raw-turn storage by default.
• MiniLM embeddings with BM25, RRF and a MiniLM cross-encoder.
• Deterministic timestamps and version history.
• Optional sub-1 GB enrichment, never a storage dependency.
• The host assistant as the reader instead of a small on-device chat model.

I am keeping the benchmark setup too. Both full runs resume without recomputing completed questions, all 2,040 rows have unique IDs and judgments, and the repository includes the exact per-question artifacts. The source and benchmark reports are in the ai-memory repository.

I started this because I wanted to know if I needed a better local model. The retrieval score almost let me stop at no. The full run gave me a less flattering answer, the small models are doing their part but the system still loses the shape of information spread across a conversation.

I prefer this problem to just downloading a bigger model, because now I know what to work on next.