Evaluating search in Yank

Yank is a clipboard manager. The thing I care about with search is pretty simple: can I describe something I copied, get the right clip back and paste it without spending more time looking for it?

I want to try other local embedding models, maybe also a local decision model or a frontier LLM for reranking. So I built an eval to check whether those changes actually improve search. A model’s score on a general embedding benchmark doesn’t tell me how it’ll behave against the things people copy.

The first dataset was too small

I started with 136 clips and 56 queries. That was enough to compare the initial approaches and decide what to build, but as I worked on search and found more options like decision models and Google’s EmbeddingGemma, I needed something I could keep comparing them against.

Also, the embedding model only handles part of a search. A lot happens around it that can change which clip comes first.

Take “the slack link from yesterday”. Yank turns “yesterday” into a date filter before searching, so clips from other days are excluded. “Link” tells it to give URL clips a small boost. The remaining text, basically “slack”, goes through keyword and semantic search, and the two result lists get merged.

The order can still change after that. If the query mentions a colour, clips with nearby colours move up. Recent clips get a small bump so the newer one wins between otherwise similar matches. Pinned clips go to the top.

Changing any of this can change the result while keeping the same embedding model, so the eval runs the app’s whole search.

The same applies to indexing. Yank builds a description of each clip before embedding it. That includes the AI label when there is one, the clip type, the source app and the content trimmed to a reasonable length. Hex codes get colour names too, so #ff6b35 also reads as orange.

The eval uses the app’s code to build these descriptions so that it’s searching the same vectors the app would create.

Building a fake clipboard

The dataset now has 1,845 clips: the original 136, another 209 targets I wrote, and 1,500 filler clips.

The filler is mostly things like log lines, hashes, chat replies, URLs, phone numbers and code. A real clipboard has a lot of junk, I wanted the eval to make search deal with that too.

For most targets I wrote a query that shares a word with the clip and another that describes it without using the same words:

query:   undo last commit keep changes
clip:    git reset --soft HEAD~1

Only 108 clips have AI labels because most people won’t run the labeller. Search needs to work with whatever they copied. dataset.py uses a fixed seed so every model gets the same clipboard.

There are 559 queries. Of those, 218 are keyword queries and 287 are paraphrases. The rest cover short or vague searches, dates and categories.

I picked that mix myself. I haven’t checked whether it matches how people actually search in Yank because I’m not collecting their searches.

Deciding which answers count

Each query has a list of accepted clips, written before running any model. The metrics check whether those clips came back and where they ranked.

I need to pool the top results from a few search strategies and review the ones that haven’t been judged yet, following the approach used by TREC.

I also care about how much irrelevant stuff search returns. Finding the right clip somewhere in the results doesn’t help much if I have to read through a bunch of wrong ones to get there.

Checking the shortlists

Local search retrieves a vector shortlist and a BM25 shortlist, then merges them. Jev, which I added recently, reranks a BM25 shortlist.

I score these shortlists separately because a re-ranker can only reorder what it receives. If the right clip never made it into the shortlist, I had to work on retrieval. If it’s there but ends up low, I had to look at ranking. That’s what I had in mind while analyzing the results from the evals.

What I’m measuring

Hit@1 is the main metric because getting the clip first is what lets me paste and carry on. I split it by query kind too, so an improvement on keyword searches can’t hide a regression on paraphrases.

Alongside that I track Hit@10 and MRR@10. nDCG@10 helps with queries that have multiple accepted answers, and Recall@50 tells me how much of the accepted set search retrieved.

I dropped P@5 and P@10 because most queries have just one correct clip. Even if that clip comes first, P@5 is only 0.2, which doesn’t tell me much about whether the search worked.

Also found a bug in Recall@50 later. I’d capped the denominator at 50, so retrieving 50 out of 117 phone clips showed up as full recall. It now divides by the total number of correct clips, which makes that result 42.7%. There’s a regression test for it.

For latency I measure p50 and p95 after the model has warmed up, and indexing time in milliseconds per clip. I still need a separate cold-start measurement.

What I still need to fix

There’s no held-out test split or confidence intervals yet, so I can’t tell how much to trust small differences between models. When I add the split, queries for the same clip need to stay together. Otherwise I’d be tuning on one wording and testing on another wording for the same clip.

I also want a small private set of searches I’ve actually typed against my own clipboard, so I can check whether the synthetic dataset ranks models the same way real use does. At least it’ll reflect how one person uses Yank.

For cloud models I need pinned versions and cached responses, maybe also repeated runs to see how often the top result changes. Jev sends clip text off the machine, so that needs to be part of the comparison alongside cost.

I looked at Promptfoo’s retrieval evals, which could reuse these answer keys. The Rust harness already calls the real search though, and reviewing the dataset is more useful right now, so I’m leaving that for later.

For now I’m using this to find failures and compare bigger changes. I’m not changing the default over a two-point difference until I have the held-out results and some searches from my own clipboard.