Your assistant answers from the wrong page and you cannot tell whether that is the model, the prompt, or the fact that the answer got sliced in half at chunk 47. Every guide gives you a number — 512 with 50 overlap, 1000 with 200 — and none of them has seen your documents.
The number of results is not fixed. You set a context budget — the tokens you
are willing to paste into your model — and chunkforge retrieves
floor(budget ÷ chunk size) chunks. That matters more than it sounds: at a fixed
top-10, 128-token chunks return about 1,300 tokens and 8,192-token chunks return about 54,000,
so "bigger chunks retrieve more of the answer" is arithmetic rather than a finding. Holding
the price constant is what makes two settings comparable at all.
Beside every score is a chance level: exactly what picking chunks at random would score at that setting, in closed form. At 96 tokens a 4,096-token budget retrieves 42 chunks out of about 116, so a perfect score there is most of the way to what a coin would do. Without that column, that row looks identical to a genuinely good one.
And rank is never summed into a headline. Rank is not comparable across chunkings — bigger chunks mean fewer of them, so ranking first out of eleven is easier than ranking fourth out of a hundred and seventy-four. It is shown per query, always beside the number of chunks it came from.
The idea of scoring a chunking against marked spans of the source is not ours: it follows Smith and Troynikov's Evaluating Chunking Strategies for Retrieval (Chroma, July 2024). Their metrics need a corpus of pre-annotated relevant excerpts, which chunkforge deliberately does not ask you for, so this is a simpler measurement and it does not use their names for its numbers. Treat it as a lower bound: other passages in your documents may answer the query perfectly well and were never marked.
A static embedding model —
sentence-transformers/static-retrieval-mrl-en-v1 — whose entire graph is a lookup
table and a mean. There is no inference runtime and no WASM: an embedding is a weighted sum of
rows, which is why a whole corpus can be re-embedded between two frames of a slider drag.
It is not as good a retriever as the model you will actually deploy. On the MTEB retrieval benchmark it scores about 35 where bge-small scores about 52. What it is good at is telling you whether the passage was reachable — whether the chunking put it somewhere a retriever could find it — and that is what a chunk size decides.
Chunk size only means something relative to a tokenizer, so chunkforge loads the real
tokenizer.json for the embedding model you choose and counts in its tokens. Both
WordPiece and byte-level BPE are implemented exactly; the BPE path was checked token for token
against a reference implementation across an eleven-thousand-token document. A tokenizer that
cannot be reproduced exactly is refused by name rather than approximated.
localStorage; document text never does. A build gate checks the shipped
JavaScript for the storage APIs it must not contain.
huggingface.co as anonymous GETs of constant URLs, with credentials omitted.
That request necessarily reveals your IP address and which file you asked for. It reveals
nothing about your documents or your queries, and after the first run it is served from
cache — the tool then works with the network switched off entirely.
chunks.jsonl contains your document text
by design. Where it goes afterwards is up to you.
huggingface.co for two public model files, pinned to an immutable revision for
the scoring model.
feedback.benrichardson.dev. Nothing is
sent unless you open the feedback form and press Send; your files and data never are.
chunkforge is a browser tool for choosing a chunking strategy by measuring it on your own documents and your own failing queries, rather than by copying a number out of a blog post.
It is built and maintained by Ben Richardson. The rest of the catalogue of tools and sites lives at lab.benrichardson.dev.
Source: github.com/ben-gy/chunkforge.
sentence-transformers/static-retrieval-mrl-en-v1, Apache-2.0.
The sample handbook is fictional and was written for this tool. The company, the people and every policy in it are invented.