Jev tied our reranker. Then it did something a reranker can't.

Tested September 2026 · Jev jev-1.13 (TypeSafe AI) via OpenRouter · flashrank 0.2.10 (ms-marco-MiniLM-L-12-v2)

The short version

• We tested Jev, a new type of AI model that makes quick yes/no decisions, inside the part of our AI assistant that picks which documents the AI reads.
• At sorting documents, it tied a free, open-source tool. Both put the right passage first in about 8 out of 10 questions, up from 6.
• But Jev can also say "this doesn't help". Used as a filter, it cleared out ~90% of the clutter and never threw away the right passage.
• And when the answer wasn't in the documents, it said so, in 20 out of 22 cases, instead of handing the AI irrelevant text to guess from.
• Our takeaway: sort with a free tool, filter with Jev. About $0.90 per 1,000 questions.

---

The problem: the AI can only answer from what it reads

When you ask an AI assistant about your own documents, it doesn't read all of them. It works in steps:

1. Search grabs about 50 passages that look related to your question.
2. Sort reads them more carefully and puts the most useful ones on top. This step is usually called a reranker.
3. Filter (the step we added) throws away passages that don't actually help.
4. Answer: the AI reads only the top few and writes its reply.

The AI never sees what gets left behind. If the passage with the answer ends up in position 7, the AI either says it doesn't know or, worse, answers confidently from the wrong text. So what reaches the AI matters as much as how smart the AI is.

What is Jev?

Most AI models you've heard of (ChatGPT, Claude, Gemini) write answers. Jev, from TypeSafe AI, is a decision model: it doesn't write, it decides. You give it a question and a fixed set of options, and it tells you which option applies and how sure it is.

For us, the question was simple: "Does this passage help answer the user's question? Yes or no?" Jev answers with a percentage, like 3% or 99%.

That percentage is the interesting part. In our other tests (covered in part 2 of this series) Jev's confidence behaved like a good weather forecast: when it said 90%, it was right about 9 times in 10. That means you can act on the number, for example "throw away anything below 50%".

🔧 For engineers: Jev takes a JSON state plus typed questions (yes/no, choice, or score) and returns a probability per option. The yes/no question we used:

{ "state": { "question": "…", "passage": "…" },
  "questions": { "helps": { "type": "noul",
    "instructions": "Does this passage contain information that directly helps answer the question?" } } }
→ { "helps": 0.03 }

The output can only ever be one of the options you defined, so there's no parsing and no malformed responses.

How we tested it

We took 62 real test questions where we already knew which passage holds the answer, gave every contestant exactly the same 50 search results per question, and checked where the right passage ended up.

The contestants:

• Search alone: the order the search step returned (our starting point)
• flashrank: a small, free, open-source sorting tool that runs on an ordinary server
• Jev: a "does this help?" percentage for every passage
• Laya: a free, open-source model from the same family as Jev

🔧 For engineers: 20 questions over our platform's articles (expected document known) and 42 over research papers in a test account's document store (expected passage known; these questions were auto-generated from the papers, and a few are a little odd). Search is our production hybrid retrieval (BM25 + embeddings, fused with RRF), top 50 per question. The right passage was in those 50 for 58 of 62 questions, which is the ceiling for any reranker. flashrank is a cross-encoder running on CPU.

One question, four answers

"What is the title of the paper by R. Jozsa cited as reference [69]?"

Search result	What it actually contains	Jev: "does this help?"
#1	references [32]–[33] from the same paper	3%, thrown away
#2	references [61]–[62], so close	3%, thrown away
#3	"[69] R. Jozsa, Fidelity for mix…"	99%, kept
the other 47		all thrown away

Search did what search does: it found passages that look like the answer (the same reference list, the same formatting). Both flashrank and Jev moved the right passage to the top. But Jev also cleared away the other 49, so the AI would read exactly the right text and nothing else.

Finding 1: at sorting, it's a draw

What to notice: the blue bars (flashrank and three ways of asking Jev) are nearly level, and all of them are far ahead of search alone (gray).

Both flashrank and Jev put the right passage first in about 8 out of 10 questions, up from about 6 with search alone. The gap between them was too small to call a winner. The open-source Laya, used off the shelf, did worse than doing nothing.

	Search alone	flashrank	Jev	Laya
Right passage first	58%	81%	77%	35%
Right passage in what the AI reads (top 5)	89%	92%	94%	60%
Time per question	–	1.9 s	0.66 s	1.4 s
Cost per 1,000 questions	free	free	~$0.90	free

So if all you need is sorting, a free tool does the job.

🔧 For engineers: we asked Jev three ways: yes/no per passage (MRR 0.847), a graded 4-level scale (0.868), and all 50 passages in a single call (0.845). flashrank scored 0.849 and search alone 0.709. Jev vs flashrank: −0.002 (95% CI −0.08 to +0.08), a tie; both vs search alone about +0.14 (CI +0.05 to +0.23). The table shows the single-call setup, the practical one at 0.66 s and $0.0009 per question. Laya's published checkpoint is a specialist for other tasks; part 3 of this series fine-tunes it.

Finding 2: Jev works as a bouncer

A sorting tool can only order passages. Because Jev gives a confidence for each one, you can also set a bar and throw out everything below it, like a bouncer checking each passage at the door.

What to notice: Jev's lines (blue, orange, green) stay at the top even at the far left, where only a handful of passages are kept. Laya (yellow) loses the right passage as soon as it starts cutting.

Jev threw away about 90% of the passages and kept the right one every single time (in all 58 questions where search had found it).

When we combined the two tools, flashrank sorting first and Jev filtering after, the AI got the same or slightly better results while reading about half as much:

	flashrank alone	flashrank, then Jev
Right passage first	81%	82%
Right passage in what the AI reads	92%	94%
Passages the AI reads (typical)	5	2–3
Right passage wrongly thrown away	–	0 of 58

Less to read means fewer distractions for the AI, and a shorter, cheaper request.

🔧 For engineers: thresholds were P ≥ 0.5 (yes/no), "partially useful" or better on the 4-level scale, or ≥ 1.5 of 3 in the single call. All three kept 4.6–5.9 of 50 passages with 100% survival; Laya at the same threshold kept 17.3 with 84% survival. This matches the open-source jev-reranker result on a public benchmark (92% dropped, every relevant passage kept).

Finding 3: it knows when to say "not found"

This was the most useful result. If the bouncer lets nothing through, the honest reply isn't an answer at all: it's "I couldn't find that in your documents."

To test it, we wrote 22 questions our documents genuinely can't answer:

• 11 near misses: on-topic for one of the documents, but asking for something it doesn't contain. "How many hectares burned in the 2025 fire season according to the wildfire satellite paper?" This is the realistic, hard case.
• 11 off-topic: "How many moons does Neptune have?"

	Jev let nothing through
Off-topic questions	11 out of 11
Near misses	9–10 out of 11
Answerable questions (where search found the answer)	wrongly emptied 0 out of 58

Two answerable questions did come back empty, and in both cases search had never found the answer to begin with, so "not found" was the right reply. Without the filter, the AI would have received five loosely related passages for each of those questions and had to decide by itself that none of them answered it.

🔧 For engineers: unanswerability was verified by searching all 572 chunks of the corpus for each question's key terms. 6 of 28 drafted questions were excluded because a term appeared somewhere. Near misses: 9/11 empty with yes/no, 10/11 with the single-call setup. Same thresholds as Finding 2, not tuned on these questions.

What other teams have found

Others report big gains from using Jev for sorting, but almost all of them start from basic keyword search, which leaves a lot of room to improve. Our search was already decent: it put the right passage first 58% of the time before any sorting. How much a sorting tool helps depends on how good your search already is. How much a filter helps doesn't.

🔧 For engineers:

Source	Starting retrieval	Reported
MindStudio	BM25	top-1 21% → 54%
TypeSafe cookbook (legal search)	BM25, 30 candidates	top-1 5% → 18%, top-10 38% → 62%
jev-reranker	hybrid	nDCG@10 0.83 → 0.97; filter dropped 92%, kept all positives
refix.ai	–	relevance, contradiction, injection and citation checks as separate questions

Takeaways

If you build AI products:

1. Sort with a free tool like flashrank. It's as good as Jev at ordering, and it runs on your own server.
2. Filter with Jev. One call per question clears ~90% of the clutter without losing the answer, for about $0.90 per 1,000 questions.
3. Use the empty result. When nothing passes the filter, say "not found" instead of letting the AI guess.

If you're choosing or buying AI tools: The most valuable thing an AI can say is sometimes "I don't know". Ask vendors how their system behaves when the answer isn't there, and test it yourself with questions you know it can't answer.

What we'll test next: whether the filter improves the AI's final answers, not just what it reads; screening documents for hidden instructions aimed at the AI (a known risk with uploaded files and web pages); and checking each claim in an answer against its source.

What this test doesn't prove

• It measures what the AI reads, not what it writes. Better passages should mean better answers, but we haven't measured that yet.
• It's a small test: 62 answerable and 22 unanswerable questions. Small differences (a few points) should be read as ties.
• One known answer per question. Some passages Jev threw away might also have contained useful information.
• Some tests came after the first round. The single-call setup and the "not found" questions were added later, but we didn't tune any settings on them.

---

Part 2 looks at where decision models earn their place across the rest of an AI product, and where a simple rule still wins. Part 3 is about training the free, open-source model to match Jev on your own tasks.