Part 2 of 3 · Decision models in a real AI product
Where a decision model earns its place in an AI product (and where a regex still wins)
TL;DR
- Behind every message to our AI assistant sit several small decisions: which specialist answers, does the user want an image, a web search, a plan first, what kind of website? We tested Jev, a hosted "decision model", against the code that makes those decisions today.
- Jev won where the decision needs understanding the wording: catching image requests (92% vs 31% for our regex), web-search requests, trick questions, and tagging news articles (86% agreement with our editors).
- Our existing code won or tied where a rule already captures the answer: arXiv IDs, attached images, website build types.
- On the prompts our own team wrote, Jev's routing advantage shrank to nothing measurable. Averages hid that; slices showed it.
- We ended up with a four-question test for "should this decision use a decision model?", plus a map of where it fits in our product, including labeling, moderation and recommendations.
What a decision model is
A large language model writes answers. A decision model picks from a list you give it and tells you how sure it is:
LLM: "Which assistant should handle 'draw me a cat'?"
→ "Great question! I think the visual assistant would be best because…"
Jev: same question + options [general, coding, research, visual, build]
→ visual (0.99)
That's the whole interface. You get a choice, a yes/no probability or a score on a scale, always one of the options you allowed, with a confidence that means something. Jev (TypeSafe AI) is the best-known one; Laya is an open-source model of the same shape.
We measured three properties first, because everything else depends on them:
- Consistency: the same 100 routing questions, asked three times each. Answers moved by 0.004 on average, and no decision changed.
- Honest confidence: when Jev said ~90% on routing, it was right about 90% of the time (calibration error 0.034).
- Many questions for the price of one: nine questions in one call gave the same answers as nine separate calls (average difference 0.0025, 1 flip in 900), for 3.5× less money, because the message is billed once.
The contestants and the exam
For each decision we compared today's code (the regex, embedding router or LLM call that runs in production), Jev, and Laya off the shelf. The exam: ~300 messages, half written by our team with known answers, half synthetic, covering tricky phrasings and deliberate trick questions. Labels came from two independent LLM annotators, with disagreements settled by hand. Every contestant took the same final exam, and all tuning happened on a separate practice set.
Decision by decision
1. "Does the user want us to make an image / search the web?"
Before our assistant's AI runs, a set of regular expressions checks for common requests and shortcuts straight to the right tool.
| Intent | Regex: F1 (recall) | Jev: F1 (recall) |
|---|---|---|
| Make an image | 0.47 (0.31) | 0.73 (0.92) |
| Web search | 0.17 (0.09) | 0.88 (1.00) |
| 3D from a description / 3D from an attached image / compare papers by arXiv ID | 1.00 / 1.00 / 1.00 | 0.57 / 1.00 / 0.67 |
The regex never fires falsely on image requests, mostly because it barely fires at all. "Draw me a cat" and "design a poster" slip past it because they don't say "image". "Can you look up…" and "check online if…" slip past the web-search pattern. Jev reads the intent.
But where the answer is structural (is there an arXiv ID? an attached image?), a regex is perfect and instant, and Jev adds nothing. (These rows have only 1–3 positive examples each, so they're indicative.)
Jev also has a characteristic weakness: it reads markers literally. Every message starting with [image attached] got flagged as "the user
is asking about their documents". The lesson: send decision models clean, structured state (has_image: true), not text markers.
Verdict: a hybrid. Jev for the meaning-based intents, regex for the structural ones.
2. "Which assistant should handle this?"
Our router compares the message's embedding with example phrases for each assistant.
| Router today | Jev | |
|---|---|---|
| Overall accuracy | 76.3% | 88.2% (+11.8, 95% CI [+5.7, +18.5]) |
| On prompts our team wrote | 84.3% | 86.1% (no measurable difference) |
| On trick questions ("what's the best image model?") | 52.9% | 97.1% |
| Time per decision | ~100 ms | ~415 ms |
This is the result we'd most like other teams to see, because the average hid the story. Jev's big overall win comes almost entirely from phrasings our router's example sentences don't cover, above all questions about a topic that look like requests for it. On the kind of prompts our own team writes, the two are tied.
Jev also missed our speed budget for an in-chat decision (300 ms). But its honest confidence suggests a better design than replacing the router: a second opinion. Our router already knows when it's unsure (it was in its "weak match" band on 37% of messages). Send only those to Jev: most messages keep the fast path, and the confusing ones get the model that's good at confusing ones. Jev alone could decide 43% of messages at ≥ 98% accuracy.
3. "Does this research question need a plan first?"
While building this benchmark we discovered that our production classifier for this had never worked. It asked a small thinking model for a one-word answer with a 5-token budget. The model spent all 5 tokens thinking silently, returned an empty string, and the code read "empty" as "simple", every time. Fixing one flag made it work (F1 0.69). Jev scored 0.78, but with only 9 complex examples in the test set, we can't tell those apart statistically.
4. "Single page, multi-page site, or interactive app?"
The regex scored 94.5% (100% on our team's prompts); Jev 90.9%. Keep the regex. The rules here are crisp ("multi-page", "calculator", "dashboard with filters"), and a decision model can't beat a crisp rule on its home turf.
5. Tagging news articles by topic
There's no automated version of this today: editors pick a category. Against the editors' choices:
| Always guess "AI" | Laya | Jev | |
|---|---|---|---|
| Agreement with editors | 36% | 52% | 86% |
Jev could tag 74% of articles on its own at ≤ 5% disagreement, and most of its "disagreements" were defensible (the editors filed
"OpenAI launches GPT-5.2" under Technology; Jev said AI). Along the way we found our category field uses 16 different spellings for five topics
(ai, AI, AI & ML, AI & Machine Learning…).
6. RAG reranking
Covered in part 1: Jev ties a free local reranker at ordering passages, but as a filter it dropped ~90% of candidates while keeping the correct one every time.
7. Laya, off the shelf
It was 25× faster than Jev (11–35 ms on a laptop GPU) but never beat our existing code in any of these tests. The published checkpoint is a specialist for other tasks. Part 3 shows what happened when we fine-tuned it on Jev's answers for our decisions: it matched Jev.
The four-question test
Put a decision model on a decision when all four are true:
- The answer is one of a fixed list: a category, yes/no, low/medium/high.
- Deciding needs understanding the wording, not spotting a pattern. "Wants a picture?" needs understanding; "contains a YouTube link?" doesn't.
- A few hundred milliseconds is acceptable, because it runs in the background or replaces something slower.
- The alternative is a brittle regex or a slow, expensive LLM call.
How our decisions scored:
| Decision | 1 | 2 | 3 | 4 | Verdict |
|---|---|---|---|---|---|
| Image / web-search intent | ✅ | ✅ | ◐ | ✅ | Jev (hybrid) |
| YouTube / arXiv detection | ✅ | ❌ | – | ❌ | Regex |
| Routing | ✅ | ✅ | ❌ in-chat | ◐ | Jev as a second opinion on unsure cases |
| Build type | ✅ | ◐ | – | ❌ | Regex |
| News tagging | ✅ | ✅ | ✅ background | ✅ nothing today | Jev |
| RAG filtering | ✅ | ✅ | ◐ | ✅ | Jev filter |
What other teams use decision models for
- Routing: support tickets to queues, and even choosing which AI model handles a task (jev-router picks the cheapest capable model per coding task).
- Moderation and safety: spam, scams, toxicity, policy checks on user content.
- Agent guardrails: approve, confirm or block a tool call before it runs (LangChain).
- Replacing reliable-but-slow LLM classifiers: one team moved its classifiers to Jev yes/no questions because "the LLM was just the slow, expensive way to run them."
Where it fits in our product
Nands has about 800 pieces of content (articles, posts, news, events, projects, courses, jobs, hackathons) but, before launch, very little behaviour data: under a hundred users and a few hundred interactions. That shaped our map:
- Labeling (strong fit). One call per item, many questions at once: topic, sub-topics, level (beginner → advanced), format, audience, evergreen vs time-sensitive, region, and for events and courses "online?", "free?", "beginner-friendly?". Backfilling everything costs well under a dollar.
- Moderation (strong fit). Comments, threads, posts and DMs, including prompt injection aimed at our agents, since some agents read platform content. Honest confidence gives a clean three-way policy: allow / hide / send to a human.
- Recommendations (a component, not the engine). With so little behaviour data, "people like you also liked…" can't work yet, so recommendations have to start from what the content is about, which is exactly what the labels provide. A decision model also fits the filter step ("right level for this user?"). Finding candidates and ordering them still belongs to embeddings and scoring: calling a model for every item × every user doesn't scale, and ordering isn't its strength.
Caveats
- The test set is team-written plus synthetic prompts; real user traffic will differ. A human check of the labels is still pending.
- Some intents have very few positive examples in the test set, so treat their per-intent numbers as indicative.
- Latency numbers include the network round-trip from our dev machine.
Part 1 covers the RAG results in detail: Jev vs flashrank as a reranker, and Jev as a relevance filter. Part 3 is about distilling Jev into a free model we run ourselves.