We taught a free, local model to make our AI's decisions as well as a paid API

On a laptop, for $0.25 of API calls, in two and a half hours.

TL;DR

Our AI assistant makes a handful of small decisions on every message: which specialist should answer, does the user want an image, does this need a web search? A hosted "decision model" called Jev was the most accurate option we tested, but each call takes ~400 ms over the network.

So we used Jev as a teacher: it labeled 42,488 decisions on messages like our users', for $0.25. Then we fine-tuned a small open model, Laya (421M parameters), to copy it, on a laptop GPU in 2.5 hours.

The student matches the teacher on routing (89.1% vs 88.2%, no significant difference), beats it on tool-intent detection, and answers in ~17 ms instead of ~415 ms, for $0 per decision.

The catches: it's only that fast on a GPU (~570 ms on CPU), it changed a lot of its answers on tasks we didn't train it on, and it's weaker where we gave it little data.

The problem: a great decider that's too slow

Nands is a learning platform for science and tech, with an AI assistant at its core. Before the assistant writes a single word, the system makes several quick decisions about the message:

Routing: should the general assistant, the coding assistant, research, visual (images, figures, slides) or the website builder handle it?
Tool shortcuts: is this a request for an image? A web search? Something about the user's own uploaded documents?
Build type: if it's a website request, is it a single page, a multi-page site, or an interactive app?

Today those decisions are made by a mix of regular expressions, an embedding-based router and small LLM calls. In the first two parts of this series we benchmarked them against Jev, a new kind of model from TypeSafe AI that doesn't generate text at all. You give it a question and a fixed list of answers, and it returns the answer plus an honest probability. It was clearly the most accurate:

it caught 92% of image requests, where our regex caught 31% ("draw me a cat" slipped past the regex because it never says "image");
it routed 97% of trick questions correctly ("what's the best image-generation model?" is a question about images, not a request for one), vs 53% for our router.

But each Jev call is ~400 ms over the network. For a decision that sits in front of every chat message, that's too slow: we'd set ourselves a 300 ms budget, and Jev missed it.

We'd also tested Laya, an open-source model in the same family. It's small enough to run on our own hardware and answered in ~16 ms, but out of the box it was worse than our existing code on almost everything. Then we read its model card more carefully:

"This is a specialist… fine-tuned on four specific synthetic workflows [invoices, security incidents, customer service, agent traces]. Expect it to behave like the base checkpoint, or worse, on anything else."

And the detail that gave us the idea: that specialist had been trained on Jev's answers. Laya's authors had used Jev as a teacher for their tasks. Nothing stopped us doing the same for ours.

What "fine-tuning" and "distillation" actually mean

If you're new to this (we were), here's the short version.

A model like Laya is a very long list of numbers (weights), 421 million of them. Text goes in, gets multiplied through those numbers, and a score comes out for each possible answer. Everything the model "knows" lives in those numbers.

Fine-tuning means continuing to adjust those numbers using your examples. The loop is simple, even if it runs thousands of times:

1. show the model a small batch of examples            "which assistant should handle: 'build me a pomodoro timer'?"
2. let it answer                                       Laya: build 0.40 · coding 0.35 · general 0.25
3. compare with the answer we want                     Jev:  build 0.97 · coding 0.02 · general 0.01
4. measure how wrong it was (the "loss")               one number: big when far off, near 0 when it matches
5. nudge every weight slightly in the direction that would have made the loss smaller

Distillation is fine-tuning where the "answer we want" comes from a bigger, better model: the teacher. The student doesn't just learn the teacher's final pick; it learns the teacher's whole probability spread. "Build 0.55 / coding 0.45" tells the student this one is borderline in a way that just "build" never could, so the student learns both what to answer and how sure to be.

The analogy we kept coming back to: a smart new hire (the pre-trained model) shadowing your most experienced colleague (the teacher) on thousands of real tickets. They don't become a different person; they learn how this team decides.

Step 1: 4,015 messages that look like our users'

A student can only learn what it sees, so we needed a lot of realistic messages covering every decision. We had an LLM (gpt-oss:120b) write 4,015 messages across 16 kinds: coding questions, research requests, image requests without the word "image", website requests, questions about "the PDF I uploaded", news headlines for tagging, and deliberately messy ones (typos, Hinglish, one-word messages, three requests at once). Plus 300 trick questions that mention a topic without asking for the action.

Two rules we were strict about:

No labels at this stage. The kinds only steered variety. If a "general" message was really a research question, that didn't matter, because the teacher labels everything next.
No peeking at the exam. We had a frozen test set from earlier in the experiment. Any generated message too similar in meaning to a test item (cosine similarity ≥ 0.90 between embeddings) was thrown out, so the student could never be trained on near-copies of its own exam questions. One message was dropped.

Step 2: the teacher labels everything, for a quarter

Each message went to Jev once, with all its questions in a single call (routing, nine tool intents and build type, plus plan-first and news category where relevant). We'd measured earlier that asking Jev ten questions in one call gives the same answers as asking them one at a time, at about a third of the price, because the message is only billed once.

Jev calls	5,368
Labeled decisions	42,488
Input tokens	5.9 M
Cost	$0.25

That's the entire API bill for the training data.

Step 3: training on an 8 GB laptop GPU without melting it

We reused the training recipe the Laya authors published (their notebook targets two free Kaggle GPUs). It combines two things:

Soft cross-entropy: "make your probability spread look like the teacher's." This is classic distillation and does most of the work.
A small reinforcement-learning term: the model tries slightly randomised versions of its answer, and each is scored by a proper scoring rule (a reward that's only maximised by reporting honest probabilities). This keeps its confidence well calibrated.

We held back 10% of the examples to calibrate the model's confidence afterwards, and trained for 4 passes over the data.

The first attempt ran out of memory. Training needs far more memory than just using a model: besides the weights, the optimizer (AdamW) keeps two extra numbers for every one of the 421 million weights. On an 8 GB laptop GPU that doesn't fit. The fix was an 8-bit optimizer, which stores that bookkeeping at a quarter of the size. Peak memory dropped to 4.8 GB.

The second concern was the laptop itself. Two-plus hours at full GPU load is a good way to freeze a desktop or cook a laptop, so we added guard rails. They cost us about an hour, and we think it was worth it:

Risk	What we did
Desktop becomes unusable	Ran as a background service at the lowest CPU priority, capped at 4 of 24 cores, a 12 GB RAM ceiling, and a 30 ms pause after every batch
GPU overheats	Checked the temperature regularly: pause at 83 °C, resume at 75 °C. (It paused 157 times and peaked at 90 °C)
Laptop sleeps mid-run	Blocked sleep while the job ran
Crash or reboot loses hours of work	Saved the complete training state (weights, optimizer, position in the data, random seeds) every 10 minutes, atomically

We didn't trust the crash recovery until we'd tested it: train, "crash" at step 7, resume with the same command, and compare against a run that was never interrupted. The final weights were bit-for-bit identical.

Then it ran: 150.8 minutes, no errors, no restarts. The distill loss (how far the student's answers are from the teacher's) fell pass by pass:

Pass	1	2	3	4
Average distill loss	0.249	0.182	0.142	0.121

Results: same exam, same rules

The fine-tuned model took exactly the same frozen test set as every other contestant in this experiment, under the same rules (question wording and confidence settings chosen on a practice split, scores reported on the held-out split, once).

Decision	Our existing code	Jev (teacher)	Laya, off the shelf	Laya, fine-tuned
Routing accuracy	76.3%	88.2%	74.4%	89.1%
Routing, trick questions	52.9%	97.1%	41.2%	91.2%
Tool-intent detection (macro-F1)	0.675	0.711	0.350	0.812
… image requests / web-search requests (F1)	0.47 / 0.17	0.73 / 0.88	0.51 / 0.63	0.87 / 0.88
Website build type	94.5%	90.9%	78.2%	94.5%
News category (agreement with editors)	–	85.7%	51.6%	76.9%
Time per decision	0–100 ms	~415 ms	~16 ms	~17 ms (GPU)
Cost per decision	$0	~$0.00002	$0	$0

On routing the student is 0.9 points behind the teacher, 95% CI [−2.8, +4.7]: statistically no difference. It's 12.8 points ahead of our existing router (p = 0.001) and 14.7 ahead of off-the-shelf Laya. Its confidence stayed honest too (calibration error 0.04).

It beat its teacher on tool intents (0.81 vs 0.71). That surprised us, but it's plausible: the student learns Jev's average judgement over tens of thousands of examples, which smooths out some of Jev's one-off mistakes. We'd still be careful here, because some intents have only a handful of positive examples in the test set, and the exact-match difference vs Jev isn't statistically significant.

It's weaker on news tagging (77% vs 86%), and that one is easy to explain: news headlines were only 302 of the 42,488 training decisions. The student is only as good as the examples it sees for each task.

The honest part

1. "Fast" means "fast on a GPU"

On the laptop GPU it answers in 17 ms. On a CPU with 4 threads, which is what most application servers have, it took ~570 ms, slower than calling Jev. So this isn't a drop-in production replacement yet. The options are a small GPU host, or making it faster on CPU (exporting to ONNX, 8-bit quantisation, or the smaller 322M-parameter Laya). We haven't tested those yet, and until we do, the speed claim comes with an asterisk.

2. It changed a lot of answers on things we didn't teach it

Before training we set four targets. One was that on questions we never trained on (a sample from a public benchmark), fewer than 20% of the model's answers should change. 42% changed. We count that as a miss, because that's the point of writing targets down in advance.

Look closer, though, and the changes weren't harmful: accuracy on those untrained questions went up (29% → 37%). Our target measured change when it should have measured damage. We'd write that target differently next time ("accuracy on untrained tasks must not fall by more than X") and report the change rate only as a diagnostic.

3. The exam isn't real traffic (yet)

Our test set is ~150 prompts our team wrote plus ~150 synthetic ones, labeled by two independent LLM annotators with disagreements settled by hand. We haven't used real user messages, pending the right consent. A human spot-check of the labels is still pending. The comparison between contestants is fair, since everyone took the same exam, but the absolute numbers are provisional.

What we learned

A decision model you can't afford to call on every message can still teach one you can. Twenty-five cents of teacher labels bought a model that makes the same routing decisions locally.
Data volume per task is the lever. Where the student saw thousands of examples it matched or beat the teacher; where it saw 302, it didn't.
Pre-register your targets, and choose them carefully. Our drift target measured the wrong thing. We only know that because we wrote it down before seeing the result.
Laptop training is practical with guard rails. An 8-bit optimizer made it fit, and throttling, a temperature guard and tested resumable checkpoints made it safe. It cost an hour of extra runtime, and the laptop stayed usable throughout.
Hardware decides deployability. A 17 ms model on a GPU is a 570 ms model on a CPU. Plan where it will run before planning around its speed.

What's next

Make it fast on CPU (ONNX + quantisation), because that decides whether it can ship.
More training data for news tagging to close the gap with Jev.
A real-traffic test, once we can use real messages.
If those hold up: put it behind a feature flag as the router's second opinion on the messages our current router is unsure about.

Appendix: setup at a glance

Teacher	Jev `jev-1.13` (TypeSafe AI) via OpenRouter
Student	Laya base (`convaiinnovations/laya`, 421M, ModernBERT-large encoder + decision head), pinned revision `55cf4c4`
Training data	4,015 generated messages → 42,488 teacher-labeled decisions (routing, 9 intents, build type, plan-first, news category)
Recipe	Soft cross-entropy + proper-scoring RL (the Laya authors' RLCD recipe), 4 epochs, LR 2.5e-5 encoder / 1e-4 head, effective batch 32, bf16
Deviations from the authors' recipe	8-bit AdamW (memory); micro-batch 4 × accumulation 8 instead of 8 × 4; bf16 instead of fp16
Hardware	Laptop RTX 5070 (8 GB), 4.8 GB peak, 150.8 min with thermal throttling
Evaluation	Frozen test set from earlier in the experiment; wording and temperature chosen on a practice split; McNemar mid-p and paired bootstrap for significance
Total API cost	$0.25 for training labels (the whole experiment: $0.65)

Parts 1 and 2 of this series cover Jev in a RAG pipeline (reranking and filtering) and where decision models earn their place versus a plain regex.