Test 1: Jev against normal LLMs
I wrote 14 cases and asked the same three questions of three contenders: Jev 1.13, Gemini 2.5 Flash Lite, and GPT-5.6 Luna, all through OpenRouter in late September 2026. The LLMs got the same question wording as Jev, in JSON mode at temperature 0. Some cases were in English, some in Spanish, some in Roman Urdu, and one was a prompt-injection attempt.
On picking the right alert, all three got 93%. On the five cases where the newest alert was the wrong answer, all three got 5 out of 5. Our existing rule got 0 out of 5.
So Jev wasn’t smarter than the LLMs. It was the same accuracy, but:
Median latency
Cost per 10,000 messages
Jev 1.13
~350ms
$0.25
Gemini 2.5 Flash Lite
~640–1000ms
$0.41
GPT-5.6 Luna
~2.1–2.6s
$1.20–1.30
Latency is measured end to end from my machine through OpenRouter, network included, over two runs. That’s why Jev shows ~350ms rather than TypeSafe’s 100ms. The costs are what OpenRouter charged.
Because this runs before a reply can even be written, the latency matters more than the cost. Two seconds of GPT on every ambiguous message is noticeable. A third of a second isn’t.
Jev did worse on a different question I also tested: “can this be answered from the alert?” It scored 71% there against GPT’s 93–100%. Some of that was my fault. I’d labelled “ok 👍” as answerable, which doesn’t really mean anything. But even allowing for that, GPT was better at it, so that job stays with the reply model. GPT’s own score moved between 93% and 100% across two runs, which tells you how much to read into small differences at this sample size.
https://www.youtube.com/watch?v=fcbH8Su-YS4
Embedded video: