I wrote 14 cases and asked the same three questions of three coI wrote 14 cases and asked the same three questions of three contenders: Jev 1.13, Gemini 2.5 Flash Lite, and GPT-5.6 Luna, all through OpenRouter in late September 2026. The LLMs got the same question wording as Jev, in JSON mode at temperature 0. Some cases were in English, some in Spanish, some in Roman Urdu, and one was a prompt-injection attempt.
On picking the right alert, all three got 93%. On the five cases where the newest alert was the wrong answer, all three got 5 out of 5. Our existing rule got 0 out of 5.
So Jev wasn’t smarter than the LLMsntenders: Jev 1.13, Gemini 2.5 Flash Lite, and GPT-5.6 Luna, all through OpenRouter in late September 2026. The LLMs got the same question wording as Jev, in JSON mode at temperature 0. Some cases were in English, some in Spanish, some in Roman Urdu, and one was a prompt-injection attempt.
On picking the right alert, all three got 93%. On the five cases where the newest alert was the wrong answer, all three got 5 out of 5. Our existing rule got 0 out of 5.
So Jev wasn’t smarter than the LLMsI wrote 14 cases and asked the same three questions of three contenders: Jev 1.13, Gemini 2.5 Flash Lite, and GPT-5.6 Luna, all through OpenRouter in late September 2026. The LLMs got the same question wording as Jev, in JSON mode at temperature 0. Some cases were in English, some in Spanish, some in Roman Urdu, and one was a prompt-injection attempt.
On picking the right alert, all three got 93%. On the five cases where the newest alert was the wrong answer, all three got 5 out of 5. Our existing rule got 0 out of 5.
So Jev wasn’t smarter than the LLMsI wrote 14 cases and asked the same three questions of three contenders: Jev 1.13, Gemini 2.5 Flash Lite, and GPT-5.6 Luna, all through OpenRouter in late September 2026. The LLMs got the same question wording as Jev, in JSON mode at temperature 0. Some cases were in English, some in Spanish, some in Roman Urdu, and one was a prompt-injection attempt.
On picking the right alert, all three got 93%. On the five cases where the newest alert was the wrong answer, all three got 5 out of 5. Our existing rule got 0 out of 5.
So Jev wasn’t smarter than the LLMs
