A new model called Jev has recently taken the AI world by storm. It’s rapidly become one of the most popular models in the agentic space, and for good reason. In this post I’ll share my thoughts on why I think that is, where this really should fit in your agentic architecture.

To understand Jev’s behavior, I also experimented with creating a self-improving prompt injection detector, where the detector is a Jev call. I might talk more about that in a future post, but this post is based on a reading of the docs, and on trying it out with the detector.

What Jev Is

Unlike GPT, Claude etc., Jev is not an LLM. It takes text as an input, and returns strictly structured output, which at this point is either a boolean (true/false), one selection from a range of choices, or a score based on a rubric. It also outputs a confidence score, so you know when it thinks the evidence isn’t strong enough to justify its answer.

Jev vs LLMs

That might seem a bit underwhelming at first. After all, LLMs can also be made to output a true/false or choose from a range. There are a few critical differences though:

  • Jev provides calibrated decisions, which means that if it tells you something is true with a probability of 0.8, you can expect it’s correct 80% of the time. This confidence level is a useful signal in itself. With an LLM, there is no similar calibration. If it says it’s 80% sure, there is no way to assure that it is indeed 80% and not 60% or even 20% likely to be true.
  • And so, Jev can represent an “I don’t know” answer much more than LLMs can
  • The cost is a killer feature in itself: input tokens are much cheaper than frontier LLMs for a similar performance, while output tokens are free. In frontier LLMs, output tokens are way more expensive. Jev is also way faster than LLMs

Jev vs Traditional ML

In essence, Jev could be called a general-purpose classifier. If you’re familiar with the traditional ML space (logistic regression, SVMs, random forest classifiers), this might sound like old wine in a new bottle, but it isn’t. Unlike traditional ML algorithms, Jev doesn’t need to be retrained for every new classification problem, and it also doesn’t need extensive feature engineering to work. You can pass it a text document, along with instructions on what you want classified, and it does it.

So where does this fit?

This is where the tech meets what business needs. There are a lot of use cases for agentic AI that are really Jev-shaped problems. Any time you need to choose between a range of discrete options, Jev needs to be in the set of models to consider for the job. For example:

  • detect if an incoming request is malicious (prompt/SQL injection)
  • route a support ticket to the right queue (billing, technical, abuse) before an LLM ever touches it
  • decide whether a document chunk retrieved for RAG is actually relevant to the query, instead of trusting the retriever’s similarity score
  • flag whether a generated response violates a policy (PII leakage, off-brand tone, disallowed medical/legal claims) before it reaches a user
  • score a claim or application against a fixed rubric (fraud risk, underwriting eligibility) where the criteria are stable and well understood
  • gate whether a multi-step agent should proceed, retry, or hand off to a human, based on how confident it is in the last step’s output

Today, these are often done by LLMs with structured tool calls. This has all the problems mentioned above (overconfidence, no way to signal uncertainty).

These things always come in a package though; there’s always a trade-off. The most obvious one: it’s not a text generator. So code generation, content generation, summarisation and review (unless it’s a rubric scoring exercise) are out of the window.

Another one is the lack of an explanation. Jev will tell you how sure it is, but it won’t tell you why. That’s one disadvantage of not being able to generate text.

As an aside, reasoning traces produced by LLMs are not reliable either. They do not indicate what the model is reasoning about. So they shouldn’t be used to explain something if the stakes matter, such as for legal or regulatory reasoning.

Coming back to Jev, this poses a problem in two ways: firstly, using it anywhere explainability matters in a legal/regulatory sense, is not advisable.

Secondly, it poses a challenge when trying to figure out how to improve its responses.

How to eval a Jev agent

As Jev is not a text generator but a classification model, I think the traditional data science evaluation metrics make a lot more sense here. Metrics such as ROC/AUC or PRC, as well as Recall and Precision, may be used.

Even more interesting though, is how you might go about improving the output of these agents. As there are no LLM traces, we do not even have an unreliable narrator telling us what might have caused the model to give a wrong answer for a particular question.

One option I’ve been thinking about, though I need to try it out, is as follows: first ask a traditional LLM to “list 5 reasons why a classifier given this prompt might fail to correctly answer this question”. This is a single LLM call, which would be cheap. When it lists those reasons, pass an incorrect question/answer pair to Jev along with these options, and ask it to score which of those is the most likely reason. An LLM can then be asked to correct that by rewriting the prompt to a newer version that fixes this. This approach is at the core of my self-improving prompt injection experiment.

Closing Thoughts

Jev isn’t a replacement for LLMs, and it’s important to understand that. It’s a narrower tool for a narrower job: the parts of an agentic pipeline where the question is really “which of these options, and how sure are you”, not “generate something novel.” Routing, gating, scoring, and guardrail decisions all fall into that category, and they show up constantly in production agent systems, often hidden inside a prompt asking an LLM to output JSON.

The architectural change is more interesting. Instead of reaching for the same frontier model or even a smaller LLM for every decision in a pipeline, it’s worth asking which of those decisions are actually classification problems wearing an LLM costume. Where the answer is yes, a calibrated, cheap, fast classifier like Jev is very likely the better fit, even if it means giving up the explanation that comes for free with a text response.

That trade-off is the most important thing to come out of the Jev moment: you lose the “why,” and you need a different playbook for improving the model when it gets things wrong. I don’t think that playbook is fully solved yet, but the self-improving prompt injection detector I mentioned earlier suggests there’s a workable path, using a cheap LLM call as the “why” generator, and Jev’s calibrated scoring to pick between candidate explanations. More on that in a future post.