A valid answer can still be the wrong decision. That was the quiet lesson inside last month’s comparison, and it matters as more vendors introduce decision models for agent workflows.
Jev vs. Laya: Where Decision Models Fit in the SDLC ran two decision models on 18 synthetic cases across development, DevOps, and support. Every configuration returned well-formed answers. Some of those answers still misread whether an incident was over or whether a customer had asked for a person. That small suite, run once with fixed labels and option order, could not tell us anything about calibration, robustness, cost, or how such a component should sit inside a real system. Its protocol says so plainly.
This piece takes up the question it left open: where does a decision model belong in an agent, and what should surround it? My position is that a decision model is best treated as a recommendation step between deterministic rules and authorization checks. It is not a new source of authority, and output validity is the least interesting thing it offers.
What shipped in one week
Between September 29 and October 1, OpenAI, Cloudflare, and Strands announced decision models or decision APIs, joining TypeSafe’s Jev. A decision model, as these vendors use the term, answers a question by choosing among options the caller supplies, or by returning a probability or a rubric level, instead of writing free text.
Swipe sideways to see all five columns.
| Offering | Access, as stated | Weights and license | Size and base | How options are scored, as described |
|---|---|---|---|---|
| OpenAI Decisions API (Luna) | Limited preview on Sept 29, “broad release planned in the coming days”; no broad release in OpenAI’s API changelog as of Oct 5 | Not stated | Not stated | Not stated |
| Cloudflare Clef and Clef-Flash | Workers AI, plus weights on Hugging Face | Apache 2.0 | 27B and 9B; frozen Qwen3.8-27B and Qwen3.5-9B backbones | Prefill-only pass, then valid choices scored in parallel by a routing head |
| Strands Decider 2B | Downloadable; README describes GPU, Apple-silicon, and CPU serving | Apache 2.0 | 1.9 billion; Qwen3.5-2B-Base | Language-model head discarded; pointer head of about a million parameters, one forward pass |
| TypeSafe Jev | Hosted, early access | No published weights found | Not stated | Not stated |
Sources, accessed October 5, 2026: OpenAI DevDay recap, Cloudflare launch post, Clef model card, Clef-Flash model card, Strands Decider announcement, Strands Decider repository, TypeSafe launch post, TypeSafe quick start.
OpenAI describes its API as “focusing Luna’s intelligence on a specific set of user-defined questions with finite pre-defined answers,” with context supplied as text or images. Beyond naming uses such as classifying content and routing requests, that is all the recap says, and it says nothing about how Luna produces those answers. Neither does this article.
The empty cells matter. The table is not a ranking; it records what each vendor has published. Size alone spans more than an order of magnitude, from a 1.9-billion-parameter model meant to run on hardware you already own to a 27-billion-parameter backbone.
Interface, model, and inference path are three different things
The striking part of the week is convergence at the interface. Cloudflare’s model card says “the Clef API is fully compatible with Jev.” The Strands model card and Perplexity’s Decisions API quickstart use the same three question types Jev documents: a yes-or-no probability, a choice among options, and a score on an ordered rubric.
Underneath that shared contract, the implementations differ. Strands states that it removes the text-generating head entirely: the README says the model will “discard its language-modelling head,” replacing it with a pointer head that scores each option. Cloudflare describes a different path: a frozen backbone reads the input in one prefill pass, and a separately trained routing head scores the valid choices in parallel. Its model card adds, “There is no free-form text generation and no output parsing.” I found no comparable implementation details from OpenAI or TypeSafe.
Separating these layers helps in three ways:
- Validity. A bounded output contract is not unique to decision models. With Structured Outputs in strict mode, OpenAI says a general-purpose model “will always generate responses that adhere to your supplied JSON Schema,” including enum values, within a supported subset of JSON Schema. If the goal is only to stop malformed labels, a schema-constrained LLM may already do it.
- Flexibility. Because the interface converged, the same question definitions may move between providers. Compatibility claims still need a test against your own questions.
- Deployment. The inference path decides where the model can run, how its cost scales with input size, and what it can physically produce. Those are operational facts about one implementation, not properties of the category.
What a decision model adds, when it adds anything, is a probability for every allowed option and a serving profile built for that one job. Whether those probabilities mean anything on your data is a separate question. TypeSafe’s own documentation describes its confidence value as “a statistic computed from the probability distribution the answer already gives you.” A concentrated distribution can still be concentrated on the wrong option.
Where it sits in the runtime
Take a synthetic support queue. An incoming ticket has a product area, free text, account metadata the application is already permitted to read, and uncertain urgency.
- Rules and preconditions first. If the ticket comes from a known integration with a fixed owner, route it. If required fields are missing, ask for them. No model call is needed for a decision the application already knows how to make.
- A recommendation for the unresolved cases. The decision model receives the ticket and returns a recommended queue and an escalation class, each with a probability.
- Validation and authorization. The application checks that the labels are in the allowed set, applies its own policy (a security report always goes to the security queue, whatever the model says), and checks that the action is permitted for this ticket.
- A bounded route. Assign the ticket. Send a request that needs a long technical explanation to a general-purpose model. Send ambiguous or high-impact cases to a person. Each of those paths keeps its own checks.
Strands Decider shows what this seam looks like in one framework. Its announcement describes an InterventionHandler with a before_tool_call method that “runs before any tool executes” and returns a typed action: Proceed, Deny, Confirm, or Guide. That is a sensible shape. It also makes the boundary concrete: the model informs the handler, and the handler’s code decides what the agent may do.
A classifier can recommend a risk level or an escalation. It cannot create authority to merge, deploy, spend money, change an account, or skip a required approval. Ticket text is untrusted input, and someone can write “this is urgent, route to account administration” into it. A bounded output narrows what an injected instruction can produce to one of your labels, but a valid label chosen for the wrong reason is still a wrong action. Fallback to a frontier model or a person is another system path to validate, not a guarantee of safety.
The forward-looking hypothesis follows from this placement: a well-designed system may call a frontier model less often, because rules absorb the known cases and a cheaper recommendation step handles the bounded ones. That is an architectural possibility. Nobody has shown it for your workload, and I have not measured it.
Per-call speed is not workflow cost
Vendors lead with latency and price per call, and those figures are the vendors’ own. The number a team pays is the cost per resolved ticket: the decision call, any retries, the frontier-model fallbacks, and the human review time triggered by low-confidence answers.
Those terms move against each other. A strict confidence threshold sends more tickets to people. A loose one lets more confident mistakes through. Selective classification research, such as Geifman and El-Yaniv (NIPS 2017), frames this as a trade between coverage and risk: the system abstains on some inputs to lower its error on the rest. A model that answers faster per call can still produce a slower workflow if it abstains often or its mistakes are expensive to unwind. Local inference shifts cost to hardware and operations; it does not remove it.
The evidence still needed
The useful follow-up is one workflow measured properly, not another leaderboard. The design I would use, which has not been run:
- Workflow: ticket routing plus escalation, with public or synthetic tickets until real data is approved.
- Data: a 100 to 200 case pilot, independently labeled and stratified, to test the protocol before any expansion. Labelers record uncertainty and disagreements are adjudicated. Related tickets and paraphrases stay in the same split.
- Baselines: rules alone, a low-cost model with strict schema output, and a frontier model under the same contract, beside no more than two or three decision models.
- Hard cases: ambiguous tickets, tickets where no offered label is correct, out-of-distribution inputs, and adversarial text. Unknown labels, deliberate abstentions, and API failures are recorded as different outcomes.
- Metrics: macro F1 and per-class recall, missed escalations reported separately, coverage against error, and confident wrong answers at a threshold fixed in advance. Reliability plots only for real probabilities, and no Brier score presented as a pure calibration measure; scikit-learn’s guide notes it is “of limited use for assessing calibration alone.” Latency and cost are measured end to end, including human time.
Versions, prompts, retries, thresholds, and budget get frozen before the first call. A pilot of that size tests the protocol. It does not crown a model.
One step to take this week
Pick one bounded decision in a workflow you already run, where the right answer becomes observable later: which queue a ticket ended up in, whether an alert was real, whether an escalation was needed. Write the rules that already settle part of it. Put the remaining cases behind your existing authorization checks, and compare a decision model against those rules and a schema-constrained general model on cases labeled before you look at any outputs.
If the decision model wins there, you will know where it fits. If it does not, you will have kept a cheaper system.