# Jev and Laya: small matched synthetic evaluation ## Registration and purpose This protocol and `cases.json` are fixed before either model is run on this suite. The suite contains 18 newly authored synthetic cases: six development/review, six DevOps/incident, and six technical-support cases. It contains no private customer, employee, repository, or production data. The previously observed refund/cancellation example is excluded from this primary suite because its result is already known. Each domain uses an identical question set across its six cases. Every case asks one Choice, one Noul, and one three-level Score question, giving 54 question answers per model. Labels and rationales are set before execution. There are three positive and three negative Noul labels per domain and two cases at each Score level per domain. Every domain includes an explicit insufficient-evidence Choice case. Resolved historical failures and explicit negation appear in the suite. These are short, deliberately explicit examples to check structured decisions across familiar workflows. They are not a representative customer dataset, a blinded benchmark, or a production deployment. Report counts and individual disagreements; do not describe them as general model accuracy, proof of reliability, or a leaderboard. ## Models and execution Run all 18 cases once on each of these configurations: - Jev: pinned `typesafe/jev-1.13`, using the official OpenRouter `/api/alpha/decisions` endpoint and the existing capped key authorized by the owner. Record returned version metadata where provided. - Laya English checkpoint: explicitly select the English checkpoint with `laya.Agent` to hold the checkpoint fixed. This is the router default for these English cases and uses the same underlying `system_one`; this evaluation does not invoke or test the router API path. Record package, source, and model revision, runtime, device, and selected checkpoint. - Laya typed-decisions: run the complete same suite as a separately reported configuration at the same model-repository revision. Do not substitute its better answers into the English column. Use exactly the same case state, questions, instructions, criteria, and criterion order for every configuration. Only the endpoint/package envelope may differ. Capture the actual submitted envelope and raw returned results without any credentials. Do not train, tune prompts, select a checkpoint after observing answers, or retry an answer to improve it. A transport/API failure remains visible as unavailable; any separately authorized rerun must be identified as a rerun and never silently replace the primary record. Run one case per request. Keep all three question types together in that request. Record elapsed times only as operational context. Do not rank latency: the local CPU runtime and hosted endpoint have different execution and networking conditions. ## Scoring fixed before execution - **Choice:** compare the returned label with the exact expected label. Count matches out of six per domain and out of 18 overall. If the interface provides only a distribution, select the maximum-probability label and document the extraction rule; a tie is unresolved unless the interface itself supplies a chosen label. - **Noul:** treat values greater than or equal to 0.5 as `true`, otherwise `false`. Compare with the expected Boolean and show the raw value alongside the label. Report matches out of six per domain and 18 overall. These values are not correctness probabilities and are not calibration evidence. - **Score:** ordered criteria have positions 0, 1, and 2. Map a valid returned value to its nearest integer with exact .5 ties rounded upward (`floor(value + 0.5)`). Count matches against the expected position. Also report mean absolute error using the unrounded returned score and the expected integer. Do not silently clamp out-of-range values or coerce malformed responses; show them as invalid. Report valid-answer denominators for MAE and include all expected questions in the count summary so failures are not hidden. A missing, malformed, or unavailable answer is separately flagged and contributes no successful match. Do not combine Choice, Noul, and Score into one ranking statistic: their meanings differ. ## What the labels mean The `risk` field is merely the common question identifier. It does not ask the model to infer an unsupported future outcome. Its explicitly observable domain-specific meanings are: a required release check is currently failing; customer-facing requests are currently failing; or the customer explicitly asks for human/manager takeover. The Score question measures current workflow blockage, not subjective business severity: 0 means no workflow blocked; 1 means a blocked workflow with an explicitly working workaround; 2 means a blocked workflow without a working workaround. Scary language, yesterday's failure, or a request for a human does not itself change this score. Choice asks for the established change type, established incident cause, or stated support category. `unknown` is correct when the supplied facts do not establish one. The development/review scope here is change routing, not source-code review performance. ## Reporting and limits Publish a row-level result table, per-domain counts, configuration identifiers, invalid/unavailable answers, and a small selection of both agreements and disagreements. Do not feature only one support failure. Explain that the suite is synthetic, small, English-only, and intentionally explicit; it does not measure multilingual handling, adversarial robustness, production triage performance, cost efficiency, calibration, or safety of autonomous actions. Suggested SDLC/DevOps/support integrations remain proposed applications, not deployments tested here.