Jev: judging,
not calling tools.

A classifier, not an agent

Most of the time an LLM is one of three things. It continues the text. It calls a tool. Or it writes a completion and you hope the JSON comes out right.

Jev is a fourth interface. You do not ask it to plan, act, or keep talking. You hand it some state and a fixed menu of judgments, and it returns probabilities over those outcomes. Refund or not. Support, contradict, or unresolved. How far along this evidence says a task is. It is closer to a high-speed classifier than to an agent.

We wired that interface into Umwelten and ran the same sixty synthetic cases through two backends: TypeSafe’s Jev, which answers natively on a Decisions API, and DeepSeek Flash, which is still a generative model forced into the same typed shape. You can get the same kind of answer either way. The difference is speed and constraints.

Jev returned 60 valid answers and matched 54 fixture labels. DeepSeek Flash returned 57 valid answers, matched all 57 labels, and timed out on three calls. Jev’s successful calls were much faster in this run (p50 400 ms versus 2,377 ms). Typed output still does not mean correct output: one wrong Jev answer came back at probability 0.98. Software still owns actions, permissions, thresholds, and escalation.

What you can ask, and what you can answer

The new contract in Umwelten is small. A request is some state plus a record of questions. A backend has one job: judge().

type JudgmentState =
  | string
  | { [key: string]: JsonValue }
  | JsonValue[];

interface JudgmentRequest<Q extends JudgmentQuestions = JudgmentQuestions> {
  state: JudgmentState;
  questions: Q;
}

interface JudgmentBackend {
  identity: string;
  judge(
    request: JudgmentRequest,
    options?: { signal?: AbortSignal },
  ): Promise<JudgmentResult>;
}

State can be a string (a support message) or a structured object (a claim and a passage, a task plus evidence). The questions are not prompts in the usual sense. Each one is a kind, with a closed set of answers:

type JudgmentQuestion =
  | {
      kind: "binary";
      instructions: string;
      criteria?: { true: string; false: string };
    }
  | {
      kind: "choice";
      instructions: string;
      options: Readonly<Record<string, string>>;
    }
  | {
      kind: "ordinal";
      instructions: string;
      levels: readonly string[];
    };

What comes back is tied to that kind. Binary is a probability of true. Choice is a named option plus a distribution over exactly those names. Ordinal is a distribution over the ordered levels, plus an expected index:

type JudgmentAnswer =
  | { kind: "binary"; probabilityTrue: number }
  | {
      kind: "choice";
      selected: string;
      probabilities: Record<string, number>;
    }
  | {
      kind: "ordinal";
      probabilities: number[];
      expectedIndex: number;
    };

Umwelten validates finite probabilities, exact outcome coverage, and a distribution total within 0.0001 of one. It does not silently renormalize a malformed distribution. Native Jev probabilities are recorded separately from estimates a generative model writes into the same schema. A provider’s extra “confidence” field is kept in the raw response; it is not treated as a probability of being right.

That is the whole ask/answer surface. No tool list. No continuation. No “reply as JSON and we’ll parse it if we’re lucky” as the public interface — the schema is the interface.

Mapping a case: claim and passage

The interesting part is what you put in state. For the evidence family, state is not a blob of prose. It is two fields the application already understands:

{
  state: {
    claim: "The effect was observed in humans.",
    passage:
      "The effect was observed in mice; no human results are reported.",
  },
  questions: {
    relation: {
      kind: "choice",
      instructions:
        "Does the supplied passage support or contradict the claim, or leave it unresolved? Use only the passage, not outside knowledge.",
      options: {
        supported: "The passage establishes the claim",
        contradicted: "The passage establishes that the claim is false",
        unresolved: "The passage establishes neither truth nor falsity",
      },
    },
  },
}

The model cannot invent a fourth label. It has to put probability on supported, contradicted, and unresolved. The application already knows what those words mean, because it supplied them.

The other two families are the same contract with different state shapes:

Scroll the table horizontally to see all columns.

Refund is the simplest classifier: true or false, with instructions that a withdrawn request or a policy question is not a refund. Completion is a rubric: “no implementation,” “implementation exists but completion is not verified,” and “implementation and all acceptance checks verified complete.” The fixtures describe check results. The experiment does not run those checks, inspect real code, or authorize a release.

The September 20, 2026 pilot used 30 agent-authored synthetic contrast pairs — ten pairs per family, labels fixed before inference and never sent to either model. These are development fixtures, not an independently reviewed, held-out test set. Contrast pairs and repeated wording make observations dependent. No client data was tested or approved for publication.

Same answers, two ways to get them

Both backends implement JudgmentBackend. The application code does not change.

Jev (typesafe/jev-1.13, returned as typesafe/jev-1.13-20260917) hits Mycel’s /v1/decisions endpoint. It is not a chat completion with a schema bolted on. The wire format is the judgment itself. Umwelten marks those probabilities native.

DeepSeek Flash (deepseek/deepseek-v4.1-flash) uses Umwelten’s existing Interaction path: the same state and questions, then generateObject against the answer schema. It is still writing a completion. Umwelten marks those probabilities generated. Direct OpenRouter access was used for the full baseline.

You can sort of get the same answer. DeepSeek’s valid answers even matched the labels more often in this run. What you do not get from the generative path, at least here, is the thing the classifier interface is for: a tight outcome set at interactive speed. Jev’s successful calls had a p50 of 400 ms. DeepSeek’s were 2,377 ms, with a p95 of 12,947 ms and three 120-second timeouts. Constraints are the other half: the model cannot wander into a fourth class, and a distribution that does not sum to one is a failed call, not a quietly repaired one.

These timings used different transports and unmatched retry behavior, so they are not a controlled model-speed ranking. They are still the operational picture this interface is buying: a judgment in hundreds of milliseconds, over names the application chose.

Accuracy and failures stay separate

Accuracy below is among valid answers. A timeout is not a correct answer, and it is not silently converted into an incorrect class prediction.

Scroll the table horizontally to see all columns.

That is 90% among Jev’s 60 valid answers and 100% among DeepSeek’s 57 valid answers. Across all 60 attempted cases, the baseline supplied 57 correct answers and no answer for three.

The error a high score would still accept

Take the claim-and-passage case above. The fixture label is unresolved: “no human results are reported” does not establish that the effect was never observed in humans. Jev chose contradicted at probability 0.98, with 0.02 left for unresolved. A policy that accepts every answer above 0.9 would accept this error.

The other five Jev errors were conservative completion judgments. For fixtures that explicitly said every independent acceptance check passed, Jev favored “implementation exists but completion is not verified.” The tasks were CSV export, an empty-cart crash, Unicode filenames, pagination, and timezone offsets. Their evidence wording was nearly identical; these are not five independent demonstrations of a general weakness.

Probability scores describe this dataset

The Brier score sums squared errors across all classes: each predicted probability is compared with one for the labeled outcome and zero for every other outcome. Lower is better. This class-summed convention ranges from zero to two.

The ranked probability score (RPS) compares cumulative probabilities across ordered level boundaries. We divide by the number of boundaries, so a three-level question uses two. It gives partial credit for placing probability near the correct ordinal level.

Scroll the table horizontally to see all columns.

Completion RPS was 0.057133 for Jev and 0.001063 for DeepSeek. Ordinal accuracy selects the maximum-probability level; it does not round the expected level. The refund scores cover different case sets because the baseline timed out on three cases. Neither these scores nor descriptive reliability bins justify a calibration claim on this small synthetic dataset.

Speed is the point of the interface

Scroll the table horizontally to see all columns.

These are nearest-rank quantiles of successful calls, including the original smoke calls later reused from cache. The baseline’s three 120-second timeouts are excluded. Replay retains original call timings; these numbers do not measure replay speed.

A generative model can be prompted into the same JSON. It will often get there. It will not get there in 400 milliseconds with a closed outcome set that the application defined in advance. That combination — constraints plus speed — is what makes the classifier interface worth having, even when the slower backend is more accurate on a tiny synthetic set.

Calls were sequential within each run, with a 120-second abort deadline. Native Jev used one attempt per call; the generative baseline retained the runtime’s SDK retry behavior. Successful-call timings include any SDK-internal retries, which were not individually recorded.

Choice order changed probabilities, not winning labels

We reversed the option definitions for the 20 evidence-relation questions. Binary questions and ordinal rubrics were unchanged. Using the same output directory reused those 40 unchanged records, so the experiment made 80 distinct native calls, not 120.

Across the 20 original/reversed pairs:

• Zero winning labels changed.
• Four probability distributions changed.
• The maximum absolute per-option difference was 0.03.
• Evidence-relation Brier changed from 0.097990 to 0.098860.

One reversed ordering is not evidence of general order invariance or run-to-run determinism.

Charges and estimates answer different questions

All amounts below are USD. They have different scopes and accounting sources; they are not interchangeable model prices.

Scroll the table horizontally to see all columns.

The native ledger contained exactly one debit per request across 80 distinct requests. Mycel independently meters serialized content, so its billing units differ from upstream token usage. The direct baseline figure is an estimate, not an invoice. Usage and charges for its three timed-out calls are unknown.

A gateway failure was not a model limitation

The first attempt to run the generative baseline through Mycel failed all three smoke cases with 503 no_eligible_offer. Mycel inferred a required structured-output capability from the request, then rejected the route because the catalog did not advertise that flag. That was an incorrect automatic capability gate, not evidence that DeepSeek could not produce structured output. The original full comparison consequently used direct OpenRouter access for the baseline.

The correction removed automatic capability inference from chat request parameters. Mycel forwards response-format, tools, media, and streaming parameters for the upstream provider to accept or reject. The corrected gateway was deployed without changing Offer flags. A fresh DeepSeek Flash smoke through Mycel returned 3/3 valid and correct answers, with three actual debits totaling $0.000156. That verifies the route. It is not a full 60-case rerun.

What should happen before production use

Keep this API experimental. Add independent label review, held-out scenario groups, more varied natural inputs and longer contexts, repeated option-order experiments, matched transport conditions, and explicit asymmetric error costs. This pilot asks one question per case and establishes no batching benefit.

Use the typed answer as an input to software policy. A schema can bound the result. It cannot supply missing evidence or make a false judgment true. Actions, permissions, acceptance thresholds, and escalation remain application responsibilities.

The types and the runner live in Umwelten at commit 548fd0d: packages/core/src/judgment/types.ts for the signatures, examples/jev for the fixtures and the live/replay runner. This report is about the interface, not an install walkthrough.

Sources and verification

The report was checked against frozen requests and labels, raw responses, saved summaries, and separate ledger reconciliation. Accuracy, Brier/RPS, nearest-rank latency quantiles, choice-order deltas, and aggregate native charges were independently recomputed from the supplied records. Raw archives are not published here because they contain internal operational identifiers.

• Pinned source: merged Umwelten revision, Jev runner, and Mycel demonstration.
• Jev pilot tracking issue and Mycel pass-through correction.
• The executed prototype’s verification recorded 35 passing judgment tests and TypeScript checks for core, evaluation, and examples. The gateway correction recorded 398 passing Mycel tests, typecheck, and lint before deployment. These checks validate implementation behavior, not the dataset’s labels or the models’ general reliability.