# Privatemode 团队用 GLM-5.3-Flash 将 LLM 变成单次前向的类型化决策模型

- 来源：Hacker News 热门（buzzing.cc 中文翻译）
- 作者：flxflx
- 发布时间：2026-09-27 19:32
- AIHOT 分数：67
- AIHOT 链接：https://aihot.news/items/cmujrs8b40uo9ro9h3fuuyn6w
- 原文链接：https://www.privatemode.ai/blog/system-one-from-glm-flash

## AI 摘要

Privatemode 团队展示了一种无需微调的方法，通过编号选项、预填 choice_index: 前缀并读取选项索引的 log 概率，让 GLM-5.3-Flash 在单次前向中输出带概率的类型化决策。

## 正文

Typed decisions with a probability for every option, in a single forward pass: matching Jev's accuracy and speed with an LLM.

Johannes Hötter

VP Growth

Marko Rosenmüller, PhD

Technical Lead AI

TL;DR: In this post, we show how an off-the-shelf LLM can make typed decisions in a single forward pass. This approach makes it possible to turn an LLM into a Jev-like decision model.

We evaluate the approach using GLM-5.3-Flash running on Privatemode. Using a benchmark constructed from public data sets, we show that this setup delivers results that are on par with TypeSafe's Jev in terms of decision accuracy/correctness and speed.

As a bonus, the setup with GLM-5.3-Flash enables typed decisions on images, which is not possible with Jev.

Why typed decisions

Much of what software asks an LLM is a decision. "Which team should handle this ticket?", or "Does this contract clause belong in the liability section?".

In such cases, software typically requires that the LLM's response follows a certain format like JSON and that it comes from a pre-defined set like "yes" and "no".

Given the right instructions, LLMs can typically already fulfill this reliably. However, in the basic approach, speed and costs become an issue: For each decision, the LLM needs to write a whole JSON object, and a reasoning model may think for hundreds of tokens before that. Further, you also don't learn the confidence of the model (unless you explicitly ask it). All these aspects can matter a lot in practice and have so far prevented people from employing LLMs for decision making in high-volume/high-throughput scenarios.

Specialized decision models (or "System One" models) like Jev and Laya are designed to address this. You pass in a piece of state and a set of named options, and you get back the chosen option together with a confidence value (i.e., probability) for each one.

Turning an LLM into a decision model

Initially, we asked ourselves if an LLM could be turned into a decision model with Jev-like properties. The short answer is: "yes". In the following, we show how it works.

To understand our approach, it's important to understand how LLMs work:

An LLM never writes text directly. Given a prompt, an LLM outputs a probability distribution over its entire vocabulary of tokens. In text generation, in the simplest case, the token with the highest probability is selected as the next token. The selected token then is appended to the prompt and the whole process repeats. As described above, this is costly and slow if you just want to set a few fields in a JSON object.

Our core insight is that it's unnecessary to have the LLM predict the whole JSON object, as we already know its shape. We're only interested in the LLM's typed judgement for a given input.

We realized that it's possible to craft prompts so that we get the typed judgement in a single run of the LLM — with no fine-tuning, on the model exactly as it ships. This is the difference to Jev and Laya, which are models trained for the purpose. The basic steps are as follows:

Number the options. The state, the question, and the output options go into the prompt as JSON, with an index on every option. The instruction asks the model to answer with choice_index: followed by an index.

Prefill the answer. The prompt ends with choice_index:. Consequently, the first token the model produces will be an index into the pre-defined options.

Evaluate the output. Rather than reading the token the model emits, we read the probabilities it assigned to all option indexes at that single position. Normalized over the options, these give a probability for each answer, and we simply pick the most probable one.

媒体内容 · 前往原文查看

End the prompt inside the answer

user

{"state": "I was charged twice for my order.", "question": "Which team?", "options": [

{"index": 0, "name": "payments"}{"index": 1, "name": "complaints"}{"index": 2, "name": "technical"}

]}

assistant

choice_index:

Read one row of logits

max_tokens: 1temperature: 0allowed_token_ids

Keep the options, renormalize

0 payments62.1%

1 complaints37.7%

2 technical0.2%

Answer payments confidence 0.39, where 1 means certain and 0 means the options are equally likely

The prompt numbers the options and it ends with the assistant’s answer already begun as choice_index:, so the next token is the index. A mask on the vocabulary allows only the option indexes, the API returns their log probabilities, and normalizing them over the options gives a probability for each answer. The logit values in the middle panel are illustrative.

We implemented the above steps for GLM-5.3-Flash running on vLLM (in Privatemode). We use the /chat/completions endpoint with continue_final_message and add_generation_prompt: false, because these let the model continue the prefilled assistant turn from step 2 instead of starting a new one. They also let us pass images next to the text, which is what makes typed decisions on images possible.

For vLLM and GLM-5.3-Flash, we found the following details to matter:

vLLM's allowed_token_ids can be used to limit the LLM's output vocabulary only to allowed options. It drops every other token to -inf. We set it, but it is a guardrail rather than a requirement.

top_logprobs is not enough for step 3. It reports the distribution before the restriction is applied, so formatting tokens such as a leading space take up the top slots, and some options drop off the list and appear to have a probability of zero. vLLM's logprob_token_ids solves this: it returns the log probability of exactly the token ids you ask for.

The token ids of the indexes depend on the model's tokenizer. Digits aren't always single tokens. GLM-5.3-Flash, for example, has a single token for 12. Rather than shipping a model-specific tokenizer, the library gets the token ids from the server, which keeps it simple to use with any model: sending a prompt to /completions with echo returns its exact tokenization by the model that is actually serving.

You can find our implementation in the below repository.

edgelesssys/privatemode-decisions

The Python library: token oracle, prompt, masking and renormalization, against any vLLM-backed endpoint.

Try it

The playground below runs GLM-5.3-Flash queried with the above setup on Privatemode, directly from your browser. Pick one of the examples, among them a scanned invoice and a question that depends on your local time, or write your own questions and add images. Each answer comes back as a distribution over its options, typically within a few hundred milliseconds.

Playground

GLM-5.3-Flash on Privatemode

Everything that is not a question is context.

A question is a line followed by

choices: a, b, c

.

Options can explain themselves:

repair = broken devices

.

With no choices line, the last line is a yes/no question.

Images, added or pasted, are part of the context.

Answers appear here: a probability for every option, and how sure the model is.

By the way: what you type here is end-to-end encrypted.

How it works

The distribution is often very useful, e. g., to decide whether to include a human-in-the-loop. The model solves most classic trick questions, but not all of them.

Benchmark results

We evaluated our approach using a custom benchmark, which is available in the below repository.

edgelesssys/privatemode-decisions-benchmark

The benchmark: methodology, frozen dataset specs, harness and aggregation. Every number in this post can be recomputed from it.

We compared three systems on 29 public, labeled datasets: GLM-5.3-Flash hosted on Privatemode and queried with the technique above, TypeSafe's Jev, and Convai's Laya.

The datasets have between 2 and 151 options and cover intent routing, sentiment, topic classification, moderation, entailment, question answering, legal text, and scanned documents. Both English and German text is included in the corpus. All three systems receive the same state, the same option names in the same order, and the same instruction.

We ran each dataset twice. Even at temperature 0, our GLM-5.3-Flash and Jev changed up to 3.5% of their answers between identical runs: temperature 0 removes the randomness from sampling, but batching and floating-point arithmetic still keep a forward pass on a busy server from being bit-reproducible. We therefore treat smaller differences as noise.

We ran Jev and Laya with their default settings and did not tune our prompt on these datasets.

Accuracy

Compared across the 28 text datasets, GLM-5.3-Flash and Jev are on par. Each is more accurate on 10 datasets; on the remaining 8, the two are within one percentage point of each other. The median gap is 0.7 percentage points in Jev's favor, which is not statistically significant (p = 0.64). Laya, a model with 421 million parameters that we ran locally, scores lower than both on most datasets. Its median gap is 13 to 15 percentage points, which is statistically significant (p < 0.001).

The number of options has a larger effect on accuracy than the choice between Jev and GLM-5.3-Flash.

媒体内容 · 前往原文查看

Which one is more accurate, dataset by dataset?

10 GLM-5.3-Flash more accurate8 about the same10 Jev more accurate

The typical gap is 0.7 percentage points, slightly in Jev’s favor. Two equally accurate systems would show a gap at least this large in about 6 out of 10 comparisons, so it is well within chance.

Accuracy by number of options

GLM-5.3-FlashJevLaya

with reasoningembedding similarity

Hover or tap a mark for its numbers. Top: each square is one of the 28 datasets both hosted systems answer, colored by which of the two was more accurate on it; within one percentage point counts as the same, the variation between two identical runs. The chance estimate is a two-sided Wilcoxon signed-rank test over the per-dataset differences (p = 0.64). Bottom: mean accuracy by number of options, averaged only over datasets all three systems answered, so each point covers the same questions; the bands along the bottom are not evenly sized. The dashed and dotted lines are controls on the same datasets, both zero-shot like the rest: GLM-5.3-Flash allowed to reason before it answers, and plain embedding similarity with no decision model.

Across datasets, the number of options changes along with everything else about the task. Three datasets, however, label the same questions twice, once coarsely and once finely, so the task stays the same and only the number of options changes. On TREC, going from 6 to 42 options, Jev drops from 92.1% to 85.6%, GLM-5.3-Flash from 91.2% to 79.6%, and Laya from 88.4% to 51.2%. On MASSIVE, going from 18 scenarios to 59 intents raises the scores of Jev and GLM-5.3-Flash in both languages, while Laya's score drops. The number of options alone doesn't determine how hard a task is.

媒体内容 · 前往原文查看

GLM-5.3-Flash on PrivatemodeJevLaya

TREC questions

MASSIVE, English

MASSIVE, German

Hover or tap a mark for its numbers. The same questions labeled twice, once coarsely and once finely, so the only variable that changes is the number of options. TREC splits 6 question types into 42; MASSIVE splits 18 scenarios into 59 intents, in English and in German. Numbers on the right are accuracy at the finer granularity.

Latency and cost

We measured latency in separate runs with one request at a time, because timings taken under load measure the queue rather than the model.

As Privatemode is hosted in the EU and Jev is hosted in the US, we ran four of the datasets from Germany and from the US at the same time. From Germany, Privatemode answered in 180 ms and Jev in 264 ms. From the US, the order reverses: 164 ms for Jev against 299 ms for Privatemode.

On cost, Jev is cheaper. One million decisions cost about EUR 62 with GLM-5.3-Flash and about EUR 16 with Jev, at each service's list prices.

媒体内容 · 前往原文查看

Time per decision

From Germany

GLM-5.3-Flash on Privatemode180 ms (173–263)

Jev264 ms (235–331)

From the US

GLM-5.3-Flash on Privatemode299 ms (271–402)

Jev164 ms (142–227)

0 ms450 ms

Cost per million decisions

GLM-5.3-Flash on Privatemode€62

Jev€16

Hover or tap a mark for its numbers. Time per decision is the model call as a user sees it, network included, measured one request at a time from Germany and from the US at the same time, on four datasets. The dot is the median; the band runs from fast requests (10th percentile) to slow ones (95th percentile). Cost is what one million decisions cost at each service’s list prices, the median over the 28 datasets both systems answer, so the scanned documents Jev cannot read are not in it.

Most of the difference comes from the price per input token, and some from how each system packages a question. Jev adds roughly 270 tokens of fixed overhead and about 10 tokens per option. The GLM-5.3-Flash prompt adds about 55 tokens of fixed overhead and about 20 per option. Below about 21 options, it sends fewer tokens than Jev; above that, it sends more.

媒体内容 · 前往原文查看

Extra input tokens GLM-5.3-Flash sends per question, compared with Jev

Below zero GLM-5.3-Flash sends fewer tokens, above zero more.

Hover or tap a mark for its numbers. One point per dataset: how many input tokens GLM-5.3-Flash on Privatemode sends per question, minus how many Jev sends for the same question. The question text is identical for both, so it cancels; what is left is the packaging. Jev adds a large fixed block and little per option, GLM-5.3-Flash a small fixed block and more per option. The dashed line is the trend across all datasets: GLM-5.3-Flash starts about 219 tokens below Jev and adds about 11 more per option. The outlier at 13 options is scotus, whose long court opinions the two tokenizers split differently.

Laya runs locally, so there is no comparable latency or price per decision for it.

Multimodal decisions

The state doesn't have to be text. GLM-5.3-Flash is a vision-capable model, so a question can come with images, such as a scanned invoice, a photo of a damaged parcel, or a screenshot. The image goes into the same prompt, and the answer is still a single token with a probability for every option. The playground's Scanned document example shows this; you can also paste or drop in your own image.

According to its documentation, Jev works on text, and Laya is a text encoder, so neither takes images as input. On RVL-CDIP, a set of 1,600 scanned business documents in 16 classes, GLM-5.3-Flash reaches an accuracy of 70.2% and is the only one of the three that can answer. A document costs more than a sentence: the image adds about 1,350 input tokens, so a million document decisions cost about EUR 270.

Capabilities

With many options, each system hits a limit. Laya's option names share a budget of 192 tokens, which is enough for the 77 intents of banking77 but not for the 151 intents of CLINC150. Privatemode's deployment of GLM-5.3-Flash reports at most 128 entries in logprob_token_ids, while the mask in allowed_token_ids takes every option. So the library sends a question with 151 options twice, identically, and reads the probabilities of 128 options from the first response and of the other 23 from the second. Both requests run the same forward pass, so the merged result is the distribution a single request would return, up to the run-to-run noise above. On CLINC150, GLM-5.3-Flash reaches 87.5% this way, against 78.4% for Jev. The second request and the long option list cost time: a decision takes 719 ms, against 249 ms for Jev. Past that, the ceiling is the 191 option indexes that GLM-5.3-Flash spells as a single token.

Some tasks only one or two of the systems can handle at all.

媒体内容 · 前往原文查看

GLM-5.3-Flash on PrivatemodeJevLaya

Scanned documents

RVL-CDIP, 1,600 business documents, 16 types

70.2%text onlytext only

Choice of model

Same code, another model

any model on the APIfixedfixed

Numbers are accuracy on the named dataset.

Further findings

Some of the remaining errors are in the labels. When most systems agree on an answer and the dataset's label disagrees, the label is often one of two defensible answers. banking77 has many such pairs, for example get_physical_card and order_physical_card, or declined_transfer and failed_transfer. About 17% of banking77's examples fall into this category, so the highest score any system could reach there is about 85% rather than 100%.

Reasoning helps, at a price. As a control, we let the same GLM-5.3-Flash reason before it answers, on all 29 datasets. It is more accurate in every band of option counts, from 89.9% against 85.5% with two options to 82.0% against 79.2% between 21 and 80. It also writes hundreds of tokens per decision instead of one and costs about EUR 350 per million decisions, against EUR 62. At the other end, plain embedding similarity, which picks the option closest to the text with no decision model at all, reaches between 45.9% and 72.8% depending on the band.

Renaming the options affects the systems differently. We re-ran every dataset with each option replaced by a synonym and nothing else changed. On boolq, where true and false became correct and wrong, GLM-5.3-Flash lost 20 points, while the other two lost less than three. Because this renaming also changes the meaning of the question, it doesn't isolate memorization. The results for every dataset are in the benchmark repository.

Build it yourself

Our library is written in Python and works with any vLLM-backed endpoint; it relies on vLLM's extensions to the OpenAI API, such as allowed_token_ids and logprob_token_ids. Its README explains how to set it up with Privatemode, and an AGENTS.md file tells coding agents what an implementation has to get right. The benchmark repository contains the methodology, the dataset specifications, the test harness, the aggregation, and every raw run as a download, so every number in this post can be reproduced without running anything again. If you know a setting that serves any of the three systems better, find a mistake, or want to add a dataset or another system, we welcome pull requests.

With Privatemode, these decisions are protected by confidential computing: your data stays encrypted in memory even during processing, and the client verifies the deployment's attestation report before sending anything. You can read more on our security page.

Build it on Privatemode

Create an account, then give this post to Claude Code, Codex, or the coding tool of your choice. Together with the GitHub repository, it has everything needed to build typed decisions into your own code, protected by confidential computing.

Create an account

Articles

Further reading

Explore other articles

View all articles

Confidential containers in Kubernetes: what a real operator exclusion requires

A technical operator exclusion in Kubernetes is possible today, but only one architecture actually delivers it. The short version, plus where to read the full paper from the 21st German IT Security Congress.

Read article

Jul 29, 2026

Building AI services for professionals bound by confidentiality obligations: a guide with Privatemode AI

How to use LLMs productively without the provider accessing the data – confidential computing and the confidentiality agreement under § 203 StGB that technology can't replace.

Read article

Jul 16, 2026