The first question I asked the model was the natural one, and the model answered it honestly, which is how I learned the question was wrong. I had a pile of Telegram messages and a simple need: keep the ones with substance, drop the empty greetings. So I asked the decision model, does this message have substance? It scored a bare “gm” at 0.45 and a member’s real, answerable question at 0.45. Two different messages, one number, no separation at all. The model was not broken. It was telling me the truth about the question I had asked, and everything I know about calibrating a decision model starts there.

Ask what the text says, not what to do about it

The first lesson is about grammar, in the linguistic sense. A decision model of this kind is an encoder doing something close to textual entailment: it reads a passage and a hypothesis and reports how much the passage supports the hypothesis. That shape has consequences. The question works when it asks what the text is; it misfires when it asks what should be done about the text.

The documentation’s own cautionary example is a bird flying through gaps in pipes: “where is the bird relative to the gap?” produced clean graded probabilities, while “which way must the bird move?” came out inverted on every checkpoint, because the option word “up” pulls toward the word “above” in the state.^[1] Perception questions, then let your code map the answer to an action. The model judges; the program acts.

The second lesson follows from the first and paid my rent this month. Questions can be true in one direction and false in the other, and the direction matters. “Does this have substance?” asks about an absence, and absences are weak entailments. The phrasing that finally separated greetings from everything else was the reversed one: “Is this message ONLY a greeting or pleasantry?” Scored on real messages, the reversed question put greetings at 0.72 to 0.91 and everything substantive at 0.46 or below. The natural question had put a greeting and a genuine question at the same score; the unnatural one put a whole language’s worth of small talk on one side of a line.^[2]

Four options beat one

Spam got the same treatment. The obvious question, “is this spam?”, is a yes/no question, and yes/no questions force the model to squeeze every kind of unwanted message and every kind of awkward message through the same gate. Half the greetings in my sample set came out mislabeled as spam, which in production means silently deleting members who said good morning.

The fix was to stop asking a yes/no question and ask a classification question with room to be right: “What kind of message is this?”, with four described options: greeting, question, discussion, spam. Descriptions matter, because the description is what the model matches the text against; “spam: an advertisement, scam, phishing, fake giveaway, or promotion with links or contact solicitation” gives the entailment something to grip. On the sample set, the four-way question separated spam cleanly at the top: spam messages scored 0.92 to 1.00 and everything else scored 0.00 to 0.31. One question, one forward pass, and two verdicts read off the same distribution: the spam probability drives the delete ladder, and the greeting probability drives the cleanup, because both questions ride along in a single call for free.^[2]

Calibration is the product

Here is the part that deserves more attention than any benchmark: the model ships with honest probabilities, and you ship with thresholds, and the gap between those two sentences is where every false delete lives. The multilingual checkpoint that makes the bot language-agnostic also ships deliberately uncalibrated, temperature 1.0, which means it happily reports 100 percent and 0 percent, and if you read those numbers as certainty you will have a bot that deletes with total confidence and total error. The defaults in my configuration, delete at 0.85, flag at 0.60 for spam, delete at 0.70 for greetings, were not chosen from the documentation. They came from measurement, and the measurement came from the pipeline running in dry-run mode, logging what it would have done, against real messages in a real group.^[2]

An independent evaluation of the model gives the wider picture and the correct amount of humility. On 500 labelled examples with no tuning, the model was strong on clear-cut categories, 96 percent on SMS spam, and weak on graded ones, 35 percent on five-star ratings, with calibration error tracking the same gradient. The evaluation’s conclusion is the one to internalize: expect a decision model to work out of the box for clear-cut categories and yes/no checks, and plan to rephrase, fine-tune or cascade for anything ordinal or nuanced.^[3]

The evaluation script that ships with the bot is the calibration loop in fifty lines: feed it a JSONL of your group’s messages, choose thresholds, and it reports what the pipeline would have done, per question, at those thresholds. Collect fifty to two hundred real messages, including the borderline ones, and the guesswork ends. If the numbers still fall short, the model can be fine-tuned on decisions from your own domain, and the jump is not subtle: 0.362 to 0.766 on its typed-decisions benchmark for the specialized checkpoint, trained on free cloud GPUs from a notebook the repository ships.^[4]

The case against my own craft

The uncomfortable truths first. Fifteen hand-picked messages separating cleanly is not evidence; it is an anecdote with a decimal point, which is why the bot’s own documentation prints the sample size and the warning next to every number. Selection bias compounds it: messages chosen because they are interesting are not messages as they arrive at 3 a.m. Over-fitting the phrasing to one group’s dialect is the subtler trap, the question that separates your community’s spam may be reading your community and not the world. And there is a theater of measurement that a decision model invites: thresholds and confidence numbers feel like rigor even when the labelled set under them is thin. The numbers are only as honest as the examples.

Against all that stands one solid fact: a threshold you measured on your own data, even crudely, is better than a threshold you imported from someone else’s guess. The model returns its probabilities in milliseconds; the calibration is the part only you can do.

A threshold is a promise you make to your members about what will happen to their words. Measure before you make it.

---

1. NandhaKishorM/laya, Apache-2.0, and the Laya integration guide: the model performs something close to textual entailment; perception questions outperform action questions; option descriptions are the matching surface; option lists degrade past roughly 20 entries. The bird-and-gaps example is from the guide’s own measured cases.
2. erubboli/telegram_spam_bot, README section “Question phrasing (measured, not guessed)”: the four-way choice (greeting / question / discussion / spam) separated spam at 100% of the sample set (spam 0.92 to 1.00, everything else 0.00 to 0.31) where a binary “is this spam?” mislabeled half the greetings; the reversed greeting question scored greetings 0.72 to 0.91 against substantive messages at 0.46 or below, where “does this have substance?” scored “gm” and a real question identically at 0.45. Defaults: delete at 0.85, flag at 0.60, greeting delete at 0.70. Sample size approximately 15 hand-picked multilingual messages; recalibration on 50 to 200 real messages via scripts/evaluate_laya.py is stated as mandatory. CPU latency approximately 130 to 150 ms per message.
3. Laya: open-source alternative to TypeSafe Jev, independent 500-example evaluation, September 2026: 96% on SMS spam and 93% on news topic for Laya untuned, against 35% on a five-level rating and 65% on prompt-injection detection; expected calibration error 0.05 to 0.06 on the easy tasks, 0.34 to 0.40 on the hard ones.
4. NandhaKishorM/laya: the typed-decisions checkpoint scores 0.766 against 0.362 for the base English model on the typed-decisions benchmark; fine-tuning notebook runs on free Kaggle GPUs. The bot and its pipeline are described in A Telegram Spam Bot on a Local Decision Model; the model itself in From LLMs to System One Models: Jev and Laya.