Xyntetik xyntetik

Truncation and constrained generation

When the budget runs out mid-call, the call still parses.

An agent asks a model to call a tool. The model starts writing the call as JSON. The token budget expires halfway through. In most engines the caller now holds a broken document: nothing to execute, nothing to parse, sometimes an HTTP 500. Runner finishes the document legally instead, and the agent loop continues.

The practical benefit first

On a local model, a broken tool call is not an error. It is the whole retry budget.

What a budget is

Every request carries max_tokens, the most tokens the model may generate. Agents set it tight on purpose: local models are slow and context is scarce. A tool call is JSON, and JSON that stops early is not JSON.

What usually happens

The engine returns finish_reason: "length" with an empty call, or leaks the model's tool-call framing into the assistant's prose, or hands back a tool_calls object whose arguments fail to parse. The agent cannot proceed. It retries from scratch, burning the tokens, the time and the context window it was already short of.

What Runner does

Runner is already constraining the call to the tool's JSON schema as it decodes. When the budget expires, it emits the smallest schema-legal ending for the document that has been started, marks finish_reason: "length" so the caller knows the budget was hit, and returns a tool_calls entry whose arguments parse. The agent gets an executable call and a truthful signal, and continues.

"Arguments parse" is a bound the engine holds, not a hope. The streaming validator refuses to nest a document deeper than the parser can read back, so a close can never produce something the engine itself would reject. That was not always true: the fuzzer found a band where the two disagreed, and the fix was to make the validator refuse early rather than to widen the parser, because refusing is the direction that cannot ship a broken document.

Measured

Same box, same schema, same prompt, six engines, seven budgets.

tool_choice: required, temperature 0, budgets 1, 2, 3, 5, 8, 16 and 64 tokens. The 64-token control proves the failure is truncation, not misconfiguration: every engine completes there. Below it, only Runner returns a parseable call.

executable tool call, arguments parse no usable call
Per token budget, does the client receive an executable tool call123581664 ctrlRunnerRunner, max_tokens 1: executable tool call, arguments parseRunner, max_tokens 2: executable tool call, arguments parseRunner, max_tokens 3: executable tool call, arguments parseRunner, max_tokens 5: executable tool call, arguments parseRunner, max_tokens 8: executable tool call, arguments parseRunner, max_tokens 16: executable tool call, arguments parseRunner, max_tokens 64: executable tool call, arguments parsevLLM 0.27.1vLLM 0.27.1, max_tokens 1: no usable callvLLM 0.27.1, max_tokens 2: no usable callvLLM 0.27.1, max_tokens 3: no usable callvLLM 0.27.1, max_tokens 5: no usable callvLLM 0.27.1, max_tokens 8: no usable callvLLM 0.27.1, max_tokens 16: no usable callvLLM 0.27.1, max_tokens 64: executable tool call, arguments parsellama.cpp b10488llama.cpp b10488, max_tokens 1: no usable callllama.cpp b10488, max_tokens 2: no usable callllama.cpp b10488, max_tokens 3: no usable callllama.cpp b10488, max_tokens 5: no usable callllama.cpp b10488, max_tokens 8: no usable callllama.cpp b10488, max_tokens 16: tool_calls present, arguments do not parse{ "llama.cpp b10488, max_tokens 64: executable tool call, arguments parseOllama 0.32.14Ollama 0.32.14, max_tokens 1: no usable callOllama 0.32.14, max_tokens 2: no usable callOllama 0.32.14, max_tokens 3: no usable callOllama 0.32.14, max_tokens 5: no usable callOllama 0.32.14, max_tokens 8: no usable callOllama 0.32.14, max_tokens 16: HTTP 500500Ollama 0.32.14, max_tokens 64: executable tool call, arguments parseTensorRT-LLM 1.2.1 †TensorRT-LLM 1.2.1 †, max_tokens 1: no usable callTensorRT-LLM 1.2.1 †, max_tokens 2: no usable callTensorRT-LLM 1.2.1 †, max_tokens 3: no usable callTensorRT-LLM 1.2.1 †, max_tokens 5: no usable callTensorRT-LLM 1.2.1 †, max_tokens 8: no usable callTensorRT-LLM 1.2.1 †, max_tokens 16: no usable callTensorRT-LLM 1.2.1 †, max_tokens 64: executable tool call, arguments parseSGLang 0.5.17 †SGLang 0.5.17 †, max_tokens 1: no usable callSGLang 0.5.17 †, max_tokens 2: no usable callSGLang 0.5.17 †, max_tokens 3: no usable callSGLang 0.5.17 †, max_tokens 5: no usable callSGLang 0.5.17 †, max_tokens 8: no usable callSGLang 0.5.17 †, max_tokens 16: no usable callSGLang 0.5.17 †, max_tokens 64: executable tool call, arguments parse
Measured 2026-08-19 on granite-4.1-3b. Runner c7d6c3d (0.1.19-alpha), vLLM 0.27.1, llama.cpp b10488, Ollama 0.32.14, TensorRT-LLM 1.2.1, SGLang 0.5.17. † measured on a Qwen3-1.7B substitute because those registries did not carry granite-4.1-3b; recovery is a property of the runtime, so the substitution measures the engine, not the model. The claim covers the engines and versions measured, not engines that were not.

How the five failures differ

They rank differently for a caller. An executable call beats a detectable empty or error response, which beats protocol rendered as prose, which beats a tool_calls object whose arguments silently fail to parse. vLLM and llama.cpp leak the framing into content; llama.cpp emits unparseable arguments at 16 tokens; Ollama hides the framing but returns HTTP 500 at 16; TensorRT-LLM and SGLang return an empty, callless message with HTTP 200. None of the five closes the document.

A gate, not a one-off

The property is an engine guarantee, grammar plus closer, not model quality. It holds identically for the random two-layer CI fixture and for granite-4.1-3b, which is why make test-truncation runs on any CPU with no GPU and no competitor, on every release. The agent-torture gate tests the same failure inside multi-turn agent loops. Recipe, raw responses and the substitute-model notes: docs/truncation-benchmark.md.

The technical detail

Forced-truncation recovery is not ordinary constrained decoding.

Ordinary JSON-Schema constrained decoding restricts which token may come next, so the output is valid if it finishes. It says nothing about what happens when the output cannot finish. Runner's schema compiler produces a streaming validator that knows, at every byte, what the smallest legal completion of the document is: close the string the pattern still accepts, complete the number inside its declared range, supply the required properties, close the object. When the budget expires, that completion is emitted.

The same compiler backs --json, OpenAI response_format, Responses text.format and tool parameter schemas: objects, arrays, enums, const, type unions, numeric bounds, string lengths and anchored patterns, array counts, and the tool-discriminated union agent clients use. Unsupported or ambiguous constraints fail at compile time rather than being silently weakened.

./runner -m model.gguf --serve
curl localhost:8080/v1/chat/completions -d '{
  "messages":[{"role":"user","content":"Book a table for two at 19:00."}],
  "tools":[{"type":"function","function":{"name":"book_table",
    "parameters":{"type":"object","properties":{"people":{"type":"integer"},
    "time":{"type":"string"}},"required":["people","time"]}}}],
  "tool_choice":"required","max_tokens":8}'

The response carries a tool_calls entry whose arguments parse, and the usual finish_reason: "length" so the caller knows the budget was hit.

Shape is guaranteed at any quantization. Contents are not.

Across a full quant ladder, constrained decoding held schema conformance and tool selection at 100% down to Q4_0 on two model families, while argument agreement with the Q8_0 reference decayed to 50%. The closer guarantees the call is well-formed and names the right tool; what goes in the arguments is still the model's. The ladders.

Decisions with a confidence signal.

Because the validator knows the legal branches at a decision point, Runner can record choice_logprobs: each schema branch as a legal alternative, a posterior renormalized over them, and the probed probability mass. Routing and calibrated classification need that, and an included calibration tool turns labeled decisions into accuracy, Brier-score and ECE gates.