What Runner adds

The contracts above, made concrete. The two capabilities that matter most have their own sections below; the rest follow as a list, ordered by how much difference each makes in practice.

Truncated tool calls that still parse: closing the JSON when max_tokens runs out

When a tool call runs past its token budget, most engines return an empty or malformed tool_calls - commonly finish_reason: "length" with nothing usable, or truncated JSON the caller cannot parse and has to repair or retry. Runner closes the call to the smallest schema-legal document instead, so the arguments still parse. This is forced-truncation recovery, not ordinary JSON-Schema constrained decoding: once a document starts, Runner emits a legal ending when the budget expires. On local models, where context is tight and generation is slow, it is the difference between an agent loop that finishes and one that retries from scratch - burning tokens, time, and context window.

What each engine hands the caller when the token budget cuts a tool call short

• same box, same tool schema, same prompt, tool_choice:"required", temperature 0, budgets 1→64:

The control rung proves the failure is truncation, not misconfiguration: every engine completes at 64. Below that, only Runner returns an executable call; the others each hand back something broken or absent. This is the behaviour across every OpenAI-compatible engine we have measured - not a claim about engines we have not. † TensorRT-LLM and SGLang were measured on a Qwen3-1.7B substitute (their model registries did not carry the granite-4.1-3b used for the other four); truncation recovery is a property of the runtime, so this measures the engine, not the model.

The truncation benchmark has the full recipe and raw responses and pins Runner’s column as a per-release regression gate (make test-truncation); the agent-torture gate tests the same failure mode. Tool-call fidelity under quantization is measured too: on a full quant ladder, constrained decoding held schema conformance and tool selection at 100% down to Q4_0 while argument agreement decayed to 50% - it guarantees the SHAPE of a call at any quantization, not its contents (docs/quant-fidelity.md).

Train the GGUF you actually serve

Runner trains LoRA adapters directly through the frozen quantized GGUF used for inference. There is no FP16 training copy and no separate training framework: the serving forward pass is the training forward pass, so the policy you sample is the policy you train - the train/infer numerical mismatch that silently breaks on-policy learning cannot occur between two codepaths that are one codepath. And training is deterministic in the strongest sense: same data + same seed + same config produce a byte-identical adapter file, with a machine-written provenance record (base/data/adapter sha256s, seed, full config) beside every adapter - adaptation as an auditable artifact, not a run that is merely repeatable “within tolerance.”

Measured, on a public artifact you can download and reproduce (Qwen3-4B-Runner-ToolUse-Q4_K_M):

--score gives teacher-forced logprobs for evals and rewards, --lora serves any adapter back, --merge-lora folds an adapter into the base for a standalone GGUF any runtime can serve (with its own provenance record - and the honest caveat that a quantized merge rounds the delta; --lora is the exact form), and scripts/train-grpo-lite.py closes the loop into seeded, replayable reinforcement fine-tuning. Design, gates, failure modes and every number above: docs/adaptation-engine.md.

The rest of what sets Runner apart, ordered by how much difference each makes:

• A shared GPU stops being first-come, first-crash. Run a coding agent beside an embeddings model beside a draft model and the usual outcome is that one load kills another. Runner processes on the same GPU share a VRAM registry: a refused load names every live holder by PID, model, bytes, and uptime, --wait-for-vram turns that refusal into a bounded queue, and records left by dead processes are reaped. It makes a GPU something you can schedule rather than something you hope fits.
• You can ask what fits before loading anything. The usual way to find out whether a model fits is to load it and wait for the failure. --caps needs no model file and returns one JSON document containing live RAM/VRAM, backend and GPU limits, CPU and GPU quant lists, admitted architectures, placement modes, and model-count limits. A supervisor, tray controller, or CI job can reject an incompatible placement before dispatch, which removes a whole class of load-wait-fail-retry loops. For a specific file, --fit answers the same question from the model’s GGUF header - the first few megabytes - so a ranged read decides whether the rest of the download is worth starting.
• Constrained decisions come with a confidence signal. choice_logprobs records each JSON-schema branch as legal alternatives, a posterior renormalized over them, and the probed probability mass - how confident the model was choosing one branch over another, which is what routing and calibrated classification actually need. The included calibration tool turns labeled decisions into accuracy, Brier-score, and ECE gates; the included boundary lane does the opposite job, recording where serving conditions DISAGREE on a tool choice with no labels at all. This is a decision record rather than ordinary token logprobs, and a power-user feature: most workloads will never reach for it.
• A hardware switch has a correctness contract. If you move a workload between backends and the output quietly changes, that is a bug, not a tuning artifact. CPU/GPU identity here belongs to an exact SHA-256-pinned model and execution path, and faster kernels that reassociate floating-point sums must pass numerical tolerance gates rather than inherit a correctness claim from the backend name. Most users never compare outputs across backends; this is documented because the project treats correctness as a gate, not because it is a headline.

The full compatibility method is in docs/compatibility-program.md and performance measurements are in docs/performance.md. Work that was built, measured and rejected is kept too, so it is not attempted twice: docs/negative-result-expert-cache.md for MoE expert caching, and docs/negative-result-metal-multirow-matvec.md for the multi-row Metal decode matvec - which also records what the CPU/GPU byte-identity contract costs in reachable GPU optimizations.

This page is the README section of the same name, copied at build time; the README on GitHub is the source.