Xyntetik Runner, a Zenova AB product
Local inference you can prove.
Runner is a single-binary engine in plain C that serves, scores and trains GGUF models on hardware you own. It closes tool calls that run out of tokens so they still parse, trains LoRA adapters through the same quantized weights it serves, and records any run as a signed receipt that replays.
Every number on this site names the document it comes from and the day it was measured, including the rows where Runner loses.
The mark is an ensö, the Zen circle drawn in one uninhibited stroke and often left open, and Xyntetik tries to work the way it is drawn. One stroke: plain tools that do one thing whole, a single binary with no stack hidden underneath. Whole, imperfections included: every number carries its measurement, the rows where we lose too. Open where the work is not finished: what is not claimed is written down as carefully as what is. And present: models run here, on hardware you own, in the moment.
Three things that should not work
Each one breaks an assumption about local inference. Each one has the raw result underneath.
Cut the model off in the middle of a tool call. It still returns a valid, executable call.
Six engines, seven token budgets from 1 to 64. Below the control rung, five of them hand back nothing usable. Runner closes the JSON to the smallest schema-legal document and the agent keeps going.
measured 2026-08-19 vs vLLM, llama.cpp, Ollama, TensorRT-LLM, SGLang
How that works →Train the LoRA through the 4-bit file you actually deploy. Run it twice: the same adapter, byte for byte.
No FP16 training copy. Held-out exact tool calling went from 0.69 to 1.00, two independent runs wrote the same sha256, and stock llama.cpp scores the adapter at the same 1.00.
measured 2026-08-22, Qwen3-4B Q4_K_M, public artifact
How that works →Serve a 63 GB model on your workstation. While it idles, the operating system can take the memory back.
Loaded and idle, Runner holds +35 MB of wired memory against +60.8 GB for llama-server on the same 128 GB machine; on an 8 GB laptop it is +8 MB against +3,819 MB. The cost is a slower cold first token, and it is published.
measured 2026-09-01 (M5 Max, gpt-oss-120b) and on an 8 GB M1
How that works →Why Runner exists
Serving tokens locally is solved. Trusting what came out is not.
Excellent local engines already exist and Runner rides their formats rather than competing with them. It exists because four things go wrong the moment a local model has to do real work, and none of them is a speed problem.
An agent loop dies at the token budget
When max_tokens runs out in the middle of a tool call's JSON, most engines hand back nothing usable: an empty call, framing leaked into the prose, or arguments that do not parse. On a local model, where context is tight and generation is slow, the agent retries from scratch and burns the window it was short of.
Runner closes the call to the smallest schema-legal document instead, so the arguments parse and the loop continues. How, and what six engines do
What you trained is not what you ship
The usual fine-tuning path trains through an FP16 copy and then serves a quantized file. Two numerical paths means the policy you sampled is not the policy you trained, and a 4-bit merge can quietly erase the adaptation while perplexity still looks fine.
Runner trains through the frozen quantized GGUF it serves. One codepath, and a reproducibility contract with named edges. The measured claim and its caveat
A log is not evidence
If you have to show what a model said, what it learned from, or which file you actually deployed, a printout of a chat is not going to hold. You need the model hash, the adapter hash, the binary, the seed, the exact bytes, and a way for someone else to replay them.
Runner writes that record as a signed, chained receipt, and a verifier answers VERIFIED, DIVERGED or UNVERIFIABLE with one exit code. Receipts and the determinism scope
The server takes over the workstation
A resident inference server on the machine you also work on wires down gigabytes and ticks in the background while it serves nothing. That is the reason people kill it every time.
Runner keeps weights as evictable mappings the OS can reclaim, wakes about once every two seconds while idle, and hands everything back on /unload. The cost is a slower cold first token, and that cost is published. The idle measurement
Where it wins, where it does not
Use llama.cpp. Then read this table.
llama.cpp is the ecosystem, and if you want maximum architecture coverage and raw speed you should use it. Runner adds contracts the ecosystem does not make. Every row below is a measurement from the repository, and the comparison covers the engines and versions that were actually measured.
| Question | Runner | llama.cpp | vLLM, Ollama, TensorRT-LLM, SGLang |
|---|---|---|---|
A tool call is cut short by max_tokens. What does the caller get? |
Executable tool_calls, arguments parse, at every budget from 1 to 16 tokens |
No call, framing leaked into content; at 16 tokens a call whose arguments do not parse (b10488) | No usable call at any truncated budget; Ollama returns HTTP 500 at 16 (versions on the evidence page) |
| Decode throughput, dense models, CUDA | 81 to 93% of llama.cpp on the same slice and files | Faster | Not measured |
| Prefill throughput, CUDA | 5 to 10% of llama.cpp. Runner's honest weak column, and the next piece of work | An order of magnitude faster | Not measured |
| Architecture coverage | A named list; unknown architectures are refused by name, never guessed | The widest | Not compared |
| Memory held while idle beside your other work | +8 MB wired on an 8 GB M1; +35 MB with a 63 GB model on a 128 GB M5 Max; /unload gives it all back |
+3,819 MB and +60.8 GB on the same two machines; no unload | Not measured |
| Cold time to first token when the server is allowed to hold everything | Slower: 3.2 s vs 0.15 s on the pressured M1, 10.1 s vs 0.25 s at 120B. Residency is what llama-server buys | Faster | Not measured |
| Train a LoRA through the quantized file you serve, reproducibly | Yes. Same forward pass as serving; same data, seed and config write a byte-identical adapter | Runner-trained adapters load and score identically in stock llama.cpp; community F16 adapters load back into Runner | Not measured |
| Prove what ran: signed receipts, replay verdicts, model-signature checks | Ed25519- or ML-DSA-44-signed chained receipts, VERIFIED / DIVERGED / UNVERIFIABLE replay, OpenSSF Model Signing verification of the loaded GGUF | Not claimed | Not claimed |
| Vulkan, TLS, remote bind | Not implemented. Loopback only by design; put a reverse proxy in front for anything else | Vulkan available | Not compared |
Speed rows: MIG 1g.24gb slice, runner 6d705e9 vs llama.cpp b10353, 2026-09-02. Truncation row: 2026-08-19, granite-4.1-3b, versions named on the evidence page. Idle rows: 2026-09-01 and the README's M1 measurement. Sources: docs/benchmarks.md, docs/truncation-benchmark.md, docs/idle-coexistence-120b-m5max-2026-09-01.md.
If the machine is dedicated to inference and you want the fastest prefill and the widest model list, llama.cpp is the right call and we mean that sincerely. If you need an agent loop that finishes on a small budget, adapters trained on the exact file you deploy, receipts that a third party can replay, or an engine that can live on your workstation all day, that is what Runner is for.
Proof
We tested this. Here is what happened.
Three of the measurements the project rests on. Each figure names its source document and date; the raw responses, scripts and pinned files are in the repository, and the regression gates run on every release.
tool_choice: required, temperature 0, max_tokens from 1 to 64. The 64-token control proves the failure is truncation, not misconfiguration: every engine completes there. Runner 0.1.19-alpha, vLLM 0.27.1, llama.cpp b10488, Ollama 0.32.14, TensorRT-LLM 1.2.1, SGLang 0.5.17. † measured on a Qwen3-1.7B substitute because those registries did not carry granite-4.1-3b; recovery is a property of the runtime, so the substitution measures the engine. Measured 2026-08-19. Raw responses: tests/torture/truncation/. Method: docs/truncation-benchmark.md.--lora takes the score from 0.69 to 1.00, and the same 1.00 is reproduced in stock llama.cpp. Merging into F16 or Q8_0 keeps it. Merging into the 4-bit base erases it: 98.55% of weight bytes round back to the base's own codes. Measured 2026-08-22. Artifact: Qwen3-4B-Runner-ToolUse-Q4_K_M. Method: docs/adaptation-engine.md.Three reasons to use Runner
Inspectable, trainable, unbreakable under a budget.
Evidence and reproducibility
Results you can inspect, test and replay rather than merely generate. Recorded runs are signed and chained; a one-exit-code verifier replays them; the determinism claim has named edges, including what is deliberately not promised.
Receipts, replay tiers and the scope →LoRA training on the file you serve
Train adapters directly through the frozen quantized GGUF, with no FP16 training copy. Same data, seed and config write a byte-identical adapter with a provenance sidecar. Measured 0.69 to 1.00 on a public artifact, with the merge caveat stated beside it.
The measured claim and how to run it →Tool calls that survive the token limit
When the budget expires mid-call, Runner emits the smallest schema-legal ending so the arguments still parse and the agent continues. Five other engines were measured on the same rungs; none of them closes the document.
The problem, the mechanism, the six-engine table →Published artifacts
Sixteen repositories on Hugging Face, gated the same way.
Nine carry weights, seven are measurement reports over third-party files with no weights republished. A derivative is published only after its stated gate against the named parent, and a near-miss stays published with its numbers in the open rather than being quietly removed.
- passes barAttention Q8_0, experts Q4_0, 17.99 GB. Passes the fidelity bar where the official uniform Q4_K_M fails it.
- reproducedThe LoRA behind the 0.69 to 1.00 claim, produced twice with the same sha256; the card carries the sidecar and the training command.
- KLD 0.0223First sublayer-removal artifact: one attention sublayer physically dropped, 64 MiB of KV cache freed at 4k context, KLD 0.0223 on 44,413 held-out positions.
- fails current barExpert-pruned derivative whose originally published number did not reproduce. Kept published with the measured status at the top of the card.