Xyntetik Runner

Xyntetik Runner is a single-binary GGUF inference and LoRA training engine written from scratch in plain C: one binary serves, verifies, scores, adapts and trains GGUF models on CPU (x86 AVX2/FMA, ARM NEON), CUDA and Metal. It trains LoRA adapters directly through the same quantized GGUF weights it serves, with no separate FP16 training copy, and two runs with the same data, seed and config write a byte-identical adapter file. Every claim is tied to a measurement you can re-run, and every recorded run is a signed receipt that replays. No Python, no pip, no third-party runtime, no ggml. One caveat travels with the training claim: a quantized merge rounds the delta, and merging an adapter into a 4-bit base erases the fine-tune (8-bit and F16 keep it), so serve adapters with --lora or merge into Q8_0 or better.

Three questions this engine answers, one page each: train a LoRA on the quantized GGUF you serve, reproducible LoRA training with receipts, tool calls that survive the token limit.

Sixty seconds to a served model

curl -LO https://github.com/Joakimpalm-Zen/xyntetik-runner/releases/latest/download/runner-macos-arm64
curl -LO https://github.com/Joakimpalm-Zen/xyntetik-runner/releases/latest/download/SHA256SUMS
shasum -a 256 --check --ignore-missing SHA256SUMS
chmod +x runner-macos-arm64 && mv runner-macos-arm64 runner
curl -L -o model.gguf https://huggingface.co/ibm-granite/granite-4.1-3b-GGUF/resolve/main/granite-4.1-3b-Q8_0.gguf
./runner -m model.gguf --serve

curl localhost:8080/v1/chat/completions \
  -d '{"messages":[{"role":"user","content":"Say hello in one sentence."}]}'

Linux: the asset is runner-linux-x86_64 and the check command is sha256sum -c --ignore-missing. Windows: runner-windows-x86_64.exe. The checksum line is not decoration - this project’s whole culture is receipts, and it starts at the download. The model above is the smallest that passes this project’s fidelity gate against its own BF16 parent; alternatives and the reasoning are in the quick start below.

macOS note: the binaries are not yet notarized. A curl download runs as shown; a browser download gets quarantined by Gatekeeper - clear it with xattr -d com.apple.quarantine runner or right-click → Open once.

Testing Runner? The project is pre-1.0 and hardware coverage is still limited - that is an invitation, not an apology. If you have an NVIDIA/Apple setup, an unusual GGUF, a coding agent, or a model family not in the support matrix, the result is genuinely wanted, success or failure alike: open an issue with runner --version, runner --caps, the model’s exact filename and the load log. Independent reproductions of the determinism claims get credited in the docs, as the first one already is.

Contents: try it · why Runner · what it adds · training · models · APIs · support matrix · CLI reference

Xyntetik Runner is independent and bootstrapped: the engine is free forever under Apache 2.0 - consulting and enterprise work fund the hardware. Built in Sweden, runs on your hardware; your data never leaves the building.

Why this and not llama.cpp?

Use llama.cpp - it is the ecosystem, and Runner deliberately rides its formats rather than competing with them: GGUF in, llama.cpp-convention adapter files in and out. An adapter Runner trains scores identically (1.000 on its held-out eval) served by stock llama.cpp, and community F16 adapters load straight back into Runner - both measured, not assumed.

What Runner adds is not a longer feature list; it is a set of contracts the ecosystem does not make. Determinism as a hard promise: the same executable and inputs reproduce the same sampled tokens across runs and thread counts, and training reproduces the same adapter file sha256, gated in CI. Independent rebuilds are explicitly outside that byte-identity claim because compiler and ISA libm can differ; the full claim boundary, including everything deliberately NOT promised, is one page: docs/determinism-scope.md. Scope as a promise: supported architectures are named, unknown ones are refused, and every backend claim is tied to an executable gate and pinned model evidence. Honesty as an artifact: the benchmark tables below include the rows where Runner loses, and the docs keep the failed experiments. If you want maximum architecture coverage and raw speed, use llama.cpp and we mean that sincerely. If you need to prove what your model said, what it learned from, or what you actually shipped - that is what this runtime is for.

Designed to stay on

There is a cost benchmarks rarely show: what a resident inference server does to the machine while it serves nothing. We measured it - four-state lifecycle (loaded idle, post-inference idle, after unload), granite-4.1-3b Q8_0 at c=4096, stock defaults, engines run sequentially, llama.cpp from the prebuilt b10639 release. Reproduce it with scripts/idle_coexistence.py.

On an 8 GB M1 (quiet machine), while loaded and idle:

llama-server wires the whole model into unified memory and holds it until the process dies, and its idle loop ticks at ~160 Hz. Runner keeps weights as evictable zero-copy mappings the OS can reclaim whenever another app needs the RAM, wakes about once every two seconds, and hands everything back on /unload (or automatically with --ttl). The flip side is real and we report it: when llama-server is allowed to hold everything, its time to first token is faster (0.15 s vs 3.2 s on the pressured M1), because residency is exactly what it buys. On a discrete-GPU box (RTX 3070) the footprint gap disappears - both engines hold ~4 GB of VRAM loaded - and the remaining differences are idle discipline (0.00 vs 0.3-0.6 CPU seconds per minute) and lifecycle control (/unload returned all 4 GB of VRAM; llama-server has no unload).

These are different optimization goals, not a defect: llama-server is built to answer the next request as fast as possible, Runner is built to be left running all day next to your actual work. If the machine is dedicated to inference, that residency is pure win - use llama.cpp. If the machine is also your workstation, an engine that wires down half your RAM and ticks 160 times a second while idle is the reason you kill it every time - and the reason Runner does not need killing.

The same lifecycle measured on a 128 GB M5 Max with gpt-oss-120b (63 GB) says abundance does not dissolve the difference, it scales it: loaded idle, llama-server (010be968) wires +60.8 GB — half the machine held by an idle process — against Runner’s +35 MB, and /unload hands everything back (RSS 229 MB). The latency flip side scales identically: 0.25 s vs 10.1 s cold first token. Full table and method: docs/idle-coexistence-120b-m5max-2026-09-01.md.

For release history and benchmark narratives, see CHANGELOG.md and docs/benchmarks.md (last re-measured 2026-09-02 on three hosts: dense decode 81-93% of llama.cpp on the MIG, prefill 5-10%).

The rest of the README, from the command-line reference to the compatibility evidence, is on GitHub.