Runner v1.4.2
One file that runs AI models on your machine, and shows its work.
Runner loads an AI model file and answers with it, on your own computer. Written from scratch in plain C, nothing to install underneath it, on ordinary processors, NVIDIA graphics cards and Apple Silicon. Your apps and coding agents talk to it through the OpenAI and Anthropic APIs.
It answers the questions other engines leave to trial and error: will this model fit, what did shrinking it cost, and what exactly did it say. From 1.0 its interfaces are held to a versioning policy.
Real commands, real output. The numbers are on this page with their dates: fit, shrinking cost, receipts.
--kv k8v4, full agreement with the full-size cache on three modelsTry it
Sixty seconds to a served model
Download the binary and its checksum file, verify, fetch a model, serve. The checksum line is not decoration: this project's whole culture is receipts, and it starts at the download.
The model below, granite-4.1-3b at Q8_0, is the smallest that passes the project's fidelity gate against its own BF16 parent. Alternatives and the reasoning are in the manual's model choices.
curl -LO https://github.com/Joakimpalm-Zen/xyntetik-runner/releases/latest/download/runner-macos-arm64
curl -LO https://github.com/Joakimpalm-Zen/xyntetik-runner/releases/latest/download/SHA256SUMS
shasum -a 256 --check --ignore-missing SHA256SUMS
chmod +x runner-macos-arm64 && mv runner-macos-arm64 runner
curl -L -o model.gguf https://huggingface.co/ibm-granite/granite-4.1-3b-GGUF/resolve/main/granite-4.1-3b-Q8_0.gguf
./runner -m model.gguf --servecurl localhost:8080/v1/chat/completions \
-d '{"messages":[{"role":"user","content":"Say hello in one sentence."}]}'The asset is runner-linux-x86_64; check with sha256sum -c --ignore-missing SHA256SUMS.
runner-windows-x86_64.exe, with a notification-area controller.
The binaries are not yet notarized. A curl download runs as shown; a browser download gets quarantined by Gatekeeper. Clear it with xattr -d com.apple.quarantine runner or right-click and Open once.
make builds the release binary with no dependencies beyond a C compiler; CUDA and Metal kernels are embedded at build time. Build and platforms.
A Dockerfile ships in the repository. Container image.
Runner is at 1.0 and hardware coverage is still limited; that is an invitation, not an apology. An unusual GGUF, a coding agent, an NVIDIA or Apple setup, or a model family not in the support matrix: the result is wanted, success or failure alike. Open an issue with runner --version, runner --caps, the model's exact filename and the load log. Independent reproductions of the determinism claims are credited in the docs, as the first one already is.
Will it fit
Know whether a model fits before you download it.
Finding out that a model does not fit usually means downloading 20 GB and watching it crash. The answer is already written in the first few megabytes of the file. Runner reads just that part and tells you, with the arithmetic.
curl -r 0-16777215 -L -o head.gguf \
https://huggingface.co/ORG/REPO/resolve/main/MODEL.gguf
./runner --fit head.gguf -c 16384On an RTX 3070 it named the same split the loader then used in 70 of 72 cases and was one layer cautious in the other two (six models, two context sizes, four cache formats, 2026-10-03). --caps says what the machine can run and --doctor loads a model, probes it and reports. How the verdict is computed
Make your own file
Shrink a model and see what it cost.
A model file is a huge table of numbers, and the usual way to shrink it squeezes every part the same amount, including the parts that matter most. Runner lets you choose which parts to squeeze and which to keep sharp, and then tells you how far the result drifted from the original.
A 17.99 GB Qwen3-30B file built this way is smaller than the official 4-bit file and closer to the original: 99.50% agreement where that file reads 94.75%. Plan formats and limits
{"default": "keep", "rules": [{"match": "_exps.weight", "type": "q4_0"}]}# keep some tensors sharp and squeeze others, from a plan file
./runner -m model-Q8_0.gguf --quantize out.gguf --type-plan plan.json
# drop the experts a mixture-of-experts model rarely uses
./runner -m model.gguf --quantize pruned.gguf --prune-experts keep.json
# remove one block's attention or feed-forward sublayer
./runner -m model.gguf --quantize cut.gguf --remove-sublayer attn:48
# then score the result against its parent
python3 scripts/kld-compare-raw.py --model-a out.gguf \
--model-b model-Q8_0.gguf --runner ./runner \
--corpus corpus.txt --max-positions 400 --out fidelity.jsonMore context, less memory
Fit more context in the same memory.
While a model reads a long document it keeps notes on every word so far, and on a long context those notes can take more memory than you have left. Runner can write the notes smaller. It keeps 8 bits for the half that decides where to look, which measurement showed is the half that matters, and shrinks the other half to 4 bits.
./runner -m model.gguf --serve -c 32768 --kv k8v4
./runner -m model.gguf --serve -c 32768 --kv q8 # the gentler setting, about half--kv k8v4 stores the cache in about 41% of the usual bytes. On Qwen3.5-4B, Muse-Glimmer-30B and Qwen3.8-27B it agreed with the full-size cache on 100% of clear-cut tokens over 500 positions each. What each setting costs, model by model
| while loaded and idle, 8 GB M1 | Runner | llama-server b10639 |
|---|---|---|
| wired (unevictable) memory | +8 MB | +3,819 MB |
| CPU wakeups per second | 0.5 | 161 |
| CPU time per idle minute | ~0.00 s | 0.2 to 0.3 s |
| give the memory back | POST /unload, 360 MB to 24 MB | kill the process |
| 128 GB Apple Silicon Mac, gpt-oss-120b (63 GB), 2026-09-01 | Runner 0.4.4 | llama-server 010be968 |
|---|---|---|
| wired memory while loaded and idle | +35 MB | +60.8 GB |
after /unload | RSS 229 MB | no unload |
granite-4.1-3b Q8_0 at c=4096, stock defaults, engines run sequentially, four-state lifecycle. Reproduce with scripts/idle_coexistence.py. Full 120B table and method: docs/idle-coexistence-120b-m5max-2026-09-01.md.
Designed to stay on
What a resident server does while it serves nothing
Benchmarks rarely show it, so it was measured. llama-server wires the whole model into unified memory, holds it until the process dies, and its idle loop ticks at about 160 Hz. Runner keeps weights as evictable zero-copy mappings the OS can reclaim when another app needs the RAM, wakes about once every two seconds, and hands everything back on /unload or automatically with --ttl.
What that costs is the first answer after the memory has been taken back, which waits for the model to be read in again: about 3 s for the 3.6 GB model on the 8 GB M1, about 10 s for the 63 GB model. Runner is built to be left running all day next to your actual work.
On a discrete GPU (RTX 3070) the footprint gap disappears, both engines hold about 4 GB of VRAM loaded, and the remaining differences are idle discipline and lifecycle control: /unload returned all 4 GB of VRAM; llama-server has no unload.
Also in the box
Everything else the one file does.
A shared GPU stops being first-come, first-crash
Runner processes on the same GPU share a VRAM registry. A refused load names every live holder by PID, model, bytes and uptime; --wait-for-vram turns the refusal into a bounded queue; records left by dead processes are reaped. A GPU becomes something you can schedule rather than something you hope fits.
Ask what fits before loading anything
--caps needs no model file and returns one JSON document with live RAM and VRAM, backend and GPU limits, admitted quant types and architectures, placement modes and model-count limits. --fit answers the same question for a specific file from its GGUF header, the first few megabytes, so a ranged read decides whether the rest of the download is worth starting. On a discrete GPU it computes the loader's own layer split for each KV cache format and judges RAM by what stays on the host: on an RTX 3070 it named the loader's exact layer count in 70 of 72 cases and one layer fewer in the other two.
Decisions you can gate
choice_logprobs records each JSON-schema branch as legal alternatives, a posterior renormalized over them, and the probed probability mass. confirm_below turns that into one answer per turn: the lowest posterior a chosen branch had, and whether it fell below your threshold. Served Llama-3.2-3B with three tools picked get_weather at 0.92 for a weather question and at 0.78 when send_email was wanted, and asked for confirmation at 0.9. One versioned contract, docs/decision-record.md.
Long agent turns stay on track
A reasoning budget closes an over-long reasoning turn with a short sentence in the model's own voice, so the answer keeps its whole budget; the loop guard closes a reasoning turn that has started repeating; a tool call cut short says which values the closer wrote rather than the model. Each is a serving control, named in the response telemetry when it fired.
Suspend, resume, fork
With --sessions DIR a generation can stop after a set number of tokens, be written to a session image, and resume later exactly where it stopped, or fork from the same point under a new seed. The image names the model, the binary and the constraint it ran under.
One report to attach
runner --doctor loads the model, runs one short probe and prints one JSON report: version, placement, template, sampler, memory, timings, and findings each with a next step. It holds no prompt or reply text unless you ask for it. --caps, --fit and --tool-info answer the narrower questions as JSON.
Smaller caches, longer contexts
KV caches in f16, q8, fp4 or the split k8v4 (K at q8_0, V at fp4, about 41% of f16). The 4-bit cost is not symmetric, so it is measured per model against the f16 cache and published: k8v4 reads a mean KLD of 0.002 to 0.004 with 100% margin-qualified top-1 on Qwen3.5-4B, Muse-Glimmer-30B and Qwen3.8-27B, and 0.04 on Qwen2.5-1.5B, where fp4 for both sides reads 2.00. An opt-in ring for sliding-window layers keeps only the rows the window can reach, bit-identical, taking gemma-3-4b at 32k from 4,563 MB to 800 MB of cache.
Models fetched and checked
-hf owner/repo[:TAG] downloads a GGUF from the Hugging Face Hub, verifies it against the Hub's SHA-256 record before it loads, and refuses a repository whose file list would write outside the cache.
Model surgery, with provenance
--quantize with per-tensor type plans, --prune-experts for stacked MoE layouts, --merge-lora with a provenance record, --context-surgery to compile a YaRN target into a native GGUF, and --remove-sublayer to physically drop a block's attention or FFN and declare the absence in the header. Each rewrite is deterministic and reparsed before success.
Speculative decoding from four draft sources
A draft model, an n-gram lookup, the model's own MTP head where a single-block export carries one (--mtp, token-identical, measured 1.31x on code on a Threadripper; the gain depends on the machine, and both measurements are published), and context-grounded drafts that need no weights at all.
Structured output as a sampler guarantee
--json emits one valid JSON object. JSON-Schema, Responses text.format and tool parameter schemas compile to a streaming conformance validator covering objects, arrays, enums, unions, numeric bounds, string lengths and anchored patterns. Unsupported or ambiguous constraints fail at compile time rather than being silently weakened.
What it promises
Six promises, each one tested.
Runner deliberately rides llama.cpp's formats rather than competing with them: GGUF in, llama.cpp-convention adapter files in and out. An adapter Runner trains scores identically, 1.000 on its held-out eval, when served by stock llama.cpp, and community F16 adapters load straight back. Both measured, not assumed. What it adds on top are promises the ecosystem does not make.
Determinism as a hard promise
The same executable and inputs reproduce the same sampled tokens across runs and thread counts. Training reproduces the same adapter file sha256. Both are gated in CI. Independent rebuilds are explicitly outside the byte-identity claim, because compilers and ISA libm differ, and the full boundary is one page: docs/determinism-scope.md.
Scope as a promise
Supported architectures are named. Unknown ones are refused by name rather than run on a guess. Every backend claim is tied to an executable gate and pinned model evidence, and a hardware switch has a correctness contract: CPU/GPU identity belongs to a SHA-256-pinned model and execution path, and faster kernels that reassociate sums pass tolerance gates rather than inherit correctness from the backend's name.
Evidence as an artifact
Every table names its method and its date. Experiments that were built, measured and rejected are kept as documents so they are not attempted twice. A published file whose number did not reproduce stays published with the measured status at the top of its card.
Concurrency without crosstalk
Parallel slots decode in one batched step, and on CUDA and Metal that step is gated to match decoding each request alone, byte for byte, including steps with more requests than one microbatch holds. The gate grew to eleven sequences after a 16-slot server in the lab showed what happens without it.
Records a reviewer can check
A served turn is a receipt; receipts chain; --export-pack gathers them with the model signature and the measured envelope into one evidence pack that --check-pack verifies offline. Receipts and the audit demo.
Compatibility as a promise
From 1.0 the command line, the exit codes, the HTTP API, the record formats and the model files that load stay compatible across 1.x. What is bound to the exact build, and what is not covered, is written down beside it. What 1.x keeps.
Runner 1.0
What 1.x keeps, and what it does not.
A 1.0 is a promise about interfaces, not about speed. Runner follows semantic versioning from 1.0.0: what this list keeps changes only at 2.0, and a flag that has to go is deprecated with a warning for the rest of 1.x first. The full policy is one page: docs/versioning.md.
Every flag runner --help documents and its meaning; the exit codes, including the --verify verdicts; the HTTP endpoints and request fields, with response fields only added, never renamed; every record format by its schema_version, so a 1.0 receipt still verifies under a later 1.x build; every architecture and quantization format that loads today; the Python and TypeScript client APIs.
Replay agreement, session images and measured envelopes belong to the exact binary that produced them. That was never a cross-version promise, and the envelope gate matches the version exactly so it cannot become one by accident.
Generated tokens can change between releases when a kernel or sampler is fixed, and the CHANGELOG says so. Speed figures, the support matrix and certification verdicts follow the evidence. RUNNER_* switches and log text are not interfaces; the JSON outputs are.
The latest 1.x release is supported; there are no backports. Security reports go through SECURITY.md.
Support matrix
Named, gated, and refused by name outside the list.
runner --caps publishes the architecture IDs the current binary admits. The full per-family notes, with the measured evidence behind each row, are in the manual's support matrix. Summary as of v1.4.2:
CPU (x86 AVX2/FMA, ARM NEON), CUDA (driver API, embedded PTX), Metal (embedded shaders, unified memory). Linux x86_64, macOS arm64, Windows x86_64 release binaries.
llama, mistral, smollm, stablelm, qwen2, qwen3, qwen35 (Gated DeltaNet hybrid), qwen4exp (experimental; placement/context and reference limits in the manual), qwen3moe, gemma3, gemma4 (dense, MoE, E-series, native tool protocol), phi3, gpt-oss, apertus, afmoe, muse-glimmer, granite, granitehybrid and nemotron_h (Mamba-2 hybrids), nemotron_h_moe. Admission is layout-specific: an unsupported tensor arrangement is refused even when the architecture ID is listed.
GGUF v2/v3, mmap host weights, standard multi-part sets. k-quants and legacy quants, MXFP4, NVFP4 with its per-tensor scale companion, the IQ1 to IQ4 codebook families. CUDA serves all of them, NVFP4 with the export's per-tensor scale companion applied in the kernel and the IQ1 to IQ3 codebook types with device twins of the CPU decoders; Metal serves the IQ1 to IQ3 families with its own kernels too, and only NVFP4 stays on the CPU there. A backend without the kernel refuses loudly, naming the tensor and type.
OpenAI Chat Completions, Responses, legacy completions, embeddings and rerank, Anthropic Messages, /v1/decide, SSE streaming, parallel slots, model swap and unload, prefix reuse, reasoning budgets and the loop guard, Prometheus /metrics, native tool protocols for gemma4, granite4, Apertus, Harmony, atem, Muse recipients, Qwen's JSON calls and the function-XML families (Ornith, Granite 4.2, Qwen 3.8, Qwen3-Coder), each rendered from the publisher's template and checked token for token. A GGUF's own sampling defaults are used when it carries them.
LoRA through the quantized base on the CPU, and with the optional CUDA assist (RUNNER_TRAIN_GPU=1) that is the faster arm on a consumer card: Qwen2.5-7B Q4_K_M at 64.8 s per step against 76.5 on eight CPU threads, RTX 3070, with the adapter bytes gated identical either way. Adapters served with --lora on the CPU, CUDA and Metal; --adapt-info says whether a model can take one before you train; --train-dpo trains from preference pairs against the frozen base.
Replay-verifiable transcripts; Ed25519- or ML-DSA-44-signed, chained receipts with a one-exit-code verifier; evidence packs that gather receipts, the model signature and the measured envelope for offline review; OpenSSF Model Signing verification of the loaded GGUF by the key method; session images that suspend, resume and fork a generation.
macOS menu bar and Windows notification-area controller. Release binaries for Linux, macOS and Windows with a SHA256SUMS file, and a container image on GHCR.
Vulkan; TLS, auth and remote bind (loopback only, by design); remote or streamed GGUF parts; qwen2moe, deepseek2, kimi and MLA attention; Mamba and Jamba; MTP on GPU backends; OMS certificate and keyless signature methods; full GBNF; image and document inputs; hosted tools; response persistence; parallel tool calls on the Responses and Messages surfaces.
Coding agents
Client compatibility is a dated observation, not an inference from an API name.
Current endpoint choices, tool/context limits, tested workflow instructions, and the scope of installed-client evidence are in the coding-agent guide. Library smoke tests run with pinned SDK dependencies; their success does not certify every newer client or model task.
# ~/.codex/config.toml
model = "runner"
model_provider = "runner"
[model_providers.runner]
name = "Xyntetik Runner"
base_url = "http://127.0.0.1:8080/v1"
wire_api = "responses"
env_key = "RUNNER_API_KEY"export RUNNER_API_KEY=none
./runner -m model.gguf --serve -c 16384
codex "list the files here"Codex's system prompt and tools consume roughly 10k input tokens before the user request, so use at least a 16k context. Runner does not implement a response store; clients send history each turn.
src/gguf.c GGUF metadata and tensor-table parser
src/tokenizer.c SPM/BPE tokenization and family pre-tokenizers
src/quants.c scalar, AVX2/FMA, and NEON quantized dot kernels
src/model.c tensor admission, weight wiring, and forward pass
src/sample.c sampling filters and token selection
src/jsonmode.c incremental JSON-prefix validation
src/schema.c JSON-Schema compiler and streaming validator
src/schema_tools.c native tool-call grammars for each model family
src/template.c chat templates, thinking channels, and tool syntax
src/engine.c prompt feeding, prefix cache, constrained sampling, speculative decode
src/quantize.c requantization and stacked-MoE expert pruning
src/cuda.c CUDA driver backend; kernels.cu becomes embedded PTX
src/metal.m Metal backend; kernels.metal is embedded at build time
src/server.c loopback HTTP server, slots, routing, and lifecycle
src/vramreg.c cross-process VRAM ownership and bounded waiting
src/ed25519.c signing subset of TweetNaCl (receipts)How it is built
Readable in an afternoon.
Host weights stay quantized in the mapped GGUF and are dequantized while computing. CUDA copies selected weights and compute buffers to VRAM; Metal wraps the mapped weights in unified memory. The load log and --caps are the sizing sources for an exact model and machine.
The complete map, the command-line reference and the compatibility evidence are in the manual; release history with each version's measurements is in the changelog.