Xyntetik xyntetik

Evidence

Every claim, with the measurement behind it.

This page collects what has actually been measured, when, on what, and where the raw material lives. Results that went against the project are on it too. If a claim you find elsewhere is not backed here or in the repository, treat it as unproven.

Truncation recovery

Six engines, seven token budgets, one that closes the document.

Claim under test: tool calls survive the token limit. Everything that could move a verdict is pinned as data in the probe: the tool schema (get_weather(city, units), both required, no additional properties), the prompt, tool_choice: required, temperature 0, budgets 1, 2, 3, 5, 8, 16 and 64. The 64-token control must complete on every engine, which proves the small rungs measure truncation and not misconfiguration.

executable tool call, arguments parse no usable call
Per token budget, does the client receive an executable tool call123581664 ctrlRunnerRunner, max_tokens 1: executable tool call, arguments parseRunner, max_tokens 2: executable tool call, arguments parseRunner, max_tokens 3: executable tool call, arguments parseRunner, max_tokens 5: executable tool call, arguments parseRunner, max_tokens 8: executable tool call, arguments parseRunner, max_tokens 16: executable tool call, arguments parseRunner, max_tokens 64: executable tool call, arguments parsevLLM 0.27.1vLLM 0.27.1, max_tokens 1: no usable callvLLM 0.27.1, max_tokens 2: no usable callvLLM 0.27.1, max_tokens 3: no usable callvLLM 0.27.1, max_tokens 5: no usable callvLLM 0.27.1, max_tokens 8: no usable callvLLM 0.27.1, max_tokens 16: no usable callvLLM 0.27.1, max_tokens 64: executable tool call, arguments parsellama.cpp b10488llama.cpp b10488, max_tokens 1: no usable callllama.cpp b10488, max_tokens 2: no usable callllama.cpp b10488, max_tokens 3: no usable callllama.cpp b10488, max_tokens 5: no usable callllama.cpp b10488, max_tokens 8: no usable callllama.cpp b10488, max_tokens 16: tool_calls present, arguments do not parse{ "llama.cpp b10488, max_tokens 64: executable tool call, arguments parseOllama 0.32.14Ollama 0.32.14, max_tokens 1: no usable callOllama 0.32.14, max_tokens 2: no usable callOllama 0.32.14, max_tokens 3: no usable callOllama 0.32.14, max_tokens 5: no usable callOllama 0.32.14, max_tokens 8: no usable callOllama 0.32.14, max_tokens 16: HTTP 500500Ollama 0.32.14, max_tokens 64: executable tool call, arguments parseTensorRT-LLM 1.2.1 †TensorRT-LLM 1.2.1 †, max_tokens 1: no usable callTensorRT-LLM 1.2.1 †, max_tokens 2: no usable callTensorRT-LLM 1.2.1 †, max_tokens 3: no usable callTensorRT-LLM 1.2.1 †, max_tokens 5: no usable callTensorRT-LLM 1.2.1 †, max_tokens 8: no usable callTensorRT-LLM 1.2.1 †, max_tokens 16: no usable callTensorRT-LLM 1.2.1 †, max_tokens 64: executable tool call, arguments parseSGLang 0.5.17 †SGLang 0.5.17 †, max_tokens 1: no usable callSGLang 0.5.17 †, max_tokens 2: no usable callSGLang 0.5.17 †, max_tokens 3: no usable callSGLang 0.5.17 †, max_tokens 5: no usable callSGLang 0.5.17 †, max_tokens 8: no usable callSGLang 0.5.17 †, max_tokens 16: no usable callSGLang 0.5.17 †, max_tokens 64: executable tool call, arguments parse
Measured 2026-08-19 on granite-4.1-3b Q4_K_M (Runner and llama.cpp on CPU, Ollama on GPU from the same GGUF, vLLM from the HF safetensors on GPU). † TensorRT-LLM and SGLang on a Qwen3-1.7B substitute because their registries did not carry granite; SGLang on an RTX 3070 under WSL2. Runner c7d6c3d (0.1.19-alpha), vLLM 0.27.1 with xgrammar 0.2.3, llama.cpp b10488, Ollama 0.32.14, TensorRT-LLM 1.2.1, SGLang 0.5.17 with xgrammar 0.2.1.
What each engine hands the caller, rung by rung
max_tokensRunnervLLMllama.cppOllamaTensorRT-LLM †SGLang †
1parsesnone; leaks <tool_call> into contentnone; leaks <tool_call> into contentnone; empty contentnone; leaks bare tagnone; leaks bare tag
2parsesnone; leaknone; leaknone; emptynone; emptynone; empty
3parsesnone; leaks {"none; leaks {"none; emptynone; emptynone; empty
5parsesnone; leaks {"name":none; leaks {"name":none; emptynone; emptynone; empty
8parsesnone; leaks {"name": "get_weathernone; leaks {"name": "get_weathernone; emptynone; emptynone; empty
16parsesnone; leaks {"city": "tool_calls present, arguments {"city": " do not parseHTTP 500none; emptynone; empty
64parses, completesparses, completesparses, completesparses, completesparses, completesparses, completes

The property is an engine guarantee, grammar plus closer, not model quality: it holds identically for the random two-layer CI fixture and for granite-4.1-3b, which is why make test-truncation can run on any CPU with no GPU and no competitor, and does on every release. The claim covers the engines and versions measured, not engines that were not. Method, the substitute-model notes and the raw responses: docs/truncation-benchmark.md, tests/torture/truncation/2026-08-19-granite-4.1-3b/. The agent-torture gate tests the same failure inside multi-turn loops.

Throughput

Against llama.cpp, same files, same box, losing rows included.

One machine, one method, both engines on the same GGUF files. Runner via --bench-json -n 128 -b 64 (mean of 2), llama.cpp via llama-bench -p 512 -n 128 -ngl 99 -r 2, full GPU offload verified on every row. Absolute numbers depend on the hardware; the ratios are the result, and even those shift with the compute-to-bandwidth balance.

Runner throughput as a percentage of llama.cpp, decode and prefillDecode tokens/s, runner as % of llama.cpp0%25%50%75%100%Llama-3.2-3B Q4_K_MLlama-3.2-3B Q4_K_M: runner 87.9 vs llama.cpp 101.8 tok/s (86%)86%Phi-4-mini Q8_0Phi-4-mini Q8_0: runner 53.8 vs llama.cpp 57.7 tok/s (93%)93%granite-3.3-8b Q4_K_Mgranite-3.3-8b Q4_K_M: runner 39.7 vs llama.cpp 45.4 tok/s (87%)87%granite-4.1-8b Q4_0granite-4.1-8b Q4_0: runner 41.7 vs llama.cpp 46.5 tok/s (90%)90%Phi-3.5-mini Q4_K_MPhi-3.5-mini Q4_K_M: runner 72.0 vs llama.cpp 83.2 tok/s (87%)87%gemma-4-12B Q4_K_Mgemma-4-12B Q4_K_M: runner 23.2 vs llama.cpp 28.8 tok/s (81%)81%Qwen3-30B-A3B Q4_K_M (MoE)Qwen3-30B-A3B Q4_K_M (MoE): runner 62.2 vs llama.cpp 86.6 tok/s (72%)72%gemma-4-26B-A4B Q4_0 (MoE)gemma-4-26B-A4B Q4_0 (MoE): runner 23.3 vs llama.cpp 56.2 tok/s (41%)41%Qwen2.5-32B Q3_K_SQwen2.5-32B Q3_K_S: runner 1.8 vs llama.cpp 14.3 tok/s (13%)13%Prefill tokens/s, runner as % of llama.cpp0%25%50%75%100%Llama-3.2-3B Q4_K_MLlama-3.2-3B Q4_K_M: runner 525.7 vs llama.cpp 6387.6 tok/s (8%)8%Phi-4-mini Q8_0Phi-4-mini Q8_0: runner 355.6 vs llama.cpp 5960.6 tok/s (6%)6%granite-3.3-8b Q4_K_Mgranite-3.3-8b Q4_K_M: runner 220.5 vs llama.cpp 2288.9 tok/s (10%)10%granite-4.1-8b Q4_0granite-4.1-8b Q4_0: runner 151.1 vs llama.cpp 2359.2 tok/s (6%)6%Phi-3.5-mini Q4_K_MPhi-3.5-mini Q4_K_M: runner 284.9 vs llama.cpp 5120.3 tok/s (6%)6%gemma-4-12B Q4_K_Mgemma-4-12B Q4_K_M: runner 123.7 vs llama.cpp 1439.2 tok/s (9%)9%Qwen3-30B-A3B Q4_K_M (MoE)Qwen3-30B-A3B Q4_K_M (MoE): runner 128.5 vs llama.cpp 2326.3 tok/s (6%)6%gemma-4-26B-A4B Q4_0 (MoE)gemma-4-26B-A4B Q4_0 (MoE): runner 34.3 vs llama.cpp 2429.7 tok/s (1%)1%Qwen2.5-32B Q3_K_SQwen2.5-32B Q3_K_S: runner 15.3 vs llama.cpp 469.0 tok/s (3%)3%
MIG 1g.24gb slice of an RTX PRO 6000 Blackwell, runner 6d705e9 vs llama.cpp b10353 CUDA, measured 2026-09-02. The box carried a load average of 18 from other work during this run, which lowered both sides' absolute numbers against August; the ratios are the result. Source: docs/benchmarks.md.
Tokens per second, all three hosts
host, 2026-09-02modeldecode runner / llama.cppprefill runner / llama.cpp
MIG 1g.24gb, llama.cpp b10353Llama-3.2-3B Q4_K_M87.9 / 101.8 (86%)525.7 / 6387.6 (8.2%)
Phi-4-mini Q8_053.8 / 57.7 (93%)355.6 / 5960.6 (6.0%)
granite-3.3-8b Q4_K_M39.7 / 45.4 (87%)220.5 / 2288.9 (9.6%)
granite-4.1-8b Q4_041.7 / 46.5 (90%)151.1 / 2359.2 (6.4%)
Phi-3.5-mini Q4_K_M72.0 / 83.2 (87%)284.9 / 5120.3 (5.6%)
gemma-4-12B Q4_K_M23.2 / 28.8 (81%)123.7 / 1439.2 (8.6%)
Qwen3-30B-A3B Q4_K_M (MoE)62.2 / 86.6 (72%)128.5 / 2326.3 (5.5%)
gemma-4-26B-A4B Q4_0 (MoE)23.3 / 56.2 (41%)34.3 / 2429.7 (1.4%)
Qwen2.5-32B Q3_K_S1.8 / 14.3 (13%)15.3 / 469.0 (3.3%)
RTX 3070 8 GB, Windows 11, llama.cpp b10754 CUDA 13.3Qwen2.5-7B Q4_K_M65.6 / 81.6 (80%)112.3 / 2224.5 (5.0%)
granite-4.1-3b Q8_078.6 / 94.3 (83%)493.5 / 5614.1 (8.8%)
Qwen2.5-3B Q4_K_M113.3 / 161.2 (70%)234.0 / 6069.9 (3.9%)
Apple M1 8 GB, Metal, llama.cpp Homebrew ggml 0.22.0granite-4.1-3b Q8_011.6 / 14.4 (80%)47.8 / 266.5 (17.9%)
gemma-3-4b Q4_K_M8.4 / 20.3 (41%)62.9 / 250.3 (25.1%)

Decode: parity is the physical target

Single-stream decode is memory-bandwidth-bound on the slice; both engines read the same quantized weights per token. Dense decode moved from 73 to 79% in July to 81 to 93% in September through decode GEMV work and a KV-ring accounting fix.

Prefill: llama.cpp's win, published as such

llama.cpp's prefill comes from a mature tensor-core GEMM stack across every quant. Runner's tensor-core path is held to a teacher-forced tolerance gate (0/64 top-1 flips, at most 0.012% mean logit deviation on every promoted row) and its tile is 64 columns wide. The gap is reported, not hidden.

The M1 rows show the coexistence trade from the losing side

Measured with 1.3 GB of RAM available, Runner logged its own eviction warning and paged while llama.cpp wired the model in. The gemma row is that effect, not a kernel result, and the document says to re-measure on a quiet machine before reading it as one.

A July table in the same document was taken before a CUDA prefill correctness bug (uninitialized shared memory for token columns 16 to 63) was found; it is kept as history and superseded by the August re-measurement on the fixed kernel. The benchmark's own control run also found one silent MoE GPU-to-CPU fallback, since fixed and guarded by a test.

Quantization fidelity

Constrained decoding guarantees the shape of a tool call at any quantization. Not its contents.

A full quant ladder, one Q8_0 reference against several variants, 16 agent-torture requests at temperature 0, with a zero-point self-check that two independent reference spawns agree on every call at 0 KLD before anything is measured. Three axes: does the output validate against the schema, did the variant call the same function as the reference, and do the parsed arguments match exactly.

granite-4.1-3b, argument agreement Hermes-4-14B, argument agreement schema conformance and tool selection, both families
Tool-call fidelity down the quantization ladder: shape holds at 100 percent, argument agreement decays40%60%80%100%Q8_0Q6_KQ5_K_MQ4_K_MQ4_0schema conformance and tool selection: 100% on every rung, both familiesgranite-4.1-3b Q8_0: argument agreement 100.0%granite-4.1-3b Q6_K: argument agreement 64.3%granite-4.1-3b Q5_K_M: argument agreement 57.1%granite-4.1-3b Q4_K_M: argument agreement 57.1%granite-4.1-3b Q4_0: argument agreement 50.0%granite-4.1-3b50% at Q4_0Hermes-4-14B Q8_0: argument agreement 100.0%Hermes-4-14B Q6_K: argument agreement 78.6%Hermes-4-14B Q5_K_M: argument agreement 78.6%Hermes-4-14B Q4_K_M: argument agreement 64.3%Hermes-4-14B Q4_0: argument agreement 50.0%Hermes-4-14B50% at Q4_0
granite-4.1-3b measured 2026-08-14; Hermes-4-14B (bartowski's ladder, hashes verified against the source repository) measured 2026-08-15. Schema conformance and tool selection held at 100% on every rung of both ladders; argument agreement decayed to 50% at Q4_0 on both. Source: docs/quant-fidelity.md.

Whole-model fidelity has a size threshold

Mean KLD against the model's own reference at Q4_K_M, under the adopted bar (margin-qualified top-1 at least 97% and mean KLD at most 0.05 against the named parent, 400 teacher-forced positions). Below about 8B parameters, uniform 4-bit fails the bar; from 8B up it passes. That threshold, and not a blanket "4-bit is unshippable", is what the ladder found.

model at Q4_K_Mparamsmean KLDverdict
gemma-4-E2B~5B MoE, 2B active0.3624fail
granite-4.1-3b3B0.1376fail
Phi-4-mini3.8B0.1258fail
Qwen3-8B8B0.0419pass
Hermes-4-14B14B0.0279pass

Training

One base, one task, every number on a public artifact.

Held-out exact tool call score before and after training, and after merging0.000.250.500.751.00Base Qwen3-4B Q4_K_MBase Qwen3-4B Q4_K_M: exact-call score 0.690.69Base + adapter (--lora)Base + adapter (--lora): exact-call score 1.001.00Merged into F16Merged into F16: exact-call score 1.001.00Merged into Q8_0Merged into Q8_0: exact-call score 1.001.00Merged into Q4_0Merged into Q4_0: exact-call score 0.690.69 erased
Held-out exact tool-call score, Qwen3-4B. Measured 2026-08-22. Artifact: Qwen3-4B-Runner-ToolUse-Q4_K_M; design, gates and failure modes in docs/adaptation-engine.md.
base

Qwen3-4B Q4_K_M, the frozen 4-bit serving weights

training path

directly through the quantized inference artifact, on the CPU

held-out exact call

0.69 before, 1.00 after

reproducibility

two independent runs, same sha256 on the adapter file

precision study

adapters trained through BF16 and Q8_0: cosine 0.9998; through Q4_K_M: 0.9926. Measurably different objects, capability-equivalent on this task's supervised decisions. An external 36-prompt boundary bank finds 3 of 36 prompts where the three adapters choose different tools; a property of that bank, not a rate

neutral-corpus drift

nll per token 4.063 to 4.026, unrelated text left alone

merge study

into Q8_0 or F16 keeps the 1.00 (verified in stock llama.cpp); into the 4-bit base erases it, 0.69 again, 98.55% of weight bytes round back to the base's codes. Survival is monotone in delta magnitude: erased through 2x, partial at 4x, full at 8x

interop

the adapter scores the same 1.00 served by stock llama.cpp; a community F16 adapter loads back into Runner, which also found and fixed the F32-only loader gap

scope

one base, one task. The 8B and 14B ladder is open work and these numbers are not a promise about other families

Terminal recording: two independent training runs print the same adapter sha256
Two independent training runs producing byte-identical adapters. Recording from the repository README.
Terminal recording: merging the adapter into the 4-bit base returns the base score; the 8-bit merge keeps it
Merging the adapter into the 4-bit base erases it; 8-bit keeps it. Recording from the repository README.

Receipts

A recorded run is a signed record, and a verifier answers with one exit code.

--transcript records model, adapter and binary hashes, the effective execution profile, seed and sampling configuration, prompt and output token ids and the exact streamed output bytes. --verify replays it: VERIFIED (0), DIVERGED at a token or output byte (2), UNVERIFIABLE for an invalid record or an artifact mismatch (3). Receipts chain, are signed with an Ed25519 key you generate, and the loaded model can be verified against an OpenSSF Model Signing bundle.

Receipts and the determinism scope
Claimed, gated in CI

Same executable, same inputs, same sampled tokens, across runs and thread counts. Training byte-determinism on the same build. Quantize and merge write the same bytes for the same inputs on the same build.

Replay tiers

T1, same binary: bit-exact replay. T2, cross-ISA: token-level replay on another machine; floating-point internals may differ and the boundary is libm.

Explicitly not claimed

Independent rebuilds are not byte-identical. Cross-engine token identity is not promised. Numerical identity of hidden states is never promised. Timing and throughput are not part of the contract.

First external reproduction

Adapter byte-equality was reproduced independently on a Tesla T4 by a Hugging Face community tester and is credited in the docs.

Compatibility gates

Pinned files, declared checks, executed on every release.

The machine-readable manifest tests/compatibility/models.json pins files by SHA-256 and declares each check independently. The release check refuses a report with zero executed rows, and the CUDA smoke gate on a second device family is mandatory before a tag.

load

The pinned file hashes and loads.

tokenizer

The committed 721-string corpus is compared with the model's Hugging Face tokenizer; revision-bound captures make declared rows replayable offline.

greedy_reference

Greedy tokens are compared with a pinned llama.cpp revision.

cpu_cuda

CPU and CUDA scalar-path greedy output are compared; byte-identity is the contract on the scalar path.

chat

A real Chat Completions request answers through the model template.

Cross-engine identity, stated with its envelope

Nemotron-Nano-9B-v2 Q8_0: 5 of 6 greedy completions byte-identical to llama.cpp b10353, the miss a single-token near-tie. granite-4.0-h-small: token-identical on deterministic prompts, divergences at the quantization noise floor. Nemotron-3.5-Lightning-30B: 4 of 5. Each envelope is published with the artifact, not rounded up.

Method: docs/compatibility-program.md; reports: docs/compat-reports/; measured-envelope convention: README.

Artifact ledger

Published only after the stated gate. Kept published when the gate is missed.

Every fidelity claim below is measured under the adopted dual-column bar: margin-qualified top-1 at least 97% and mean KLD at most 0.05 against the named parent over 400 teacher-forced positions, zero point exact, with plain top-1 always reported beside it. Read each card's provenance before treating a derivative as equivalent to an original checkpoint.

  • Qwen3-30B-A3B selective precision
    Attention Q8_0, experts Q4_0, 17.99 GB, from a byte-verified first-party Q8_0 source. Passes the bar where the official uniform Q4_K_M fails it. Built with --type-plan; the plan is on the card. The artifact class the project leads with.
    passes bar
  • Qwen3-Coder-30B keep-120
    Expert-pruned, 17.5 GB. Passes both the original and the current bar; the only published artifact to clear the original bar unaided.
    passes bar
  • Qwen3-4B Runner ToolUse adapter
    The LoRA behind the training claim, produced twice with the same sha256. Card carries the sidecar and the training command.
    reproduced
  • gemma-4-31B-it, attention block 48 removed
    17.58 GB Q4_0. One attention sublayer physically dropped from Google's QAT Q4_0 release with --remove-sublayer; frees 64 MiB of KV at 4k and 512 MiB at 32k plus 74.3 MB of file; tracks its parent at KLD 0.0223 on 44,413 held-out positions. Loads only in Runner 0.4.7 and later, CPU path; llama.cpp refuses it by name, and the card says so.
    KLD 0.0223
  • Muse-Glimmer-30B surgical, Q4_K and Q8_0
    The decoder cut and healed (4.75% removed in the Q4_K file, 14.61 GB; 6.34% in the Q8_0, 27.58 GB), quantized by Runner's own quantizer. Both pass the bar against the BF16 parent (97.74% / 0.04810 and 98.34% / 0.03883). The Q4_K cost was predicted to 0.6% before the file was built. Provenance and checksums in the repository; the surgery report carries the method.
    passes bar
  • NVIDIA-Nemotron-Nano-9B-v2 Q8_0
    Not a bar-gated derivative: the first Mamba-2 hybrid artifact Runner supports, a near-lossless Q8_0 of NVIDIA's base, 5 of 6 greedy token-identical vs llama.cpp b10353. NVIDIA Open Model License.
    support artifact
  • gemma-4-E2B-it Q4_K_M/Q4_0 mix
    2.63 GB smoke-test artifact. Fails the fidelity bar, the card carries the dual-column numbers, and it remains the fastest way to try Runner on an 8 GB machine.
    fails bar
  • gpt-oss-20b keep-30 MXFP4
    11.5 GB, 32-to-30-expert derivative. Its originally published number did not reproduce; the card leads with the measured status. Kept as a near-miss with its numbers in the open.
    fails current bar
  • Measurement reports, no weights republished
    Hermes-4-14B quant fidelity, the Qwen3 speculative pair, the granite-4.0-h-small and Nemotron-3.5-Lightning support reports, and two frontier reports where nothing cleared the bar and beat upstream: the Lightning prune frontier and the Muse-Glimmer quant frontier, where Meta's own Q4_K_M passes and six Runner plans did not beat it, stated openly.
    reports

Negative results

Built, measured, rejected, and kept.

Work that did not pay is written up so it is not attempted twice, and because where the cliff is happens to be exactly what someone with 8 to 16 GB needs to know.

  • MoE expert caching
    An expert-residency cache tier for MoE models larger than RAM. Investigated, measured, rejected (2026-08-05); not in the product.
    rejected
  • Multi-row Metal decode matvec
    Rejected, and the write-up records what the CPU/GPU byte-identity contract costs in reachable GPU optimizations.
    rejected
  • Metal GEMM occupancy, expert-major MoE layout
    A prefill tile sweep on the M1 where the shipped shape was already at a local optimum (2026-08-15), and two expert-grouping shapes for MoE prefill on the M5 Max that both lose (2026-09-01), kept so the third shape starts from evidence.
    rejected
  • Harmony analysis-channel bound
    Implemented, measured, reverted (2026-08-14), with a same-day correction recorded in place when the motivating symptom turned out to be gone.
    rejected
  • Muse-Glimmer selective precision, Nemotron Lightning artifacts
    Six selective-precision candidates for Muse Glimmer built, gated and rejected (a Runner gap, not a Muse one); three of four planned Lightning derivatives rejected by cheap checks before being built. Published as reports rather than weights.
    no artifact

External reproductions and third-party work

What people outside the project have checked.

Adapter byte-equality on a Tesla T4

A Hugging Face community tester reproduced the byte-identical adapter result independently on a Tesla T4 and is credited in the documentation as the first external reproduction.

A 36-prompt tool-choice boundary bank

The same tester published a reusable 36-prompt bank (MIT or Apache-2.0) probing where adapters trained through BF16, Q8_0 and Q4_K_M choose different tools. Run in full through Runner's unlabeled lane: 3 of 36 prompts split, and the Q4-trained adapter carries the widest margins on that bank. Method and scoping: docs/tool-choice-boundary-lane.md.

A community adapter, loaded back

A third-party F16 adapter in llama.cpp's convention loaded and served in Runner, and in doing so found and fixed the F32-only loader gap. Interop is measured in both directions.