Support matrix
runner --caps publishes the architecture IDs admitted by the current binary:
Metal quant-type coverage. Both a matvec and a matmul kernel exist for
q2_K, q3_K, q4_0, q4_K, q6_K, q8_0, iq4_nl, iq4_xs, mxfp4,
f16, bf16 and f32. q4_1, q5_0, q5_1 and q5_K ship k_mv_*
only — they decode on Metal but have no k_mm_*, so prefill on those types
does not use the Metal matmul path. Requantizing such a file to q4_K or
q8_0 (--quantize OUT --quant q4_k) is the fix when prompt throughput
matters. Quantization changes only the weight encoding: it cannot give an
architecture a Metal path it lacks, because those gaps are missing kernels for
operations (SSM scan, Gated DeltaNet, weight-normed routers, gate-less shared
experts), not missing quant support.
A sharded GGUF takes a full Metal offload directly: each part’s mapping
gets its own tensor-boundary wraps, gated byte-identical to both the CPU path
and the single merged file (make test-metal-split; measured on a real
2-part 86 GB set). A partial --gpu-layers split of a sharded set refuses to
CPU — merge to one file with --quantize OUT --quant keep first if a layer
split is what you need. The loader accepts both duplicated-metadata shards
and the standard compact form where only part one carries model metadata;
explicit contradictions between parts are still rejected.
Sparse-MoE expert matvec kernels cover q2_K, q3_K, q4_0, q4_K,
q5_K, q6_K, q8_0, mxfp4, f16, and f32. This list is narrower than
the dense quant list: a type needs a dedicated indirect expert kernel, not
merely dense matvec/matmul support.
GGUF general.architecture |
Notes | ||
|---|---|---|---|
llama, mistral, smollm, stablelm |
Llama-style dense families with family tokenizers/templates. | ||
qwen2, qwen3 |
QKV-bias and per-head-QK-norm variants. | ||
qwen35 |
Dense Qwen3.5/Ornith Gated DeltaNet plus full attention; CPU and CUDA. CPU recurrent folds support speculative decode, grammar fast-forward, and exact shared-prefix restore. Any GPU-backed recurrent instance declines shared-prefix restore; a CUDA-resident recurrent layer also declines speculative decode and grammar fast-forward. | ||
qwen3moe |
Fused and legacy split sparse-MoE layouts on CPU/CUDA; supported fused layouts on Metal. | ||
gemma3 |
Regular and QAT layouts, sliding-window attention, sandwich norms. | ||
gemma4 |
Heterogeneous attention, thinking channels, E-series, supported dense/MoE layouts, and the family’s native tool protocol. Both E-series export shapes load. A layer at or past block_count - attention.shared_kv_layers computes no K and no V (it attends over the cache an earlier layer filled), so the current quantized exports - the ggml-org Q4_0, Google’s own QAT Q4_0 and the community QAT F16 - omit attn_k.weight, attn_v.weight and attn_k_norm.weight on exactly those layers: 666 tensors on E4B where the BF16 export has 720. Those three are optional on the shared-KV tail and still required on every KV-owning layer, where a missing one is refused by name. |
||
phi3 |
Fused QKV and gate/up tensors, LongRoPE factors. | ||
gpt-oss |
Attention sinks, alpha-sigmoid GLU, expert biases, MXFP4 experts. Tokenizer exact (0/721 differential) and chat renders the real Harmony format (analysis channel as reasoning_content) as of 2026-08-14; cross-engine greedy identity remains inside the model’s own measured KV-precision sensitivity envelope rather than certified. |
||
apertus |
xIELU FFN; CPU and CUDA. | ||
afmoe |
Arcee Trinity sparse MoE; CPU only. CUDA and Metal refuse it loudly as gated attention plus sparse MoE, rather than misreporting a quantization problem. | ||
muse-glimmer |
Meta Muse Glimmer 30B, text path: gated attention, QK and sandwich norms, SWA with NoPE globals, softcapped logits. CPU, CUDA and Metal. Measured 2026-08-11; evidence in docs/muse-glimmer-cert-2026-08-11.md and docs/muse-atem-cert-2026-08-11.md. No vision encoder. Native atem definitions/results, recipient-constrained generation, truncation recovery, multi-call mapping, and buffered/SSE parsing are implemented and selected automatically for tool requests. |
||
granite |
IBM Granite dense (3.x/4.1): the four muP scalars (embedding, fixed attention, residual, divided logit). CPU, CUDA and Metal. Measured 2026-08-11; evidence in docs/granite-cert-2026-08-11.md. granitemoe is a separate arch id and not admitted; granitehybrid is admitted separately, below. |
||
granitehybrid |
Granite-4 h-series: a Mamba-2 selective-SSD recurrence (causal conv1d + the input-dependent state-space scan, with the gated RMS norm) interleaved with GQA attention, the layer type read per-layer from the attention.head_count_kv array (0 ⇒ recurrent); the attention layers are NoPE (rope.scaling.finetuned=false); the four granite muP scalars. Both published FFN layouts are supported: dense h-micro has a gated MLP on every layer and runs on CPU and CUDA; sparse h-small has a routed MoE FFN plus an always-on shared expert and currently runs on CPU because those two branches have no device path. The dense h-micro CUDA path is CPU-token-identical over 600/600 greedy tokens with per-run mean |
Δlp | ≤ 0.000024 (max per-position 0.000422); evidence and raw probes are in docs/compat-reports/cpu-cuda-hybrid-2026-08-21/. The sparse h-small CPU path was verified against llama.cpp b10353 at both Q4_K_M and Q8_0: greedy output is token-identical on deterministic prompts (a 256-token completion matches byte-for-byte) and holds at the quantisation noise floor elsewhere, where the divergences are synonymous-phrasing near-ties, not wrong math. Re-verified at higher precision (Q8_0, 2026-08-19): the sole non-empty divergence is a single-token near-tie whose top-2 candidates the runner and llama.cpp rank identically to within ~0.03-0.09 nats (an argmax coin-flip), with the runner’s full top-5 logit distribution matching the oracle’s - so the Mamba-2 math is correct and the Q4_K misses were pure noise floor, the same envelope noted for gpt-oss. Chunked-scan prefill: the token axis is tiled into chunks (~256), the per-head SSD recurrence runs in parallel across heads within a chunk and the SSD state + conv ring are carried across chunk boundaries - bit-identical to the serial per-token sweep (a pinned make test gate holds chunked == serial across chunk sizes) and ~1.8x faster prompt throughput on a long prompt (measured on granite-4.0-h-small Q8_0, 264 tokens). XR_SSM_SERIAL=1 forces the serial reference path. The recurrent-state cache seam is wired: the fixed-size fold is snapshotted/restored on a rewind, and stored beside the KV in the prefix cache so an exact CPU prompt-prefix hit restores it in a memcpy rather than recomputing the recurrent layers. CPU speculative decode and grammar fast-forward use a per-round fold checkpoint; a CUDA split is admitted only while every recurrent layer remains host-resident. Metal has no SSM path. |
nemotron_h |
NVIDIA Nemotron-H (Nemotron-Nano-9B-v2): a Mamba-2 selective-SSD recurrence interleaved with GQA attention and dense MLP blocks, where each block is EXACTLY ONE of three kinds (SSM | attention | MLP), typed per-layer off attention.head_count_kv (0) and feed_forward_length (0). NON-MoE and no muP scalars - unlike granitehybrid; the MLP is a gate-less squared-ReLU FFN (down(relu(up(x))^2)), attention is NoPE (rope.scaling.finetuned=false), and the SSM uses a GROUPED scan (ssm.group_count=8): B/C are shared across groups of heads and broadcast (group g covers heads [g·H/G, (g+1)·H/G)) - the same grouped scan nemotron_h_moe (Nemotron-3.5 Lightning) also uses - here first proven WITHOUT MoE, and admitted WITH MoE in the row below. CPU and CUDA: the Mamba-2 SSD scan, causal conv1d, gated RMS norm and squared-ReLU FFN all have device kernels, and full 56-layer offload is greedy byte-identical to the CPU path on the real Nano-9B-v2 Q8_0 (3 prompts x 48 decode steps plus an 88-token multi-tile prefill; re-verified independently post-merge at 32 tokens). Device prefill currently runs the per-token loop (correct, unoptimized); no Metal SSM path. Verified against llama.cpp b10353 on the real Nemotron-Nano-9B-v2 at Q8_0 (same GGUF both engines, CPU): 5/6 greedy completions byte-identical (including both 256-token generations); the sole miss is a single-token near-tie where both engines share the same top-3 candidates and llama.cpp’s own top-1/top-2 gap is ~0.075 nats (an FP-summation-order coin-flip), i.e. the quantisation noise floor, not wrong math. Chunked-scan prefill (the grouped scan tiled into chunks, parallel across heads within a chunk, SSD state + conv ring carried across chunk boundaries), bit-identical to the serial per-token sweep and pinned chunked == serial in make test; the recurrent-state cache seam is wired (fold snapshotted/restored on rewind, and stored beside the KV so an exact CPU prompt-prefix hit restores it in a memcpy). CPU speculative decode and grammar fast-forward use the per-round fold checkpoint; full GPU offload and partial splits with a CUDA-resident recurrent layer decline them. |
||
nemotron_h_moe |
NVIDIA Nemotron-3.5-Lightning-30B-A3B: nemotron_h with the dense squared-ReLU MLP replaced by a gate-less squared-ReLU MoE (128 experts / 6 used, no gate branch) plus an always-on gate-less shared expert; the router reuses the general softmax/group/scale/norm path. Same three-way block typing, grouped scan (n_group=8), and NoPE attention as nemotron_h. Runs on CPU: the SSM scan has a device kernel, but this family’s router (weight-normed, scaled) and gate-less shared expert have no device path, so the backend falls back to CPU there. Greedy vs llama.cpp ea12b27 on the real Lightning-30B Q4_0 (CPU, 8 tok × 5 prompts): 4/5 byte-identical, the one divergence a near-tie on an open-ended counting continuation (noise floor, not wrong math - the coherent Paris. … Berlin. completion matches exactly). Evidence: docs/compat-reports/ssm-greedy-reference-2026-08-20/. |
Admission remains layout-specific: an unsupported split expert layout, a
non-SiLU MoE outside gemma-4’s dual-branch form, or an architecture-specific
tensor arrangement is refused even when the architecture ID is listed. The
always-on shared expert (Qwen2-MoE/DeepSeek form: a dense FFN over the same
normed input, summed with the routed output, optionally gated) is supported
and is what afmoe uses; its width and tensors are shape-checked at load, and
expert_shared_count set without the tensors present is an error rather than a
silently dropped branch.
| Area | Current support |
|---|---|
| File format | GGUF v2/v3, mmap/file-mapped host weights, including standard local multi-part sets. |
| Tokenizers | SPM and byte-level BPE with llama, qwen2/qwen35, smollm, afmoe, tekken, llama4/gpt-4o, Gemma, and GPT-2-family pre-tokenization rules. |
| Quantizations | --caps lists the admitted tensor formats: the k-quant and legacy families plus MXFP4, NVFP4 (two-level, with its per-tensor scale companion applied) and the codebook i-quants (IQ1_S/M, IQ2_XXS/XS/S, IQ3_XXS/S, IQ4_NL/XS). NVFP4 and the IQ1, IQ2 and IQ3 families are CPU-only; CUDA and Metal refuse them loudly, naming the exact tensor and type that caused the CPU fallback. |
| Transformer | RMSNorm, adjacent-pair and NeoX RoPE, grouped-query attention, SwiGLU/GELU/xIELU family paths, tied embeddings, dense and selected sparse MoE. |
| Sampling | Greedy, temperature, top-k, top-p, min-p, repeat penalty, stop strings, JSON/schema constraints, speculative decoding. |
| Context | Batched prefill, f16/q8 KV, linear/YaRN/llama-3 scaling, automatic extension. |
| Serving | Chat Completions, Responses, legacy completions, embeddings, Anthropic Messages, SSE, parallel slots, model swap, prefix reuse. |
| Desktop | macOS menu bar and Windows notification-area controller. |
| Provenance | Replay-verifiable transcripts; Ed25519-signed, chained receipts with a one-exit-code verifier; OpenSSF Model Signing verification of the loaded GGUF (key method, P-256/384/521). |
Not implemented: Vulkan; TLS/auth; remote bind; remote/streamed GGUF parts; the
qwen2moe/deepseek2/kimi architecture IDs (their shared-expert layout is
implemented, as above - the architectures are not admitted) or MLA attention;
Mamba/Jamba; MTP/NextN draft-head consumption on the GPU backends or with more
than one predictor block (--mtp serves the single-block CPU case; without
the flag the tensors load and are skipped, so dense decoding is unchanged);
OMS model signatures by the certificate or keyless (Fulcio/Rekor) methods, or
with shard or BLAKE serialization (reported as unsupported, never as
verified); full GBNF; image/document inputs;
hosted tools; response persistence; or parallel tool calls on the Responses and
Messages surfaces (Chat Completions supports it, buffered and streaming).
This page is the README section of the same name, copied at build time; the README on GitHub is the source.