Models and published artifacts

Runner accepts GGUF v2/v3. Safetensors checkpoints must be converted to GGUF first. Standard llama.cpp multi-part sets (<prefix>-00001-of-000NN.gguf) load natively from any part: every part must be present in the same directory, and its split.no, split.count, and split.tensors.count metadata must agree. Missing or inconsistent parts are refused before model binding. Nonstandard filenames and remote/streamed parts are not resolved automatically; merge or rename those sets to the standard layout first.

Fetch the small test model with:

./download-model.sh

For manual downloads, verify both the command exit status and resulting byte size. A partially downloaded GGUF can otherwise look like a model failure.

Requantization and expert pruning

Repack weight matrices to q8_0, q4_0, q3_k, q4_k, q6_k, f16, or bf16:

./runner -m model-f16.gguf --quantize model-q4.gguf --quant q4_0

Norms, biases, and rope factors stay f32; tensors already smaller than the target are retained, as are rows the target type cannot describe (q3_k needs a row width divisible by 256, q8_0/q4_0 by 32). MoE router weights (ffn_gate_inp*) keep their source type on every path, including a --type-plan that names them: the router selects which expert runs, so an error there swaps a whole FFN, and it is a fraction of a percent of the file. Metadata is copied.

A q4_0 repack is lossless where the source is already on the q4_0 grid, which is the case for quantization-aware-trained checkpoints: every value is one per-block scale times an integer code, so the answer is already in the file. Runner recovers that scale and those codes exactly instead of re-deriving a scale from the block’s extreme value - the derived route is correct only when a block’s codes actually reach zero, and on a block where they do not it saturates the far end of the range and changes values a pure repack had no need to touch. A candidate is accepted only when the value the dequantizer will produce equals the source float bit for bit across the whole block, so a source that is not on a grid falls through to the derived scale and its output is byte-for-byte what it was before.

--prune-experts rewrites stacked-layout MoE tensors using an explicit JSON plan. It is a mechanism, not a quality claim: pruning needs a model-specific evaluation against the unpruned parent.

{"layer_0":[0,3,7],"layer_1":[1,2,5]}

# Prune only; surviving tensors keep their current quant type.
./runner -m model.gguf --quantize pruned.gguf --prune-experts keep.json

# Prune and requantize the survivors.
./runner -m model.gguf --quantize pruned-q4.gguf \
  --prune-experts keep.json --quant q4_0

A layer omitted from the plan keeps all experts. Invalid keys, empty lists, out-of-range IDs, and unsupported tensor layouts fail instead of silently producing a different model. The layer’s router (ffn_gate_inp) and its per-expert selection bias (exp_probs_b, in either on-disk spelling - the .weight of the DeepSeek-style GGUFs and the .bias of nemotron_h_moe) are sliced along with the expert banks, so the survivors in plan order become the new expert index space with no runtime remapping. Non-uniform coverage-pruned nemotron_h_moe layers resolve their expert count from each layer’s router and run on CPU; CUDA names and declines this layout because its MoE kernels require one model-wide expert count. scripts/moe-prune-plan.py can build a plan from calibration data.

A non-uniform prune describes itself with a per-layer key, and an artifact built that way still has to say where it runs. GGUF defines ONE <arch>.expert_count for the whole model. Every prune plan now also writes <arch>.expert_count_per_layer, a u32 array with one entry per block (0 for a non-MoE block) holding each layer’s real post-prune count; a plain --quantize carries it through and a later plan re-authors it. When a plan leaves every MoE layer at the same new count, the global key is rewritten as well and both agree. When layers end at different counts, or some are left unpruned, the global key deliberately stays at the parent’s number, which remains every layer’s true ceiling, and the array is the exact description. At load Runner validates the array against every router tensor and refuses, by name, a header that disagrees with its tensors; a file without the array predates it and each layer’s count comes from its router alone, as before. An engine that reads only the global key still mis-sizes such a file, so a published artifact from a non-uniform prune must state that it is Runner-correct and untested elsewhere, in the same place it states its fidelity; a uniform prune carries no such caveat. The key is a proposed convention, published here so other loaders can adopt it.

Sublayer removal

--remove-sublayer drops one block’s attention or dense-FFN tensors from the file, so a surgery that found a block’s attention dispensable saves the bytes and the KV cache instead of shipping a same-size file with zeroed weights. A mechanism, not a quality claim: which block can go is a measurement against the parent, made elsewhere.

# drop block 48's attention; survivors keep their bytes
./runner -m model.gguf --quantize cut.gguf --remove-sublayer attn:48
# several at once, and requantize the survivors in the same pass
./runner -m model.gguf --quantize cut-q8.gguf \
  --remove-sublayer attn:48,mlp:12 --quant q8_0

The absence is declared, not inferred: the writer turns the block-wide <arch>.attention.head_count and head_count_kv (or feed_forward_length) into per-block arrays with a 0 at the removed block. That is the reading llama.cpp’s own Nemotron-51B (“deci”) files already use for attention-free and FFN-free blocks, so the file describes itself in the format’s existing vocabulary rather than a private key. What goes is the branch proper: every blk.N.attn_* tensor except the attn_norm pre-norm (projections, Q/K norms, sinks, gates, biases), or every blk.N.ffn_* tensor except ffn_norm. The kept norms cost kilobytes and leave the residual plumbing identical to the zeroed form, which is what the gate compares against: a removed block scores bit-identically to the parent with that block’s output projection zeroed, and differs from the untouched parent. The writer prints exactly what it dropped, in tensors and bytes.

At load the arrays are validated against the bytes in both directions: a tensor missing without a declaration is still error: missing tensor, and a declaration whose tensors are still present is refused by name. A removed attention reserves no KV rows, so the cache shrinks by that block’s share at every context length (the -v banner lists sublayers removed). Limits, each refused rather than approximated: the CPU path only (the device decode loops still drive every block; pass --gpu off), dense blocks only (MoE FFNs, the hybrid SSM families, gemma-4 E-series shared-KV/per-layer embeddings, fused-QKV exports and NextN heads are declined by name), one head width across the file (a non-zero entry that differs from the rest is heterogeneous geometry, not a removal), and no --lora or --train on a removed file yet. docs/sublayer-removal.md has the design and the gates.

Published artifacts

Artifacts produced by this project are published only after their stated gate against the named parent. Read each repository’s provenance before treating a derivative as equivalent to an original checkpoint.

Every fidelity claim below is measured under the adopted dual-column bar (margin-qualified top-1 >= 97% AND mean KLD <= 0.05 vs the named parent, 400 teacher-forced positions, zero point exact; plain top-1 always reported beside it).

This page is the README section of the same name, copied at build time; the README on GitHub is the source.