NCM quality and inference benchmarks¶

Quality, latency and training cost answer different questions. This page defines the measurements; EXPERIMENTS.md preserves the experiment history, and paper/data/results.json contains a public extract of measured results. Raw run logs and weights are not committed. Results describe experimental small models, not a general-purpose language model or a reproduction of upstream's full text benchmarks.

Model labels. “MLX implementation” is this repository’s independently trained NCM model; all E-numbered rows refer to it. “Original NCM (Kumbong et al.)” is a cited result from the original study, not a model evaluated or trained here. The MLX AR baseline is separate from the Original AR baseline reported in that study.

Quality: TinyWorld and Sudoku¶

TinyWorld is a finite synthetic grammar. Its generated-sentence metrics check grammatical shape, number agreement, novelty relative to the training set, and diversity. These checks do not measure open-domain text quality. The historical 89.8% NCM versus 98.6% AR validity result uses temperature 1.0 and models of similar generator size.

Sudoku generation is conditioned on the puzzle cells. Greedy decoding (temperature=0.0) gives one candidate per board, with no search, reranking or solver repair. Each external difficulty set has 2,000 boards; the generated validation split has 1,000. A solved board is a valid Sudoku grid that preserves the givens. Cell accuracy against the supplied solution, blank-cell accuracy and the fraction preserving all givens are separate metrics. A valid alternative completion can matter on non-unique validation puzzles; the external boards are intended to have unique solutions.

The E19 report records the full completion, confidence intervals and synchronized A40 timings. Its doubled E15 exposure is not an equal-budget comparison. E20 repeats that scientific recipe at equal exposure and falls by 0.85 / 3.80 / 7.05 percentage points. See the two-seed assessment: scores are retained aggregates, per-board outcomes and complete remote provenance manifests are missing, and E20 benchmark contention is unknown.

These percentages retain the precision of the evaluation artifacts; historical summaries sometimes use one decimal place. E17 restored step-30,000 weights and performed 2,000 further steps with fresh optimizer moments and random state. Its uncheckpointed work before interruption is unknown. It does not establish an improvement over E13 or a controlled effect of puzzle construction. See recovery qualifications.

For an indication of finite-board uncertainty, 95% Wilson intervals for E17 are 69.23–73.19%, 29.75–33.82% and 2.96–4.62%. They assume boards are independent samples from a population and do not account for training-seed variation, related puzzle structure, hyperparameter selection or distribution shift. The benchmark sets are fixed, so the observed scores themselves are exact for those boards. Comparing models on the same boards requires paired outcomes, not an independent-binomial significance claim. Most runs use one training seed; repeated sampling seeds are not independent training repetitions.

Codec reconstruction reads true held-out solutions through the encoder and decoder. E13/E15/E17 reconstruct all 1,000 validation solutions exactly using the full hierarchy. Generated solved-board rates above measure the separate generator task. Teacher-forced code accuracy and validation cross entropy also use true coarser codes and must not substitute for free-running quality.

What is matched¶

The E11 NCM/AR comparison matches generator width/depth, dataset and 32,000 × 64 generator presentations. It reports NCM 55.2 / 18.1 / 1.6% versus AR 20.5 / 4.9 / 0.5% in the historical rounded summary. The AR baseline has a KV cache and puzzle-prefix prefill. It is a practical reference model, not an extensively tuned baseline.

Matching generator presentations does not match FLOPs, training time, optimizer path, or total system size: NCM also trains and retains a codec. E15 has no equal-budget AR quality result. Comparing E15 with the older AR run or with the original paper is an unmatched comparison. Original NCM (Kumbong et al.) reports 99.5 / 96.5 / 86.1% with a 25.46M generator and 51.2M generator presentations, separate difficulty-specific checkpoints, and a different board tokenization. Those published numbers are reference results, not measurements made by this project. Original paper, Tables 3, 5 and 6.

The strongest small-generator result is easy-board accuracy of 98.10% versus Original NCM’s 99.50%, with 6.44M versus 25.46M generator parameters (3.95× smaller) and 16.384M versus 51.2M generator presentations. Medium/hard gaps are 13.10/45.80 percentage points. This does not establish general quality parity or matched-cost efficiency; the codec is additional.

Parameter accounting¶

eval currently reports params by summing every public array in each MLX module's parameter tree. For the E6 codec reused by E13/E15/E17, that includes EMA accumulators, so it is not a count of gradient-trained weights alone.

The learned first-level down/up convolutions contribute 10,617,344 gradient-trained codec values. Earlier descriptions of this codec as about 3.3M omitted these layers. A claim that the local model is about one quarter of upstream's size refers only to the 6.44M versus 25.46M generators. Sampling needs the quantizer's codebooks/refinement/upsampling and decoder, but does not execute the codec encoder; the current loader retains the full codec. Saved-array counts, active inference components and measured memory are different quantities.

Synchronized latency methodology¶

bench_sampler times a complete generation call through conversion of its output to a NumPy host array. This waits for device work and includes output transfer; it never reports only MLX lazy graph construction. Generation also explicitly evaluates its output. Model loading, corpus preparation and training are outside the timed region. Prefix-array construction and prefix computation are inside it for conditional runs.

For each batch size the benchmark clears MLX's allocator cache, records a first call, performs three warmups, resets peak-memory accounting and times the requested iterations. It reports p50, p95, mean, sample standard deviation, and sequences/second derived from batch / p50_seconds. Peak memory is MLX's allocator peak, not total process memory or whole-device usage. First-call latency is not a fresh-process startup measurement. Three warmups and a small iteration count do not guarantee a stable steady state under changing GPU load.

Batch-one p50 is latency for one fixed-length output. Batch-256 p50 is latency for the whole batch; dividing it by 256 does not describe interactive request latency. Throughput at one batch size can be worse than at another because memory pressure and kernels change. The NCM forward count excludes the final decoder call and quantizer work; fewer forwards do not guarantee faster wall time.

Historical benchmark calls use temperature 1.0, including the E17 latency measurements below. Sudoku quality uses greedy temperature 0.0. Thus those timings do not directly time the decoding setting of the solved-board table. New runs can select and record temperature explicitly.

mkdir -p runs
python -m ncm_mlx bench --config configs/YOUR-RUN.json \
  --temperature 0.0 --batch-sizes 1 16 64 256 --iters 30 \
  --out runs/YOUR-RUN-bench.json

Compilation and hardware¶

train.compile: true wraps the generator training step with mx.compile, including optimizer and MLX random state. It does not compile the sampling loop. Codec training, AR training and both samplers are currently uncompiled. Backend kernel optimizations still apply; “uncompiled” here refers specifically to the absence of an explicit mx.compile wrapper. Compiled-training speed measurements cannot be presented as compiled-inference speedups.

Verified execution includes Apple M5 Pro/Metal and NVIDIA RTX 3080 and A40/CUDA using MLX 0.32.3. CPU runs support exactness and recovery tests; they are not published GPU latency measurements. The recorded configurations use fp32 weights; exploratory bfloat16 tests are not the reported training recipe. Metal fp32 matrix products have reduced numerical precision, so exact algorithmic equivalence tests use CPU and device tests use tolerances. Cross-backend bitwise equivalence has not been established.

This run used 20 timed iterations. GPU utilization was 80% immediately before benchmarking while other services remained running. A pre-run sample does not measure the load throughout the benchmark. Treat these as shared-desktop observations, not isolated hardware limits. Do not compare them directly with an uncontended CUDA run or infer a hardware speed ratio. Before any new benchmark, record device/software versions, configuration, checkpoint, sampling settings, compilation, batch sizes, iteration count, and contention without interrupting unrelated workloads.

Training cost and future comparisons¶

Generator metadata times its training loop after code encoding. Periodic validation and checkpoint saving occur inside that loop; data generation, encoding and the final evaluation pipeline are separate costs. E17's 2,000 recovery steps took 1,247.53 seconds; the full recovery pipeline took 1,956 seconds. Neither is the full original E17 training cost. Always name the timing boundary, including repeated work after recovery.

Higher-value future comparisons are multiple training seeds, paired per-board outcomes, a budget-matched E15 AR baseline, matched compiled samplers and uncontended repeated timing sessions. No expensive new training is required to interpret the existing release accurately.

The released-checkpoint sampler is the exception: official generate|sudoku --compile wraps its per-level step in mx.compile, and it has an opt-out generation-local prefix K/V cache. Its eager, cached and compiled timings are in released checkpoints. They do not apply to the repository's own samplers.