NCM-MLXRoadmap planning scenarios¶
These charts support the research roadmap. They are planning scenarios, not measured future results or forecasts of solve rates. Completed experimental evidence remains in the E19 report.
Training time at the current model size¶
The measured anchor is 54,419.81 training-loop seconds for 16.384M generator presentations on an A40. Holding generator size, batch, hardware and observed mean throughput constant gives:
| Generator presentations | Linear training scenario | Assumed ±20% sensitivity |
|---|---|---|
| 8.192M | 7.56 h | 6.05–9.07 h |
| 16.384M | 15.12 h | 12.09–18.14 h |
| 32.768M | 30.23 h | 24.19–36.28 h |
| 51.200M | 47.24 h | 37.79–56.69 h |
These are exposure scenarios for the current 6.44M generator. The last row matches the reported generator presentation count of Original NCM (Kumbong et al.), not its 25.46M generator, tokenizer, recipe, training cost or expected accuracy. The scenarios exclude rental startup, transfer, data generation, encoding, codec training, final evaluation, failures and hardware changes. Checkpoint/validation overhead is included only through the observed mean training-loop time. The ±20% range is a chosen sensitivity assumption; a single completed run cannot establish a calibrated prediction interval. Longer schedules need their own explicit configuration and funding.
Fixed levels and verified early exit¶
The roadmap proposes independent tests of 3, 6, 9 and 12 levels. One generator forward per level means exactly that many generator calls; the final codec decode is additional. This does not imply a proportional latency reduction: level lengths, prefix computation, attention, quantizer work and decoder overhead differ. A fixed level budget is distinct from stopping only after a Sudoku checker verifies a completed board.
Quality at these budgets is unknown. Measure solved boards, givens preservation, fallback frequency, level usage and synchronized end-to-end latency on the same inputs. Keep the full 12-level result as the baseline, retain per-board outcomes for paired comparisons, and preserve training/inference equivalence at every executed level. Do not infer early-exit quality from codec reconstruction of true solutions.
Gates before larger claims¶
| Roadmap milestone | Required evidence before proceeding | Projection status |
|---|---|---|
| Seed replication | Fresh seed, same recipe and split; complete held-out evaluation | Current experiment; no quality forecast |
| Matched AR comparison | Matched generator presentations plus separate codec/system costs | Needs an explicit run allocation |
| Fixed levels and verified exit | Paired outcomes, checker validation, timing and full-depth fallback | Forward budgets shown above; quality unknown |
| Prefix caching and compilation | Exact output checks, both samplers measured under the same protocol | No supported speedup estimate yet |
| Real-text baseline | Dataset rights, tokenizer/padding contract, reproducible metrics and matched budgets | Sudoku timings and scores do not project to text |
| Upstream checkpoint reference | Independent format conversion and parity checks | No accuracy parity claim until measured |
E20 supplies a second training seed at the funded exposure: 97.25 / 79.60 / 33.25%, versus seed 0’s 98.10 / 83.40 / 40.30%. The two-seed assessment retains both outcomes and their audit limits; none of these scenarios predicts how another seed will score.
The strongest completed small-generator finding is 98.10% versus 99.50% on easy boards, using a 3.95× smaller generator and 3.125× fewer generator presentations. Medium and hard gaps remain 13.10 and 45.80 percentage points. More training might help, plateau or worsen held-out results; the current experiments do not identify a scaling law.
Machine-readable assumptions can be rebuilt with uv run --group paper python scripts/project_roadmap.py. Private account balances and spending plans are not included here.