Reproduce training and evaluation¶

Results refer to a config, training budget, selected checkpoint, evaluation procedure and hardware. A matching model shape alone does not reproduce a result. Read platform setup before using the commands below and benchmark methodology before comparing latency or quality.

TinyWorld quickstart¶

uv sync --locked --python 3.12
uv run python -m ncm_mlx smoke
uv run python -m ncm_mlx train --config configs/toy-grammar-12k.json
uv run python -m ncm_mlx demo --config configs/toy-grammar-12k.json
uv run python -m ncm_mlx eval --config configs/toy-grammar-12k.json
uv run python -m ncm_mlx bench --config configs/toy-grammar-12k.json
uv run python -m ncm_mlx baseline --config configs/toy-grammar-12k.json --bench

TinyWorld is generated from the grammar in ncm_mlx/text/toygrammar.py; no external text dataset is downloaded. The config selects deterministic train and validation splits. Training writes weights and metadata to its out_dir under artifacts/. The autoregressive baseline trains separately. Repeat runs may differ across hardware and library versions; seeds are not a cross-backend reproducibility guarantee.

Sudoku evaluation data¶

scripts/fetch_upstream_sudoku.sh

The downloader fetches only the six puzzle/solution files from hazyresearch/ncm-datasets at revision 6580ba41a4b0a976dde876fbe84d25327617b0e6. It verifies SHA-256 hashes from scripts/sudoku.sha256 before replacing each destination. There are 2,000 boards per difficulty. ncm_mlx/text/upstream_sudoku.py reads the tokenized lists with a restricted unpickler that refuses Python globals. These boards are evaluation inputs, not the locally generated training set. See third-party provenance.

Sudoku configs can reuse a trained codec via codec_checkpoint; inspect that field before launching a generator experiment. Weight files are not included in the release. Reproduce the referenced codec training first, then use the generator config and recorded step budget from EXPERIMENTS.md. E15 uses 8.192M generator presentations. The newer E19-funded result requires configs/sudoku-e19-funded.json, the frozen E6 codec, eight million additional cached unique puzzles and 128,000 generator updates at batch 128 (16.384M presentations). Its best-validation checkpoint is step 127,000. Neither result is expected from the short quickstart. The completion report records checkpoint identity, backend, timings and uncertainty.

uv run python -m ncm_mlx train --help
uv run python -m ncm_mlx eval --help
uv run python -m ncm_mlx bench --help

For a completed Sudoku checkpoint, evaluate with --temperatures 0 --seeds 1 and retain the complete JSON report. Keep validation and the three published test difficulties separate. Report codec reconstruction independently from the generator's solved-board rate.

Record a run¶

Preserve the source revision, complete config, dependency lock, backend and GPU model, initial GPU utilization, seeds, training steps, actual sequence budget, wall time and checkpoint-selection rule. Store raw JSON reports locally under runs/ and weights under artifacts/; these directories are ignored by Git. Add the measured result and conditions to docs/EXPERIMENTS.md, and update the committed, sanitized paper/data/results.json when publishing a paper milestone. Running uv run --group paper python paper/make_figures.py collects local reports and rebuilds figures; --plot uses the committed data without collecting reports.

scripts/run_experiment.sh CONFIG runs training, evaluation and synchronized benchmarks under the memory watchdog. Mac runs default to a 20 GiB MLX memory limit and a 24,000 MB process-tree watchdog; choose a workload that fits rather than interpreting the watchdog as an allocation guarantee. Check for other accelerator workloads before launching anything.

Interruptions and continued training¶

Use full-state checkpoints and the pause/resume procedure. The recovery tests compare short interrupted and uninterrupted trajectories on the same CPU backend, including compiled generator steps. They do not establish exactness across CPU, Metal and CUDA or across different MLX versions. Keep an external copy of checkpoints if machine or disk loss matters.

E17 is a documented exception: its legacy checkpoint contained weights only. Its last 2,000 steps reset optimizer and random state, so it is not an exact continuation of the uninterrupted trajectory. Preserve that qualification in any comparison.