Use matched prior controls and isolate evaluation randomness - #3
Conversation
martyn
left a comment
There was a problem hiding this comment.
Reviewed the shared example loop, learned/frozen/fresh sampler semantics, independent training/evaluation streams, covariance diagnostics, and comparison protocol. Independent review confirmed the fixes and identified manifest, real-reference RNG, and Gaussian-spectrum issues that are now resolved. The merged runner/sparse changes pass together with this PR.
Validation: 54 CPU tests plus 22 subtests passed locally; hosted Python 3.11 and 3.12 jobs both passed at head 4bad3d3. All nine full 7000-step experiments completed from clean source f8b59bb. Report aggregates and configs were checked against all original summaries and hashes, portable data has no machine-specific paths, and the rendered curves were inspected. Claims are appropriately limited to concentration/coverage under the fixed recipe, with finite support, tail/covariance errors and opposing transport metrics reported. No outstanding merge blockers.
The old particle and no-particle examples used different architectures and stabilization recipes, and evaluation consumed training randomness. Share the example implementation, add explicit learned/frozen/fresh prior controls, and isolate data, latent, penalty and evaluation RNG streams. Historical
gaussianconfigs retain their frozen-table meaning.Add a matched three-prior study protocol using fresh paired seeds, runtime/source provenance, config hashes and validated result collection. Final metrics include per-mode covariance eigenvalues and unique sample counts. Documentation qualifies the historical baseline claims, finite-support and shape-metric limitations, corrects the distributed prior usage example, and explains installation/reproduction. Gaussian-control spectra are explicitly unsupported by the existing particle-specific diagnostic.
All nine 7,000-step experiments completed successfully on an RTX A6000 using clean source f8b59bb. Learned particles reached 100/100 high-quality modes on every seed and averaged 98.61% HQ, versus 8.07% frozen and 6.44% fresh Gaussian. The report preserves the tradeoffs: substantial particle tail/covariance error remains, and frozen Gaussian has lower mean W2 and sliced W1. Reports include every seed, portable data, provenance, curves and a figure renderer; no claim of full Gaussian calibration or general collapse prevention is made.
Validation: 54 integrated CPU tests plus 22 subtests pass. This includes actual all-prior training invariance under different evaluation intervals, all-prior example smoke runs, metric fixtures, sparse corrections, runner regression checks and the regularizer suite. Independent review findings about manifest overwrite/collection checks, evaluation reference clouds, and Gaussian spectra are resolved. The existing historical sparse tables remain explicitly marked where comparisons were invalid; those old studies were not rerun.