Skip to content

Use matched prior controls and isolate evaluation randomness - #3

Merged
martyn merged 6 commits into
masterfrom
codex/review-core
Sep 5, 2026
Merged

martyn merged 6 commits into
masterfrom
codex/review-core

Conversation

@martyn

@martyn martyn commented Sep 5, 2026

Copy link
Copy Markdown
Member

The old particle and no-particle examples used different architectures and stabilization recipes, and evaluation consumed training randomness. Share the example implementation, add explicit learned/frozen/fresh prior controls, and isolate data, latent, penalty and evaluation RNG streams. Historical gaussian configs retain their frozen-table meaning.

Add a matched three-prior study protocol using fresh paired seeds, runtime/source provenance, config hashes and validated result collection. Final metrics include per-mode covariance eigenvalues and unique sample counts. Documentation qualifies the historical baseline claims, finite-support and shape-metric limitations, corrects the distributed prior usage example, and explains installation/reproduction. Gaussian-control spectra are explicitly unsupported by the existing particle-specific diagnostic.

All nine 7,000-step experiments completed successfully on an RTX A6000 using clean source f8b59bb. Learned particles reached 100/100 high-quality modes on every seed and averaged 98.61% HQ, versus 8.07% frozen and 6.44% fresh Gaussian. The report preserves the tradeoffs: substantial particle tail/covariance error remains, and frozen Gaussian has lower mean W2 and sliced W1. Reports include every seed, portable data, provenance, curves and a figure renderer; no claim of full Gaussian calibration or general collapse prevention is made.

Validation: 54 integrated CPU tests plus 22 subtests pass. This includes actual all-prior training invariance under different evaluation intervals, all-prior example smoke runs, metric fixtures, sparse corrections, runner regression checks and the regularizer suite. Independent review findings about manifest overwrite/collection checks, evaluation reference clouds, and Gaussian spectra are resolved. The existing historical sparse tables remain explicitly marked where comparisons were invalid; those old studies were not rerun.

@martyn
martyn marked this pull request as ready for review September 5, 2026 03:20

@martyn martyn left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the shared example loop, learned/frozen/fresh sampler semantics, independent training/evaluation streams, covariance diagnostics, and comparison protocol. Independent review confirmed the fixes and identified manifest, real-reference RNG, and Gaussian-spectrum issues that are now resolved. The merged runner/sparse changes pass together with this PR.

Validation: 54 CPU tests plus 22 subtests passed locally; hosted Python 3.11 and 3.12 jobs both passed at head 4bad3d3. All nine full 7000-step experiments completed from clean source f8b59bb. Report aggregates and configs were checked against all original summaries and hashes, portable data has no machine-specific paths, and the rendered curves were inspected. Claims are appropriately limited to concentration/coverage under the fixed recipe, with finite support, tail/covariance errors and opposing transport metrics reported. No outstanding merge blockers.

@martyn
martyn merged commit d1f7848 into master Sep 5, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant