[codex] Align SPM thresholds and stabilize clone-half priors#696
Closed
MaxGhenis wants to merge 3 commits intoPolicyEngine:mainfrom
Closed
[codex] Align SPM thresholds and stabilize clone-half priors#696MaxGhenis wants to merge 3 commits intoPolicyEngine:mainfrom
MaxGhenis wants to merge 3 commits intoPolicyEngine:mainfrom
Conversation
Contributor
Author
|
Superseded by #702. This draft was opened from a forked branch and is blocked by the workflow gate, so CI cannot run here. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
policyengine-usSPM_CAPHOUSESUBbenchmark forspm_unit_capped_housing_subsidyhousing_assistance, so Census SPM capped subsidy and HUD spending/assisted-household counts are no longer mixed togetherspm_unit_spm_thresholdfor the PUF clone halfspm_unit_spm_thresholddeterministically from the donor half's geography and the current threshold formula+1sparse-reweighting prior with deterministic near-zero priors for zero-weight clone households, while keeping donor-half priors close to their survey weightsWhy
The data pipeline had three distinct SPM issues:
policyengine-us-datahad drifted from the model-side logic inpolicyengine-usspm_unit_spm_threshold, even though thresholds should be derived from donor geography plus composition, not predicted statisticallyThe housing benchmark cleanup is separate but related concept hygiene:
spm_unit_capped_housing_subsidyis a Census SPM concept and should be benchmarked to CPS ASECSPM_CAPHOUSESUBhousing_assistanceis a HUD program/spending concept and should be benchmarked separately to HUD USER assisted-household counts and spending totalsImpact
policyengine-usRoot cause
policyengine-us-datahad:policyengine-usspm_unit_spm_thresholdinto the CPS-only QRF output setValidation
uv run pytest -q tests/unit/test_extended_cps.pyuv run pytest -q tests/unit/calibration/test_calibration_puf_impute.pyuv run pytest -q policyengine_us_data/tests/test_local_area_calibration/test_spm_thresholds.pyuv run pytest -q tests/integration/test_enhanced_cps.py -k 'household_count or poverty_rate_reasonable'git diff --check