Minnesota state dataset: Population/household undercount and multi-tax-unit household concentration bug

[DTrim99]

Summary

Analysis of the Minnesota (MN) state dataset reveals significant discrepancies from actual Minnesota statistics and a data quality issue with multi-tax-unit household distribution.

Issues Found

1. Population and Household Undercount

Metric	MN Dataset	Real/Target	Difference
Population	~4.1M	5.74M	-29%
Households	1,254,857	~2,344,432	-46%

Sources:

• Census Bureau QuickFacts: Minnesota
• Census Reporter - Minnesota

2. Multi-Tax-Unit Households Concentrated in Top Income Deciles

When analyzing a CTC reform that should only affect low-income households (phases out at ~\0-40k), we found unexpected impacts in the 8th, 9th, and 10th income deciles.

Investigation revealed that affected high-income households have dramatically more tax units than unaffected ones:

Metric	Affected Top-Decile HH	Unaffected Top-Decile HH
Avg tax units per household	6.41	1.55
Household count	50,907	325,752

Distribution of tax units among affected top-decile households:

• 53% have 8 tax units
• 40% have 5 tax units
• Only 6% have 2 tax units

This is likely a bug - multi-tax-unit households should be distributed across the income spectrum, not concentrated in the top deciles.

Impact

This causes misleading results when analyzing policies that target low-income populations (like CTCs, EITC, etc.), as the impacts appear to affect wealthy households when they shouldn't.

Investigation Details

Full analysis documented in: PolicyEngine/analysis-notebooks#108

Suggested Fix

Review the household/tax-unit mapping and weighting in the state dataset calibration to ensure:

1. Population and household counts match targets
2. Multi-tax-unit households are distributed realistically across income deciles

[PavelMakarchuk]

Detailed Diagnostic Analysis

I ran a comprehensive diagnostic on the MN.h5 dataset from HuggingFace. Here are the specific findings:

1. Calibration Target Validation

The dataset is significantly missing state-level calibration targets:

Metric	Simulated	Target	Deviation
Population	3,985,277	5,737,915	-30.5%
Households	1,229,850	2,344,432	-47.5%
Tax Units	2,081,321	2,871,840	-27.5%

2. AGI Distribution Issues

The AGI distribution is severely distorted - over-representing extremes while under-representing middle brackets:

AGI Bracket	Simulated TUs	Target TUs	Deviation
<$1	108,997	10,930	+897%
$500k+	132,455	38,950	+240%
$75k-$100k	93,814	288,810	-67.5%
$50k-$75k	173,671	403,640	-57%
$40k-$50k	75,936	179,700	-57.7%
$100k-$200k	293,167	514,790	-43.1%

3. Weight Distribution Analysis

The weight distribution is extremely concentrated:

• 68.2% of household records have weight < 1 (contributing only 0.4% of total weight)
• 1% of records (664 households) account for 62% of total weight
• Top 5% of households by weight contribute 83% of total weight

4. Most Impactful Household Identified

I identified the specific household causing the largest CTC distortion:

• Household AGI: $622,863
• Structure: 5 tax units (3 adults aged 60, 29, 29 + 2 children)
• Replication: Appears in all 8 MN congressional districts
• Total weight: ~20,000
• CTC contribution: $159M weighted (9% of MN's total $1.75B CTC)

This single household structure, replicated 8 times with high weights, accounts for most of the "CTC affecting rich households" effect.

5. Root Cause Assessment

The evidence points to multiple interacting issues:

1. CD-stacking weights don't aggregate to state totals - The sum of household weights across congressional districts (~1.23M) is far below the state household target (~2.34M)

2. Extreme weight concentration - The calibration process is concentrating weight on a small subset of households rather than distributing it across the sample

3. Multi-TU amplification - When a multi-TU household gets high weight AND gets replicated across all CDs, its quirks (like having low-income TUs within a high-AGI household) get amplified dramatically

4. Missing middle-income representation - The calibration is not achieving the correct AGI distribution, with middle brackets severely underweighted

Diagnostic Code

I've created diagnostic tools in policyengine_us_data/diagnostics/ that can be used to validate any state dataset:

• mn_calibration_diagnostic.py - MN-specific multi-TU analysis
• calibration_benchmarks.py - General benchmark validation against SOI targets

Suggested Investigation Path

1. Review how CD-level calibration targets are set and whether they're constrained to sum to state totals
2. Investigate why the weight calibration is concentrating on so few records
3. Check if multi-TU households are being handled specially (or should be) in the stacking algorithm
4. Validate the source CPS data to ensure multi-TU households aren't over-represented at baseline