Every Model Starts with Assumptions
We generated 4,344 synthetic cases before we had real data. The synthetic data was calibrated to published NCES/IPEDS (federal postsecondary data) aggregate statistics — sector distributions, enrollment ranges, closure rates by institution type. But calibration to aggregates doesn't guarantee the distributions match at the institution level.
The retained public-data cohort contains 1,516 institutions. This check is itself an ADM demonstration: validation determines whether the lower-fidelity synthetic model was sufficient and which earlier claims must be withdrawn.
Synthetic vs. Real: The Numbers
High-level comparison of the two datasets. These summary statistics tell us whether our synthetic generation process produced a dataset of similar scale and closure prevalence.
Did We Get the Mix Right?
The most basic structural check: does the synthetic dataset have roughly the same proportion of institutions in each sector (Public 4-year, Private Nonprofit 4-year, For-Profit, etc.) as the real data? Getting this wrong would cascade into every downstream analysis.
Where Did Closures Actually Happen?
Sector-level closure rates are the core signal in this study. If the synthetic data nailed the overall rate but assigned closures to the wrong sectors, every investigation built on that data would inherit the bias.
Did Geography Hold Up?
Regional patterns in college closure are driven by demographics, state funding, and local economic conditions. Our synthetic data used regional closure rate multipliers from published NCES data. Here's how those calibrated rates compared to what we found in the real data.
Did Synthetic Institutions Look Real?
Beyond counts and rates, did the synthetic institutions have realistic enrollment sizes? This matters because small institutions close at much higher rates, so getting the enrollment distribution wrong would distort the risk profile. We compare the enrollment distribution for Private Nonprofit 4-year institutions — the sector with the most closures.
What Matched and What Didn't
The ADM takeaway: synthetic data was useful for building the workflow and exploring questions, but it failed numerical validation for closure prevalence and model performance. The real-data output replaces it for visitor-facing results, and its six observed closures are still too sparse for institution-level prediction.
How We Built the Comparison
Synthetic data generated using numpy random distributions calibrated to published NCES/IPEDS aggregate statistics. Real data from College Scorecard API and IPEDS direct downloads. Comparison metrics computed on matching sector and region definitions. Enrollment distributions binned using consistent bin edges across both datasets.
Sectors follow IPEDS classification (Public 4-year, Private NP 4-year, For-Profit 4-year, Public 2-year, Private NP 2-year, For-Profit 2-year, For-Profit <2-year). Regions follow Census Bureau divisions. Closure defined as institution no longer operating as of 2023.