How to tell whether two groups vary along the same directions, using principal angles between subspaces. The example is the 50 US states, grouped by Census region. Pick two groups in the tool; how to read it, and what principal angles are, follow below it.
Source, data and tests: github.com/rrenaud/state-subspaces
Both plots show all 50 states; the selected group's are full size and labeled. Axis and vector names, in quotes, are rough readings of the largest weights, and each carries its vector's symbol: u₂ “older” and v₂ “older” are different directions that happen to read alike. Hover a state: on each row the faded bar is the weight, the solid bar on the same scale is z × weight, and the black tick is the z-score on a ±3 scale. The Sum row stacks the products on its own scale, and its tick is the state's coordinate.
Data: 2022 American Community Survey 1-year estimates, 2020 Census urban share, 2022 Census Gazetteer land area. scripts/fetch_census.py downloads them from the Census Bureau and writes data/census.csv.
Each state is a point in 8 dimensions: population and density on a log scale, median household income, median age, share with a bachelor's degree, share urban, share 65 and older, and poverty rate, each z-scored across all 50 states. Each Census region is a group, and its subspace is spanned by its top r principal components.
Each basis vector is a unit-length weighted sum of the features, and its squared weights add up to 1. So a weight of 0.7 means about half of that direction (0.7² ≈ 0.49) is that one feature. Features with large weights are drawn bold, and those near zero are faded. The plots show all 50 states projected onto each basis, with each region in its own color and shape. Hover a state to see how it gets its coordinates: each row then shows the state's z-score for that feature, as a number and as a black tick on a ±3 scale, and z × weight, as a number and as a solid bar drawn over the faded weight bar on the same scale. So the solid bar is the weight bar stretched by the z-score: longer where the state is more than one standard deviation from average, shorter where it is close to average, and flipped where it is below average. A product past ±1 is cut off at the end of the track. Under the rows, the Sum row stacks the products on a scale of its own, fixed for each basis vector: positive products build rightward from zero, negative ones leftward, and the black tick is where they net out, which is the state's coordinate. The products add up to the state's coordinate along that basis vector. Try Rotate: the basis vectors, the plot and the matrix change, and the angles do not.
Take a group of data points and its top r principal components. Those components span a subspace: the directions along which the group varies most. Two groups can be compared by asking how much their subspaces overlap.
A subspace has many bases. Rotate the basis vectors among themselves and they still span the same subspace. So a comparison should depend only on the subspaces, not on which basis vectors happened to come out of the computation. Comparing basis vectors one by one fails this test.
Principal angles answer the question greedily:
If θ₁ = 0°, the two subspaces share one direction exactly. If every angle is 0°, they are the same subspace. If every angle is 90°, each direction in one is orthogonal to each direction in the other.
How to compute them. Put orthonormal bases of the two subspaces in the columns of U₁ and U₂. The small r×r matrix U₁ᵀU₂ holds every dot product between a basis vector of S₁ and one of S₂. Its entries depend on the bases chosen. Its singular values do not. The SVD U₁ᵀU₂ = P · diag(cos θᵢ) · Qᵀ finds the rotation of each basis (U₁P and U₂Q) that matches the vectors up one to one, and the singular values are the cosines of the principal angles. The greedy steps above are not an approximation: they find exactly the pairs the SVD finds all at once, and for every k, the first k pairs have the largest total cosine that any k orthonormal pairs from the two subspaces can reach.
A worked example in three dimensions, with r = 2:
A one-number summary. The overlap score ‖U₁ᵀU₂‖²_F / r = (cos²θ₁ + … + cos²θ_r) / r is the average of the squared cosines. Pick a unit vector in S₂ at random and project it onto S₁: the score is the squared length it keeps, on average. It is 1 when the subspaces coincide, 0 when they are orthogonal, and the same in either direction.
The floor. A random unit vector in ℝⁿ keeps, on average, 1/n of its squared length along any one fixed direction, so r/n when projected onto any r-dimensional subspace. Two unrelated subspaces therefore score about r/n. In the 8 dimensions of the state data, with r = 2, that is 0.25, which is not small.
The ceiling. Even one group does not produce the same subspace twice: a different sample of its data moves it. Comparing two halves of one group's data estimates how high a score can realistically go. A score means something only relative to both the floor and this ceiling.
Every number below comes from node scripts/report.js.
The national directions. The top three principal components of all 50 states are the main ways states differ from each other. The states at each end show what each one means:
| Direction | Largest weights | Variance | Low end | High end |
|---|---|---|---|---|
| 1. Affluence | 0.50 Income + 0.48 Bachelor's + 0.43 Urban − 0.41 Poverty | 41% | West Virginia, Mississippi, Arkansas | New Jersey, Massachusetts, Maryland |
| 2. Age | 0.65 Median age + 0.63 Age 65+ + 0.22 log Density − 0.20 Urban | 27% | Utah, Texas, Alaska | Maine, Vermont, New Hampshire |
| 3. Size | 0.61 log Pop + 0.51 log Density + 0.43 Poverty − 0.26 Income | 20% | Alaska, North Dakota, Wyoming | Florida, New York, Louisiana |
Each region against the nation. Each region's own top two directions, compared with the national pair (r = 2). Smaller groups have noisier principal components, so they score lower even when nothing about them is unusual. The fair comparison is with random groups of the same number of states: the percentile says where the region falls among 2,000 of them.
| Region | θ₁, θ₂ | Overlap | Random groups, same size: median | Percentile | Spread on affluence, age (nation = 1) |
|---|---|---|---|---|---|
| South (16) | 9°, 17° | 0.94 | 0.82 | 89% | 1.28, 0.87 |
| West (13) | 12°, 48° | 0.70 | 0.79 | 34% | 0.96, 1.00 |
| Northeast (9) | 21°, 56° | 0.59 | 0.72 | 27% | 0.92, 0.85 |
| Midwest (12) | 13°, 75° | 0.51 | 0.77 | 8% | 0.23, 0.14 |
The South is the most nationally typical region. It holds the bottom of the affluence scale (four of the five lowest states are Southern, New Mexico the exception) and Maryland near the top, so its spread along that direction is 1.3 times the nation's. It also spans the age direction, from young Texas, Georgia and Oklahoma to old Delaware, West Virginia and Florida. Its top two directions are the national ones, at 9° and 17°. Two checks:
The Midwest is distinct, but not in its directions. Its overlap with the nation is low, but at the 8th percentile it is only borderline unusual for 12 states. What is unusual is how little it varies along the national directions: about a quarter of the nation's spread on affluence and a seventh on age. None of 2,000 random 12-state groups is that compressed on both. Midwestern states are alike in wealth and age, so what is left to vary is size, Ohio and Illinois against the Dakotas, and that becomes the Midwest's top direction.
This is a limit of the overlap score. It measures which directions a group varies along, not how much. A group can have nearly the national directions with much less variance along them. Checking the spread, or the singular values, catches what the angles miss.
The Northeast's third direction is its own. At r = 2 it is unremarkable (27th percentile). At r = 3 its third angle is 84° and its overlap falls to the 2nd percentile. Its top direction is urban against rural, New Jersey and Massachusetts against Maine and Vermont, with the rural states older.
The West is unremarkable. Its top direction ties age to wealth (older states like Montana and New Mexico are poorer and smaller than Utah, California and Colorado), which the nation's directions do not. But it sits at the 34th percentile, so this data cannot tell that pattern from chance.
Two lessons for comparing groups this way. First, judge each group against random groups of the same size, not just against the floor: the floor ignores how noisy a small group's components are. Second, look at variance as well as angles: two groups can share directions and still differ a lot in how much they vary along them.
This page was written to explain how to compare low-rank adapters trained on different domains. Each adapter's update writes into a subspace, and the same tools apply: principal angles, a random floor and a split-half ceiling. With adapters, n is in the thousands and r is 4 to 8, so the floor r/n is close to zero, and the ceiling from a second training seed matters more than the floor.