Publication Details
Abstract
Classical multivariate inference assumes that the number of variables p is small relative to the sample size n. In contemporary genomics, finance, imaging, and climate science, p is often of the same order as n, and the classical assumptions fail: sample eigenvalues spread far from their population values, likelihood-ratio tests reject true null hypotheses, and principal components become inconsistent. This paper develops and evaluates spectrally calibrated inference (SCI), a unified workflow that uses random matrix theory (RMT) to calibrate three linked procedures: (i) tests of sphericity, (ii) estimation of the number of latent factors, and (iii) covariance estimation by shrinkage and eigenvalue cleaning. We compare the classical likelihood-ratio test (LRT), John’s corrected test, and a Tracy–Widom test on the largest eigenvalue in Monte Carlo experiments with n = 200 and p/n between 0.1 and 0.9. The classical LRT has empirical size 0.114 at p = 20 and rejects every true null at p ≥ 100, while the two RMT-based tests hold size near the nominal 5% level at p/n ≥ 0.5. Near the Baik–Ben Arous–Péché detectability threshold, the Tracy–Widom test is the most powerful (0.938 versus 0.642 for John’s test at p = 100 and spike strength 1.5). In a synthetic gene-expression-style factor model (p = 200, n = 400, three latent programs), an RMT edge rule with a Tracy–Widom margin recovers the exact number of factors in 99% of replications once factor strength is twice the critical level, and fails, as theory predicts, close to the threshold. Linear shrinkage reduces Frobenius error by 28% to 92% relative to the sample covariance, depending on p/n. All results are simulation-based with a fixed seed and are fully reproducible.