Karhunen-Loeve eigenvalue problems in cosmology: how should we tackle large data sets?

Since cosmology is no longer "the data-starved science", the problem of how to best analyze large data sets has recently received considerable attention, and Karhunen-Loeve eigenvalue methods have been applied to both galaxy redshift surveys and Cosmic Microwave Background (CMB) maps. We present a comprehensive discussion of methods for estimating cosmological parameters from large data sets, which includes the previously published techniques as special cases. We show that both the problem of estimating several parameters jointly and the problem of not knowing the parameters a priori can be readily solved by adding an extra singular value decomposition step. It has recently been argued that the information content in a sky map from a next generation CMB satellite is sufficient to measure key cosmological parameters (h, Omega, Lambda, etc) to an accuracy of a few percent or better - in principle. In practice, the data set is so large that both a brute force likelihood analysis and a direct expansion in signal-to-noise eigenmodes will be computationally unfeasible. We argue that it is likely that a Karhunen-Loeve approach can nonetheless measure the parameters with close to maximal accuracy, if preceded by an appropriate form of quadratic "pre-compression". We also discuss practical issues regarding parameter estimation from present and future galaxy redshift surveys, and illustrate this with a generalized eigenmode analysis of the IRAS 1.2 Jy survey optimized for measuring beta=Omega^{0.6}/b using redshift space distortions.