Disentangling Modified Gravity and Massive Neutrinos with Intrinsic Shape Alignments of Massive Halos

We present two new diagnostics based on the intrinsic shape alignments of group/cluster size dark matter halos to disentangle the effect of $f(R)$ gravity from that of massive neutrinos. Using the snapshot data from a series of the {\small DUSTGRAIN}-{pathfinder} $N$-body simulations for the Planck $\Lambda$CDM cosmology and three $f(R)$ gravity models with massive neutrinos ($\nu$), we first determine the probability density functions of the alignment angles between the shape orientations of massive halos and the minor principal axes of the local tidal fields. The numerically obtained results turn out to agree very well with the analytic formula derived under the assumption that the anisotropic merging along the cosmic web induces the halo shape alignments. The four cosmologies, which several standard diagnostics failed to discriminate, are found to yield significantly different best-fit values of the single parameter that characterizes the analytic formula. We also numerically determine the spatial cross-correlations between the shape orientations of neighbor group/cluster halos, and find them to be in good agreements with a fitting formula characterized by two parameters, whose best-fit values are found to substantially differ among the four models. We also discuss the limitations and caveats of these new diagnostics that must be overcome for the application to real observational data.