Analytical and numerical evaluation of the Debye and Meissner masses in dense neutral three-flavor quark matter

We calculate the Debye and Meissner masses and investigate chromomagnetic instability associated with the gapless color superconducting phase changing the strange quark mass $M_s$ and the temperature $T$. Based on the analytical study, we develop a computational procedure to derive the screening masses numerically from curvatures of the thermodynamic potential. When the temperature is zero, from our numerical results for the Meissner masses, we find that instability occurs for $A_1$ and $A_2$ gluons entirely in the gapless color-flavor locked (gCFL) phase, while the Meissner masses are real for $A_4$, $A_5$, $A_6$, and $A_7$ until $M_s$ exceeds a certain value that is larger than the gCFL onset. We then handle mixing between color-diagonal gluons $A_3$, $A_8$, and photon $A_\gamma$, and clarify that, among three eigenvalues of the mass squared matrix, one remains positive, one is always zero because of an unbroken $U(1)_\tilde{Q}$ symmetry, and one exhibits chromomagnetic instability in the gCFL region. We also examine the temperature effects that bring modifications into the Meissner masses. The instability found at large $M_s$ for $A_4$, $A_5$, $A_6$, and $A_7$ persists at finite $T$ into the $u$-quark color superconducting (uSC) phase which has $u$-$d$ and $s$-$u$ but no $d$-$s$ quark pairing and also into the two-flavor color superconducting (2SC) phase characterized by $u$-$d$ quark pairing only. The $A_1$ and $A_2$ instability also goes into the uSC phase, but the 2SC phase has no instability for $A_1$, $A_2$, and $A_3$. We map the unstable region for each gluon onto the phase diagram as a function of $M_s$ and $T$.