Nonlinear Atom-Photon Interaction Induced Population Inversion and Inverted Quantum Phase Transition of Bose-Einstein Condensate in an Optical Cavity

In this paper we explore the rich structure of macroscopic many-particle quantum states for Bose- Einstein condensate in an optical cavity with the tunable nonlinear atom-photon interaction [Nature (London) 464, 1301 (2010)]. Population inversion, bistable normal phases and the coexistence of normal{superradiant phases are revealed by adjusting of the experimentally realizable interaction strength and pump-laser frequency. For the negative (effective) cavity-frequency we observe remark- ably an inverted quantum phase transition (QPT) from the superradiant to normal phases with the increase of atom-field coupling, which is just opposite to the QPT in the normal Dicke model. The bistable macroscopic states are derived analytically in terms of the spin-coherent-state variational method by taking into account of both normal and inverted pseudospin states.