The optomechanical instability in the quantum regime

We consider a generic optomechanical system, consisting of a driven optical cavity and a movable mirror attached to a cantilever. Systems of this kind (and analogues) have been realized in many recent experiments. It is well known that those systems can exhibit an instability towards a regime where the cantilever settles into self-sustained oscillations. In this paper, we briefly review the classical theory of the optomechanical instability, and then discuss the features arising in the quantum regime. We solve numerically a full quantum master equation for the coupled system, and use it to analyze the photon number, the cantilever's mechanical energy, the phonon probability distribution and the mechanical Wigner density, as a function of experimentally accessible control parameters. We observe and discuss the quantum-to-classical transition as a function of a suitable dimensionless quantum parameter.