Bistable cooling in optomechanical system

A scheme is presented to optimize the optomechanical cooling of mechanical resonator in instability regime. Based on the stability analysis, we uncovered a distinct bistable effect of photons and phonons, which can be used to realize a strong nonlinear effect even in the single-photon weak coupling regime. Considering the experimental realization, we investigate the sideband cooling in bistable regime with and without quantum nonlinearity. It is shown that the fluctuation of the steady state phonons can be excellently suppressed at a rather low level due to the anti-rotating-wave effect, and it does not require high quality factor of the cavity. Our scheme offers a new perspective for optimizing the sideband cooling of mechanical resonators in the weak coupling regime.