Observational prospects of self-interacting scalar superradiance with next-generation gravitational-wave detectors

Current- and next-generation gravitational-wave observatories may reveal new, ultralight bosons. Through the superradiance process, these theoretical particle candidates can form clouds around astrophysical black holes and result in detectable gravitational-wave radiation. In the absence of detections, constraints$-$contingent on astrophysical assumptions$-$have been derived using LIGO-Virgo-KAGRA data on boson masses. However, the searches for ultralight scalars to date have not adequately considered self-interactions between particles. Self-interactions that significantly alter superradiance dynamics are generically present for many scalar models, including axion-like dark matter candidates and string axions. We implement the most complete treatment of particle self-interactions available to determine the gravitational-wave signatures expected from superradiant scalar clouds and revisit the constraints obtained in a past gravitational-wave search targeting the black hole in Cygnus X-1. We also project the reach of next-generation gravitational-wave observatories to scalar particle parameter space in the mass-coupling plane. We find that while proposed observatories have insufficient reach to self-interactions that can halt black hole spin-down, next-generation observatories are essential for expanding the search beyond gravitational parameter space and can reach a mass and interaction scale of $\sim 10^{-13}-10^{-12}$ eV/c$^2$ and $\gtrsim 10^{17}$ GeV, respectively.