Long-term Evolution of Tightly-Packed Stellar Black Holes in AGN Disks: Formation of Merging Black-Hole Binaries via Close Encounters

We study the long-term evolution of two or more stellar black holes (BHs) on initially separated but unstable circular orbits around a supermassive BH (SMBH). Such a close-packed orbital configuration can naturally arise from BH migrations in the AGN disk. Dynamical instability of the orbits leads to recurring close encounters between two BHs, during which the BH separation $r_{\rm p}$ becomes less than the Hill radius $R_{\rm H}$. In the rare very close encounters (with $r_{\rm p}$ several orders of magnitude less than $R_{\rm H}$), a tight merging BH binary can form with the help of gravitational wave emission. We use $N$-body simulations to study the time evolution of close encounters of various degrees of "closeness" and the property of the resulting binary BH mergers. For a typical "SMBH + 2 BHs" system, the averaged cumulative number of close encounters (with $r_{\rm p} \lesssim R_{\rm H}$) scales approximately as $\propto t^{0.5}$. The minimum encounter separation $r_{\rm p}$ follows a linear cumulative distribution $P(<r_{\rm p}) \propto r_{\rm p}$ for $r_{\rm p} \ll R_{\rm H}$. From these, we obtain a semi-analytical expression for the averaged rate of binary captures that lead to BH mergers. Our results suggest that close-packed BHs in AGN disks may take a long time ($\gtrsim 10^7$ orbits around the SMBH) to experience a sufficiently close encounter and form a bound binary, although this time can be shorter if the initial BH orbits are highly aligned. The BH binary mergers produced in this scenario always have high eccentricities when entering the LIGO band, and have a broad distribution of orbital inclinations relative to the original AGN disk. We also explore the effects of the gas disk and find that simple gas drags on the BHs do not necessarily lead to an enhanced BH binary capture rate.