Third post-Newtonian effective-one-body Hamiltonian in scalar-tensor and Einstein-scalar-Gauss-Bonnet gravity

We build an effective-one-body (EOB) Hamiltonian at third post-Newtonian (3PN) order in scalar-tensor (ST) and Einstein-scalar-Gauss-Bonnet (ESGB) theories of gravity. The latter is an extension of general relativity that predicts scalar hair for black holes. We start from the known two-body Lagrangian at 3PN order, and use order-reduction methods to construct its ordinary Hamiltonian counterpart. We then reduce the conservative two-body dynamics to the (nongeodesic) motion of a test particle in an effective metric by means of canonical transformations. The resulting EOB Hamiltonian is a modification of the general relativistic Hamiltonian, and already at 3PN order, it must account for nonlocal-in-time tail contributions. We include the latter beyond circular orbits and up to sixth order in the binary's orbital eccentricity. We finally calculate the orbital frequency at the innermost stable circular orbit (ISCO) of binary black holes in the shift-symmetric ESGB model. Our work extends F.L. Juli\'e and N. Deruelle [Phys. Rev. D 95, 124054 (2017)], and it is an essential step toward the accurate modeling of gravitational waveforms beyond general relativity.