Instability of Nagaoka State and Quantum Phase Transition via Kinetic Frustration Control

We investigate the Nagaoka-Thouless (NT) ferromagnetic instability in the strongly interacting $t$-$t'$ Hubbard model by continuously breaking particle-hole symmetry on a tunable square-triangular lattice geometry. We use an analytic approach to show that the fully spin-polarized state becomes unstable to a metastable spin-polaron when the kinetic frustration $t'/t$ exceeds a critical, dimension-dependent value. Large-scale density matrix renormalization group simulations reveal a quantum phase transition from the NT ferromagnet to a spiral spin-density wave, which evolves continuously into the Haerter-Shastry antiferromagnet in the large-frustration limit. Remarkably, this transition remains robust at low but finite hole density, making it accessible in cold-atom and moir\'e Hubbard platforms under strong interactions. A variational analysis further captures the instability mechanism at finite density via frustration-induced magnon band deformation.