Probing triple-Higgs productions via $4b2\gamma$ decay channel at a 100 TeV hadron collider

The quartic self-coupling of the Standard Model Higgs boson can only be measured by observing the triple-Higgs production process, but it is challenging for the Large Hadron Collider (LHC) Run 2 or International Linear Collider (ILC) at a few TeV because of its extremely small production rate. In this paper, we present a detailed Monte Carlo simulation study of the triple-Higgs production through gluon fusion at a 100 TeV hadron collider and explore the feasibility of observing this production mode. We focus on the decay channel $HHH\rightarrow b\bar{b}b\bar{b}\gamma\gamma$, investigating detector effects and optimizing the kinematic cuts to discriminate the signal from the backgrounds. Our study shows that, in order to observe the Standard Model triple-Higgs signal, the integrated luminosity of a 100 TeV hadron collider should be greater than $1.8\times 10^4$ ab$^{-1}$. We also explore the dependence of the cross section upon the trilinear ($\lambda_3$) and quartic ($\lambda_4$) self-couplings of the Higgs. We find that, through a search in the triple-Higgs production, the parameters $\lambda_3$ and $\lambda_4$ can be restricted to the ranges $[-1, 5]$ and $[-20, 30]$, respectively. We also examine how new physics can change the production rate of triple-Higgs events. For example, in the singlet extension of the Standard Model, we find that the triple-Higgs production rate can be increased by a factor of $\mathcal{O}(10)$.