Forward-mode automatic differentiation for the tensor renormalization group and its relation to the impurity method

We propose a forward-mode automatic differentiation (AD) framework for tensor renormalization group methods. In this approach, evaluating the derivatives of the partition function up to the order of $k$ increases the matrix-multiplication cost by a factor of $(k+1)(k+2)/2$ compared to computing the free energy alone, and the memory footprint is only $k+1$ times that of the original calculation. In the limit where the derivatives of the singular value decomposition are neglected, we establish a theoretical correspondence between our forward-mode AD and conventional impurity methods. Numerically, we find that the proposed AD algorithm can calculate internal energy and specific heat significantly higher accuracy than the impurity method at comparable computational cost. We also provide a practical procedure to extract critical exponents from derivatives of the renormalized tensor in tensor renormalization group calculations in both two and three dimensions. In addition, we discuss how to efficiently differentiate an arbitrary tensor network.