From sheaves on P2 to a generalization of the Rademacher expansion

Moduli spaces of stable coherent sheaves on a surface are of much interest for both mathematics and physics. Yoshioka computed generating functions of Poincare polynomials of such moduli spaces if the surface is the projective plane P2 and the rank of the sheaves is 2. Motivated by physical arguments, this paper investigates the modular properties of these generating functions. It is shown that these functions can be written in terms of the Lerch sum and theta function. Based on this, we prove a conjecture by Vafa and Witten, which expresses the generating functions of Euler numbers as a mixed mock modular form. Moreover, we derive an exact formula for the Fourier coefficients of this function, which is similar to the Rademacher expansion for weakly holomorphic modular forms but is more complicated. This is the first example of an exact formula for the Fourier coefficients of mixed mock modular forms, which is of independent mathematical interest.