Flavor mixing and CP violation from the interplay of $S_4$ modular group and gCP

We have performed a systematical analysis of lepton and quark masses models based on $\Gamma_4\cong S_4$ modular symmetry with gCP symmetry. We have considered both cases that neutrinos are Majorana particles and Dirac particles. All possible nontrivial representation assignments of matter fields are considered, and the most general form of fermion mass matrices are given. The phenomenologically viable models with the lowest number of free parameters together with the results of fit are presented. We find out nine lepton models with seven real free parameters including the real and imaginary parts of modulus for Majorana neutrinos, which can accommodate the lepton masses and neutrino oscillation data. The prediction for leptogenesis is studied in an example lepton model. The observed baryon asymmetry as well as lepton masses and mixing angles can be explained. For Dirac neutrinos, four lepton models with five real free couplings are compatible with experimental data. Ten quark models containing seven couplings are found to be able to accommodate the hierarchical quark masses and mixing angles and CP violation phase. Furthermore, the $S_4$ modular symmetry can provide a unified description of lepton and quark flavor structure, and a benchmark model is presented.