Smecticlike phase for modulated XY spins in two dimensions

The row model for frustrated XY spins on a triangular lattice in 2D is used to study incommensurate{IC}) spiral and commensurate{C} antiferromagnetic (AF) phases, in the regime where a C-IC transition occurs. Using fluctuating boundary conditions and specific histogram techniques, a detailed Monte Carlo (MC) study reveals more structure in the phase diagram than found in previous MC simulations of the full parameter space. On the (C) side, equilibrium configurations consist of alternating stripes of spiral phases of opposite chirality separated by walls of the (C) phase. For this same parameter regime, thermodynamic quantities are computed analytically using the NSCHA, a generalization of the self consistent harmonic approximation appropriate for chiral systems. On the commensurate side of the (C)-(IC) boundary, NSCHA predicts an instability of the (C) phase. This suggests that the state is spatially inhomogeneous, consistent with the present MC result: it resembles the smectic-A phase of liquid crystals, and its existence implies that the Lifshitz point is at ${T=0}$ for modulated XY spins in 2D. The connection between frustrated XY systems and the vortex state of strong type II superconductors suggests that the smectic phase may correspond to a vortex liquid phase of superconducting layers.