The topic of this report will deal with the study of advanced modulation formats for highspeed optical communication systems. Basically, the use of alternate polarization modulation (alP) is considered in order to reduce the inter-symbol interference (ISI) caused mainly by effects such as Chromatic Dispersion (CD), Polarization Mode Dispersion (PMD) and Nonlinear effects. The comparative analysis among the different formats is carried out by means of the obtaining of its different spectrums. A detailed analysis of these and their dependency to the modulation format that generates it is presented. The article presents numerical comparative analysis using commercial software VPI Maker Transmission, where the results obtained show that the formats of alternating polarization are the best results at the bit rate of 40 Gb/s per channel.