On the basis of 2FSK digital modulation full-duplex systems, the paper finds that technical effect of applying conventional adaptive cancellation is not optimal. Aimed at this problem, the adaptive cancellation performance is identified. In order to analyze correlation property of 2FSK from a theoretical point of view, the paper constructs a full-duplex orthogonal 2FSK systems based on the RF domain self-interference cancellation structure based on the minimum mean square error (MMSE) criterion, which can confirm the influence of signal correlation on the acceptance of Bit Error Rate(BER). Based on the properties of generalized cyclic stationary stochastic processes, the paper derives the relations between signals correlation and BER. Then, the paper inhibits the impact of relevance in the White Gaussian Noise (AWGN) by an effective method. The result shows that the correlation coefficient ρ of the 2FSK signal is the linear sum of the symbol waveform correlation coefficients of the two frequencies. The range of the correlation coefficient is 0 ～ 0.5. After the signal of 2FSK is removed by self-interference, the error performance deteriorates with the increase of correlation coefficient ρ