The channel model of MIMO system is introduced first, then the effect of Rayleigh fading on the symbol error probability for 16-QAM and the performance on the capacity of MIMO system are analyzed based on this. The performance of MIMO system in slow flat Rayleigh fading channel is studied from these two aspects including spatial multiplexing and spatial diversity. The results of analysis and simulation show that the symbol error probabilities of spatial multiplexing system realized by V-BLAST algorithm increase with the increase of the number of transmitting and receiving antennas. And different diversity combing techniques have different influence on the system performance in which MRC is the best, EGC takes second place and SC is bad; and the symbol error probabilities of spatial diversity system decrease with the increase of the number of transmitting and receiving antennas.