The transform domain LMS algorithm is integrated with variable step - size LMS algorithm and BLMS algorithm, from which a new transform domain variable step - size BLMS adaptive algorithm is presented. The algorithm, which has the advantages of the above three algorithms, can reduce the cross - correlation of input signals effectively and overcome the conflict, caused by the fixed step -size factor, between high convergence rate and low steady -state maladjustment in LMS algorithm, simultaneously it has a better real time behavior. The computer simulation results indicate that the new algorithm has higher convergence rate, lower maladjustment noise and can be applied effectively to the antenna adaptive transmitter - receiver isolation system of jammer.