Aimed at the problems that the existing malicious code detection model is iow in recognition rate of malicious code and its variants, and the amount of parameters is too large, a method of Malicious code family detection model is proposed based on GhostDenseNetSE is proposed in combination with the lightweight convolutied Ghost, densely connected network DenseNet and the channel domain attention mechanism SE. This model serves as a type of compressing the model volume and improving the recognition rate. the standard convolutional layer in DenseNet is replaced with a lightweight Ghost module, and the channel domain attention mechanism is introduced to assign different weights to the feature channels to extract the key features of malicious code and improve model checking accuracy. The experimental results on the Malimg data set show that the models recognition accuracy of malicious code families can reach 9914%, Compared with AlexNet and VGGNet, the recognition accuracy increases by 1.34% and 2.98% respectively, and the amount of model parameters is lower. This algorithm not only improves the classification accuracy, but also reduces the complexity of the model. The algorithm has important engineering value and practical significance in malicious code detection.