Aiming at the problems of low efficiency and poor accuracy of traditional steel surface defect detection methods, a steel surface defect detection algorithm based on improved YOLOv5 is proposed in this paper. Firstly, the C3 module and part of convolutional structure in YOLOv5 network are replaced by GhostBottleneck structure to realize the lightweight of network model. Secondly, SE attention mechanism is introduced in Backbone to strengthen the important feature channels. Finally, according to the characteristics of the data set, a detection layer is added to the network to strengthen the feature extraction ability, and a feature fusion structure is added in the Neck part. DW convolution is used to replace part of the standard convolution to reduce the computation. Experimental results show that the improved Yolov5sGSD algorithm reduces the model volume by 10.4%, and the mAP value on the test set is 76.8%. Compared with the original YOLOv5s network, the detection accuracy and speed are obviously higher than some mainstream algorithms. Compared with traditional steel surface defect detection methods, the algorithm proposed in this paper can detect the type and location of steel surface defects more accurately and quickly, and has a smaller model volume, which is convenient for deployment in mobile terminals.