For the current pattern recognition field, there are few researches specifically aimed at hand-held object recognition, and an analysis algorithm that can analyze the state of human hand-held objects and the types of hand-held objects in real-time and globally is proposed, preliminary processing of the image based on the human pose estimation network Openpose and the object detection network Yolo, the C++API is used to fuse the coordinates of the human body joint points and the target object coordinates obtained by the two, and then classify and separate objects of different sizes. The judgment rule is designed , and the IOU algorithm is used as the auxiliary judgment of the hand-held state, and finally the behavior analysis algorithm of the human hand-held object is realized. Collect the video stream of the hand-held object into a data set, and use a variety of methods for data enhancement and training, the final algorithm recognizes the state of the handheld object ,and at the same time the accuracy of correctly identifying the category of the handheld object can reach about 91.2%, compared with the traditional method it has increased by about 1.3%, and the running rate can trach 13 fps, which verifies the accuracy of the algorithm. The algorithm has high application value for the detection of abnormal behaviors of dangerous goods such as handheld knives and guns