In order to improve the recognition effect of tree line grounding fault detection in power system, an improved YOLOv8 model is proposed. The model enhances the feature representation ability by inserting the SimAM attention mechanism, and uses the GIoU loss function to improve the accuracy of the bounding box prediction and improve the fault recognition performance of the model in complex environments. In order to verify the performance of the improved YOLOv8 model, the ablation experiment, the insertion position change experiment of the SimAM attention mechanism module, the loss function selection experiment, and the comparison experiment with other recognition models are carried out. The experimental results show that the improved YOLOv8 model has the highest recognition accuracy, recall rate and average accuracy. The model effectively improves the recognition accuracy of the tree-line grounding fault detection image and provides support for the intelligent operation and maintenance of transmission lines.