Since the existing earthquake damage prediction methods cannot make rapid predictions for brick masonry structures. A rapid prediction method for earthquake damage of brick masonry structures is proposed. The method uses a machine learning model, considering the ground motion characteristics and structural characteristics. 12 ground motion parameters that represent the ground motion characteristics and 7 structural parameters that have a strong correlation with the damage of brick masonry structures are selected. The ground motion parameters are considered in four aspects: time domain, frequency domain, response spectrum and holding time, and the structural parameters are considered in terms of bearing capacity and stiffness. Three machine learning models based on support vector machine, random forest and artificial neural network are given for fast prediction of seismic damage of brick masonry structures. The input parameters were further optimized using correlation analysis, and the optimal model after optimizing the input parameters was given. The results show that the ANN model has the highest prediction accuracy of 91.56% when 19 input parameters were used. The prediction accuracy of the RF model-based earthquake damage prediction method was higher when 12 optimized parameters were used as inputs, reaching 90.01%. The prediction performance of the RF-based model was more stable when the input parameters were gradually reduced. The optimized input parameters of the RF model-based prediction method can achieve rapid prediction of seismic damage to brick masonry structures. The method that considers both structural and ground vibration parameters as input greatly improves the accuracy of prediction compared to the method that considers only structural parameters or only ground vibration parameters as input.