In recent years mineral image recognition has become increasingly important for mineral identification with the use of deep learning. While such application has gained some success, further improvement is needed to enhance the identification accuracy on large-scale mineral datasets. The fine differences in morphology, texture, and color between different minerals may align with the characteristics of fine-grained recognition algorithms, yet results of fine-grained recognition for mineral identification have rarely been reported. This paper proposes a fine-grained mineral identification model based on Next-ViT, which allows precise classification of mineral images by progressive model training across different granularity levels. In this approach, Next-ViT, which combines the advantages of transformer and convolutional neural network, is utilized to extract rich image features; a random jigsaw generator is then employed to create mineral puzzles at different granularity levels encompassing various information from detailed to general. The model training involves progressive learning across multiple granularity levels. In the early stages, the model primarily focuses on fine-grained features, learning detailed information from the puzzles to differentiate between different minerals; as training progresses, model learning gradually shifts to higher granularity levels, capturing more abstract and global information. Through this approach, the model can effectively utilize information across multiple granularity levels, thereby improving the accuracy of mineral identification. Experimental results demonstrated the effectiveness of this approach, with the proposed model achieving an accuracy of 86.5% in mineral identification on a dataset on 36 common minerals.