Dynamic graph convolutional neural network for image sentiment distribution prediction

SU Yu-ting, WANG Ji, ZHAO Wei, JING Pei-guang

PDF(1132 KB)
PDF(1132 KB)
J Jilin Univ Eng Tech Ed ›› 2023, Vol. 53 ›› Issue (09) : 2601-2610. DOI: 10.13229/j.cnki.jdxbgxb.20211169

Dynamic graph convolutional neural network for image sentiment distribution prediction

Author information +
History +

Abstract

Aiming at the problem that there exists semantic gap between visual features and high-level emotional semantics and the subjectivity and ambiguity of emotional labels in image sentiment distribution learning， this paper proposes an Emotional Semantic Dynamic Graph Convolution Network （ESDGCN）. In this framework， the Emotion Activation Module （EAM） is constructed to automatically locate the emotional semantic regions to effectively mine the content representation that fits the emotional semantics. In addition， the Semantic Dynamic Graph Convolution Network （SDGCN） is to adaptively capture the semantic relevance between labels. Finally， we adopt the parallel structure to jointly consider local semantic emotional information and label correlations. Experimental results on three open emotional datasets demonstrate the effectiveness of the proposed method.

Key words

information processing technology / visual sentiment computing / dynamic graph convolution / label distribution learning

Cite this article

EndNote

Ris (Procite)

Bibtex

Download Citations
SU Yu-ting , WANG Ji , ZHAO Wei , et al. Dynamic graph convolutional neural network for image sentiment distribution prediction. Journal of Jilin University(Engineering and Technology Edition). 2023, 53(09): 2601-2610 https://doi.org/10.13229/j.cnki.jdxbgxb.20211169

References

List( Publishing order | Descend order by publishing year | Descend order by cited within ) Chart analysis
1
Zhou L, Fan X, Ma Y, et al. Uncertainty-aware cross-dataset facial expression recognition via regularized conditional alignment[C]//Proceedings of ACM International Conference on Multimedia, New York, USA, 2020: 2964-2972.
本文引用 [1]
2
Farzaneh A H, Qi X. Discriminant distribution-agnostic loss for facial expression recognition in the wild[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Piscataway, USA, 2020: 406-407.
3
卢洋, 王世刚, 赵文婷, 等. 基于离散Shearlet类别可分性测度的人脸表情识别方法[J].吉林大学学报: 工学版, 2019, 49(5): 1715-1725.
Lu Yang, Wang Shi-gang, Zhao Wen-ting, et al. Facial expression recognition based on separability assessment of discrete Shearlet transform[J]. Journal of Jilin University (Engineering and Technology Edition), 2019, 49(5): 1715-1725.
4
方明, 陈文强. 结合残差网络及目标掩膜的人脸微表情识别[J].吉林大学学报: 工学版, 2021, 51(1): 303-313.
Fang Ming, Chen Wen-qiang. Face micro-expression recognition based on ResNet with object mask[J]. Journal of Jilin University (Engineering and Technology Edition), 2021, 51(1): 303-313.
本文引用 [1]
5
Huang F, Wei K, Weng J, et al. Attention-based modality-gated networks for image-text sentiment analysis[J]. ACM Transactions on Multimedia Computing, Communications, and Applications, 2020, 16(3): 1-19.
本文引用 [1]
6
Ji R, Chen F, Cao L, et al. Cross-modality microblog sentiment prediction via bi-layer multimodal hypergraph learning[J]. IEEE Transactions on Multimedia, 2018, 21(4): 1062-1075.
7
Jian M, Dong J, Gong M, et al. Learning the traditional art of Chinese calligraphy via three-dimensional reconstruction and assessment[J]. IEEE Transactions on Multimedia, 2019, 22(4): 970-979.
本文引用 [1]
8
Yang J, She D, Lai Y K, et al. Retrieving and classifying affective images via deep metric learning[C]//Proceedings of the AAAI Conference on Artificial Intelligence, Palo Alto, USA, 2018: 491-498.
本文引用 [1]
9
Yao X, She D, Zhao S, et al. Attention-aware polarity sensitive embedding for affective image retrieval[C]//Proceedings of the IEEE International Conference on Computer Vision. Piscataway, USA, 2019: 1140-1150.
本文引用 [1]
10
Li Z, Liu J, Zhu X, et al. Image annotation using multi-correlation probabilistic matrix factorization[C]//Proceedings of the ACM International Conference on Multimedia, New York, USA, 2010: 1187-1190.
本文引用 [1]
11
Li Z, Tang J, He X. Robust structured nonnegative matrix factorization for image representation[J]. IEEE Transactions on Neural Networks and Learning Systems, 2017, 29(5): 1947-1960.
本文引用 [1]
12
Yang X, Song X, Feng F, et al. Attribute-wise explainable fashion compatibility modeling[J]. ACM Transactions on Multimedia Computing, Communications, and Applications, 2021, 17(1): 1-21.
本文引用 [1]
13
Yang X, Song X, Han X, et al. Generative attribute manipulation scheme for flexible fashion search[C]//Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, New York, USA, 2020: 941-950.
本文引用 [1]
14
Geng X. Label distribution learning[J]. IEEE Transactions on Knowledge and Data Engineering, 2016, 28(7): 1734-1748.
本文引用 [6]
15
Peng K C, Chen T, Sadovnik A, et al. A mixed bag of emotions: model, predict, and transfer emotion distributions[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Piscataway, USA, 2015: 860-868.
本文引用 [1]
16
Geng X, Yin C, Zhou Z H. Facial age estimation by learning from label distributions[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35(10): 2401-2412.
本文引用 [4]
17
Yang J, Sun M, Sun X. Label distribution learning via augmented conditional probability neural network[C]//Proceedings of the AAAI Conference on Artificial Intelligence, Palo Alto, USA, 2017: 224-230.
本文引用 [4]
18
Zhou Y, Xue H, Geng X. Emotion distribution recognition from facial expressions[C]//Proceedings of the ACM International Conference on Multimedia, New York, USA, 2015: 1247-1250.
19
Ren T, Jia X, Li W, et al. Label distribution learning with label-specific features[C]//Proceedings of the International Joint Conference on Artificial Intelligence, San Mateo, USA, 2019: 3318-3324.
20
Zhao S, Yao H, Gao Y, et al. Continuous probability distribution prediction of image emotions via multitask shared sparse regression[J]. IEEE Transactions on Multimedia, 2016, 19(3): 632-645.
本文引用 [1]
21
Plutchik R. Emotions: a general psychoevolutionary theory[J]. Approaches to Emotion, 1984(1984): 197-219.
本文引用 [1]
22
Xu M, Zhou Z H. Incomplete label distribution learning[C]//Proceedings of the International Joint Conference on artificial intelligence, San Mateo, USA, 2017: 3175-3181.
本文引用 [1]
23
Jia X, Li Z, Zheng X, et al. Label distribution learning with label correlations on local samples[J]. IEEE Transactions on Knowledge and Data Engineering, 2019, 33(4): 1619-1631.
本文引用 [5]
24
Jia X, Zheng X, Li W, et al. Facial emotion distribution learning by exploiting low-rank label correlations locally[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Piscataway, USA, 2019: 9841-9850.
本文引用 [5]
25
Chen T, Yu F X, Chen J, et al. Object-based visual sentiment concept analysis and application[C]//Proceedings of the ACM International Conference on Multimedia, New York, USA, 2014: 367-376.
26
Su Y T, Zhao W, Jing P G, et al. Exploiting low-rank latent gaussian graphical model estimation for visual sentiment distribution[J]. IEEE Transactions on Multimedia, 2022,25: 1243-1255.
本文引用 [1]
27
缪裕青, 雷庆庆, 张万桢, 等. 多视觉目标融合的图像情感分析研究[J]. 计算机应用研究, 2021, 38(4): 1250-1255.
Miao Yu-qing, Lei Qing-qing, Zhang Wan-zhen, et al. Research on image sentiment analysis based on multi-visual object fusion[J] Application Research of Computers, 2021, 38(4): 1250-1255.
本文引用 [1]
28
盛家川, 陈雅琦, 王君, 等. 深度学习结构优化的图像情感分类[J]. 红外与激光工程, 2020, 49(11): 264-273.
Sheng Jia-chuan, Chen Ya-qi, Wang Jun, et al. Image sentiment classification via deep learning structure optimization[J] Infrared and Laser Engineering, 2020, 49(11): 264-273.
29
Chen T, Borth D, Darrell T, et al. Deepsentibank: visual sentiment concept classification with deep convolutional neural networks[J/OL]. [2021-10-25].
https://arxiv.org/pdf/1410.8586v1.pdf
30
Zhu X, Li L, Zhang W, et al. Dependency exploitation: a unified CNN-RNN approach for visual emotion recognition[C]//Proceedings of the International Joint Conference on Artificial Intelligence, San Mateo, USA, 2017: 3595-3601.
31
Campos V, Salvador A, Giró-i-Nieto X, et al. Diving deep into sentiment: understanding fine-tuned CNNs for visual sentiment prediction[C]//Proceedings of the 1st International Workshop on Affect & Sentiment in Multimedia, New York, USA, 2015: 57-62.
本文引用 [1]
32
Campos V, Jou B, Giro-i-Nieto X. From pixels to sentiment: fine-tuning CNNs for visual sentiment prediction[J]. Image and Vision Computing, 2017, 65(1): 15-22.
33
You Q, Luo J, Jin H, et al. Robust image sentiment analysis using progressively trained and domain transferred deep networks[C]//Proceedings of the AAAI Conference on Artificial Intelligence, Palo Alto, USA, 2015: 381-388.
本文引用 [1]
34
Yang J, She D, Sun M. Joint image emotion classification and distribution learning via deep convolutional neural network[C]//Proceedings of the International Joint Conference on Artificial Intelligence, San Mateo, USA, 2017: 3266-3272.
本文引用 [5]
35
徐冰冰, 岑科廷, 黄俊杰, 等. 图卷积神经网络综述[J].计算机学报, 2020, 43(5): 755-780.
Xu Bing-bing, Cen Ke-yan, Huang Jun-jie, et al. A survey on graph convolutional neural network[J] Chinese Journal of Computers, 2020, 43(5): 755-780.
本文引用 [1]
36
Chen T, Xu M, Hui X, et al. Learning semantic-specific graph representation for multi-label image recognition[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, Piscataway, USA, 2019: 522-531.
37
He T, Jin X. Image emotion distribution learning with graph convolutional networks[C]//Proceedings of the International Conference on Multimedia Retrieval, New York, USA, 2019: 382-390.
本文引用 [1]
38
Zhou B, Khosla A, Lapedriza A, et al. Learning deep features for discriminative localization[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Los Alamitos, USA, 2016: 2921-2929.
本文引用 [1]
39
Yang J, Sun M, Sun X. Label distribution learning via augmented conditional probability neural network[C]//Proceedings of the AAAI Conference on Artificial Intelligence, Palo Alto, USA, 2017: 224-230.
本文引用 [2]
40
Peng K C, Chen T, Sadovnik A, et al. A mixed bag of emotions: model, predict, and transfer emotion distributions[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Piscataway, USA, 2015: 860-868.
本文引用 [1]
41
Jia X, Li W, Liu J, et al. Label distribution learning by exploiting label correlations[C]//Proceedings of the AAAI Conference on Artificial Intelligence, Palo Alto, USA, 2018: 3310-3317.
本文引用 [4]
42
Xiong H, Liu H, Zhong B, et al. Structured and sparse annotations for image emotion distribution learning[C]//Proceedings of the AAAI Conference on Artificial Intelligence, Palo Alto, USA, 2019: 363-370.
本文引用 [4]

Comments

PDF(1132 KB)

Accesses

Citation

Detail
Sections
Recommended
Copyright © J Jilin Univ Eng Tech Ed, All Rights Reserved.
Powered by Beijing Magtech Co. Ltd.

/

〈 〉