MAGNUM: Multi-Modal Attention and Gated Network For Understanding Memes

Abstract

Understanding the underlying meanings of online memes requires analysis of both the visual and verbal semantics in tandem for the identification of complex social signals such as humor, offensiveness, sarcasm and motive. This is a big issue in the multi-modal learning system. The multi-modal Attention and Gated Network for Understanding Memes (MAGNUM) is our state-of-the-art deep learning model that is designed to achieve this multi-task categorization goal. For visual feature extraction, MAGNUM uses ResNet50, and for textual feature derivation, it uses RoBERTa-base using concurrent encoding. The Cross-Attention Fusion method is a game-changer as it enables the learning of the representation in one modality to be adaptively improved through the context signal provided by the other modality. The Gated Multi-modal Fusion (GMF) layer is then used to merge the processed data adaptively. This helps to find the best balance of each modality, improving decisionmaking stability. In the trained and validated manner (based on a strict 5-fold stratified cross-validation procedure), and guided by an Uncertainty Weighting Multi-Task Loss to harmonize inter-task objectives, MAGNUM displays competitive performance on all tasks. The model's advanced fusion approach was able to effectively resolve the complex semantic relationships in the interpretation of the meme, with an overall average Macro F1-score of 0.4278±0.0083.