This is the implementation of the paper Prompting for Multimodal Hateful Meme Classification (EMNLP 2022).
Sign in to like and favorite skills
This is the implementation of the paper Prompting for Multimodal Hateful Meme Classification (EMNLP 2022).
Hateful meme classification is a challenging multimodal task that requires complex reasoning and contextual background knowledge. Ideally, we could leverage an explicit external knowledge base to supplement contextual and cultural information in hateful memes. However, there is no known explicit external knowledge base that could provide such hate speech contextual information. To address this gap, we propose PromptHate, a simple yet effective prompt-based model that prompts pre-trained language models (PLMs) for hateful meme classification. Specifically, we construct simple prompts and provide a few in-context examples to exploit the implicit knowledge in the pre- trained RoBERTa language model for hatefulmeme classification.
In order to run the code, you need to pre-process data in the following step:
Image Cleaning: This step removes the meme texts of images. We follow the implementation provided in the project, which use OCR detection tools to detect texts of image first, remove and impaint the image.
Caption Generation: We leverage a pre-trained image caption generation tool. To run the code, you need to download the pre-trained models provided by the project and execute codes in
captions-for-hatefulmeme.ipynbEntity and Demographic Information Extraction: We use Google Vision for entity detection and FairFace detection for demographic information extraction. More details can be found in the project.
We also provide the pre-processed data in the
data/domain_splitscaptionOur code is built on transformers and we use its
4.19.2To replicate the reported performance, you can simply run:
bash run.sh
Please cite our paper if you use PromptHate in your work:
@inproceedings{rui2022prompthateemnlp, title={Prompt-Based Hateful Meme Detection}, author={Rui Cao, Roy Ka-Wei Lee, Wen-Haw Chong, Jing Jiang}, booktitle={Empirical Methods in Natural Language Processing (EMNLP)}, year={2022} }
Part of our work is based on the following works:
@inproceedings{gao2021making, title={Making Pre-trained Language Models Better Few-shot Learners}, author={Gao, Tianyu and Fisch, Adam and Chen, Danqi}, booktitle={Association for Computational Linguistics (ACL)}, year={2021} }
@article{mokady2021clipcap, title={ClipCap: CLIP Prefix for Image Captioning}, author={Mokady, Ron and Hertz, Amir and Bermano, Amit H}, journal={arXiv preprint arXiv:2111.09734}, year={2021} }