Research
Multimodal Attention in Recurrent Neural Networks for Visual Question Answering
Visual Question Answering (VQA) is a task for evaluating image scene understanding abilities and shortcomings and also measuring machine intelligence in the visual domain. Given an image and a natural question about the image, the system must ground the question into the image and return an accurate answer in a natural language. A lot of progress has been done to address the challenges of this task by combining latest advances in image representation and natural language processing. Several recently proposed solutions include attention mechanisms designed to support “reasoning”. These mechanisms allow models to focus on specific part of the input in order to generate the answer and improve its accuracy. In this paper we present a novel LSTM architecture for VQA that uses multimodal attention to focus over specific parts of the image and also on specific question words to generate the answer. We evaluate our model on the VQA dataset and demonstrate that it performs better than state of the art. We also make a qualitative analysis of the results and show the abilities and shortcomings of our model.
