MAiVAR: Multimodal audio-image and video action recognizer

Document Type

Conference Proceeding

Publication Title

2022 IEEE International Conference on Visual Communications and Image Processing, VCIP 2022

Publisher

IEEE

School

School of Science

RAS ID

53122

Funders

Australian Government

Grant Number

/HRDI-UESTPs/UETs-I/Phase-1/Batch-VI/2018

Comments

Shaikh, M. B., Chai, D., Islam, S. M. S., & Akhtar, N. (2023). MAiVAR: Multimodal audio-image and video action recognizer. In 2022 IEEE International Conference on Visual Communications and Image Processing (VCIP) (pp. 1-5). IEEE. https://doi.org/10.1109/VCIP56404.2022.10008833

Abstract

Currently, action recognition is predominately performed on video data as processed by CNNs. We investigate if the representation process of CNN s can also be leveraged for multimodal action recognition by incorporating image-based audio representations of actions in a task. To this end, we propose Multimodal Audio-Image and Video Action Recognizer (MAiVAR), a CNN-based audio-image to video fusion model that accounts for video and audio modalities to achieve superior action recognition performance. MAiVAR extracts meaningful image representations of audio and fuses it with video representation to achieve better performance as compared to both modalities individually on a large-scale action recognition dataset.

DOI

10.1109/VCIP56404.2022.10008833

Access Rights

subscription content

Link to Full Text

Share

 
COinS