Abstract

Nowadays there is a large amount of image and text data available in several large databases, however, properly aligned image and sentence data, a crucial requirement of any joint learning model, is quite hard to come by. To address this problem, we present a system that can jointly cluster images and text descriptions, and provide aligned data from unaligned uni-modal databases. The contribution of this work is twofold: we first present a system that learns a common embedding space in which image and text description form valid and sensible clusters. Second, we use our system for image captioning and retrieval tasks, and compare to the state of the art approaches. We use Caltech Birds Dataset [29] as reference data set for training and testing.

Date

2024

Authors

Mahyar Khayatkhoei, Aditya Chukka, Chaitanya Mitash

Journal

downloaded from the internet on Dec

Volume

6