In this paper, we consider a different setting for cross-modal retrieval where data from different modalities are implicitly linked via concepts that must be inferred by high-level reasoning we call this setting implicit concept association. Traditional cross-modal retrieval assumes explicit association of concepts across modalities, where there is no ambiguity in how the concepts are linked to each other, e.g., when we do the image search with a query 'dogs', we expect to see dog images. We demonstrate our approach on both image-text and video-text retrieval scenarios using MS-COCO, TGIF, and our new MRW dataset. To facilitate further research in video-text retrieval, we release a new dataset of 50K video-sentence pairs collected from social media, dubbed MRW (my reaction when). Here, we also tackle a more challenging case of video-text retrieval. Most existing work on cross-modal retrieval focuses on image-text data. To learn visual-semantic embedding, we tie-up two PIE-Nets and optimize them jointly in the multiple instance learning framework. In this work, we introduce Polysemous Instance Embedding Networks (PIE-Nets) that compute multiple and diverse representations of an instance by combining global context with locally-guided features via multi-head self-attention and residual learning. This hinders its use in real-world scenarios where individual instances and their cross-modal associations are often ambiguous. Unfortunately, injective embedding cannot effectively handle polysemous instances with multiple possible meanings at best, it would find an average representation of different meanings. Most current methods learn injective embedding functions that map an instance to a single point in the shared space. Visual-semantic embedding aims to find a shared latent space where related visual and textual instances are close to each other.

Mfw gif machine gun