Using media fusion and domain dimensions to improve precision in medical image retrieval

Saïd Radhouani, Jayashree Kalpathy-Cramer, Steven Bedrick, Brian Bakke, William Hersh

Research output: Chapter in Book/Report/Conference proceedingConference contribution

6 Scopus citations


In this paper, we focus on improving retrieval performance, especially early precision, in the task of solving medical multimodal queries. The queries we deal with consist of a visual component, given as a set of image-examples, and textual annotation, provided as a set of words. The queries' semantic content can be classified along three domain dimensions: anatomy, pathology, and modality. To solve these queries, we interpret their semantic content using both textual and visual data. Medical images often are accompanied by textual annotations, which in turn typically include explicit mention of their image's anatomy or pathology. Annotations rarely include explicit mention of image modality, however. To address this, we use an image's visual features to identify its modality. Our system thereby performs image retrieval by combining purely visual information about an image with information derived from its textual annotations. In order to experimentally evaluate our approach, we performed a set of experiments using the 2009 ImageCLEFmed collection using our integrated system as well as a purely textual retrieval system. Our integrated approach consistently outperformed our text-only system by 43% in MAP and by 71% in precision within the top 5 retrieved documents. We conclude that this improved performance is due to our method of combining visual and textual features.

Original languageEnglish (US)
Title of host publicationMultilingual Information Access Evaluation II
Subtitle of host publicationMultimedia Experiments - 10th Workshop of the Cross-Language Evaluation Forum, CLEF 2009, Revised Selected Papers
Number of pages8
StatePublished - 2010
Event10th Workshop of the Cross-Language Evaluation Forum, CLEF 2009 - Corfu, Greece
Duration: Sep 30 2009Oct 2 2009

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume6242 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


Other10th Workshop of the Cross-Language Evaluation Forum, CLEF 2009


  • Domain Dimensions
  • Image Classification
  • Image Modality Extraction
  • Media Fusion
  • Medical Image Retrieval
  • Performance Evaluation

ASJC Scopus subject areas

  • Theoretical Computer Science
  • General Computer Science


Dive into the research topics of 'Using media fusion and domain dimensions to improve precision in medical image retrieval'. Together they form a unique fingerprint.

Cite this