Multimodal medical image retrieval: Image categorization to improve search precision

Effective medical image retrieval can be useful in the clinical care of patients, education and research. Traditionally, image retrieval systems have been text-based, relying on the annotations or captions associated with the images. Although text-based information retrieval methods are mature and well researched, they are limited by the quality and availability of the annotations associated with the images. Advances in computer vision have led to methods for using the image itself as the search entity. However, the success of purely content-based techniques, when applied to a diverse set of clinical images, has been somewhat limited and these systems have not had much success in the medical domain. On the other hand, as demonstrated in recent years, a combination of text-based and content-based image retrieval techniques can achieve improved retrieval performance if combined effectively. There are many approaches to multimodal retrieval including early and late fusion of weighed results from the different search engines. In this work, we use automatic annotation based on visual attributes to label images as part of the indexing process and the subsequently use these labels to filter or reorder the results during the retrieval process. Labels for medical images can be categorized along three dimensions - imaging modality, anatomical location and image finding or pathology. Our previous research has indicated that the imaging modality is most easily identified using visual techniques whereas the caption or textual annotation frequently contains the finding or pathological information about the image. Thus, it is best to use visual methods to filter the modality and occasionally, anatomy while it is better to use the textual annotation to find the finding of interest. We have created a modality classifier for the weakly labeled images in our collection using a novel approach that combines affinity propagation for the selection of class exemplars, textons and patch-based descriptors as visual features and a NaiveBayes Nearest Neighbor technique for the classification of modality using visual features. We demonstrate significant improvement in precision attained using this technique for the ImageCLEF medical retrieval task 2009 using both our textual runs as well as runs from all participants in 2009.