Detecting soil-transmitted helminth and Schistosoma mansoni eggs in Kato-Katz stool smear microscopy images: A comprehensive in- and out-of-distribution evaluation of YOLOv7 variants

Abstract

Background: Soil-transmitted helminth (STH) and Schistosoma mansoni (S. mansoni) infections remain significant public health concerns in tropical and subtropical regions. Deep Convolutional Neural Networks (DCNNs) have already shown promising accuracy in identifying STH and S. mansoni eggs in the same, in-distribution (ID) settings. However, their performance in real-world, out-of-distribution (OOD) scenarios, characterized by variations in image capture devices and the appearance of previously unseen egg types, has not been thoroughly investigated. Assessing the robustness of DCNNs under these challenging conditions is crucial for ensuring their reliability in field diagnostics.

Methodology: Our study addresses the gap in evaluating DCNNs for identifying STH and S. mansoni eggs by rigorously testing multiple variants of the You Only Look Once (YOLO) version 7 model under two OOD conditions: (i) a dataset shift due to a change in the image capture device, and (ii) a combination of this device change and the presence of two egg types not occurring during training. We adopted a 2 3 montage data augmentation strategy to enhance OOD generalization. Additionally, we used the Toolkit for Identifying object Detection Errors (TIDE) and Gradient-weighted Class Activation Mapping (Grad-CAM) to perform a comprehensive analysis of the results.

Principal findings: In ID settings, YOLOv7-E6E outperformed other models, achieving an F1-score of 97.47%. For the OOD scenario involving only a change in the image capture device, the 2 3 montage strategy significantly enhanced performance, increasing precision by 8%, recall by 14.85%, and mAP@IoU0.5 by 21.36%. However, for the more complex OOD scenario that involves both a change in the capture device and the introduction of two previously unseen egg types, the proposed augmentation technique, while beneficial, did not fully address the generalization challenges across all YOLOv7 variants, highlighting the necessity of testing beyond ID scenarios, on which state-of-the-art models predominantly focus.

Conclusions/significance: This study underscores the critical importance of utilizing comprehensive test sets and conducting rigorous OOD evaluations when designing machine learning solutions for STH, S. mansoni or any other helminth infections. Understanding the true capabilities of DCNNs in real-world settings depends on such thorough testing. Expanding AI-driven diagnostic assessments to account for the complexities encountered in the field is essential for creating robust tools that can significantly contribute to the global elimination of STH and S. mansoni infections as public health problems by 2030, a goal put forth by the World Health Organization.