Variability among observers utilizing the CellSolutions BestCyte Cell Sorter imaging system for the assessment of urinary tract cytology specimens

Introduction: Image analysis systems are not currently commonly used for evaluating urinary cytology specimens. We evaluated whether the BestCyte Cell Sorter (CellSolutions, Greensboro, NC) imaging system can reliably identify atypical cells in urinary cytology specimens. Methods: Fifty-three consecutive urine cytology specimens underwent 2 preparations: one slide using SurePath (SP; BD Diagnostics, Sparks, MD)^™ for routine clinical evaluation, and a second slide using the CellSolutions F50 system for analysis by the BestCyte Cell Sorter (BCCS) scanning system. Eight observers reviewed atypical cells flagged by BCCS and assigned a BCCS diagnosis to each of the 53 specimens. The observers also blindly reviewed the SP preparation (when available) and assigned an SP diagnosis. The SP diagnoses given by one “expert” observer was considered as a reference diagnosis. Results: There was fair-to-moderate agreement among observers for identifying any atypia and high-grade atypia (Fleiss kappa: 0.417 and 0.338, respectively) using BCCS. Review of SP preparations had slightly better agreement (Fleiss kappa: 0.558 and 0.564, respectively). Intraobserver agreement between the two methods varied greatly between individuals (Cohen's kappa range: 0.260 to 0.647). When a consensus diagnosis could be reached among the observers for cases with surgical follow-up, the consensus diagnosis was concordant in 11 of 12 instances, with one instance being a one-step discrepancy. Conclusions: Specimen review by BCCS resulted in slightly greater interobserver variability than review of routine SP preparations. This may have been due to variations in observer experience and comfort with the use of a digital imaging system, which is further suggested by the wide range of intraobserver agreement among individuals.