Ground Truthing for Complex AI Solution for FDA Regulatory Submission

Our client Medcase, the global network for on-demand access to medical expertise, was invited to compete for a contract to provide end-to-end data annotation and labeling services – including recruitment of specialized physicians– on a corpus of complex, medical data. The dataset consisted of neurological MRI studies that would serve as the external validation dataset for regulatory validation studies of an imaging AI solution designed to identify and flag urgent, neurologic findings. In absence of clinical data confirming the diagnosis or condition of patient data in an external validation dataset, the manufacturers must use expert physicians to interpret the data and make independent diagnoses of the data. Because many image findings in brain MRI studies are equivocal, multiple, expert, physicians are used to establish a ‘consensus’ interpretation.

Challenge

The manufacturer built a complex algorithm because brain MRI is a very complicated imaging study which typically contains 5 or more imaging series. Radiologists need to consider all the imaging series and information. Developing a process and tooling to allow neuro-radiologists to render an unbiased yet informed decision about the study requires expert knowledge of imaging physics, neurologic pathophysiology, radiology reading and reporting tools, medical informatics, and clinical performance study methodology for regulated, imaging AI products. 

Goals

Provide guidance and review of study protocols; align and develop reporting guidelines and instruments; provide a clinically equivalent reading environment allowing for annotation and labeling of findings using the structured response per study protocol; create a pipeline to provide reader results and outputs to the manufacturer for use in subsequent clinical validation activities for pre-market clearance. 

Solution

AOG partnered with an expert radiologist to review and refine the study protocol and created a schema for the readers to report their findings. We developed a customized MRI reading environment using a diagnostic viewer solution (Softneta, MedDream v 7.8.0) and a structured reporting tool (Smart Radiology, v1) coordinated by a custom-built web-application allowing the concurrent reading of 300 MRI studies by 3 radiologists in different parts of the US. Any one MRI study may have had 4-5,000 images within it, so a performant and stable reading environment was critical. We developed custom methods to extract structured data from the viewer tooling for localizing and marking-up regions of the imaging studies by parsing the DICOM GSPS information from the viewer and integrated with the reporting data. We created an automated, case-report form that allowed for QC and final consensus determination for a corpus of 300 MRI studies. 

Outcome

Our expertise and knowledge helped our client ‘win’ the business. Because the reference standard “truth” in imaging AI studies is the foundation for the validation studies needed for regulatory clearance of an AI product, it is CRITICAL to ensure the experts truthing the data have access to as much of the imaging study as possible, can interrogate the imaging data as they would in their day-to-day practice, and can report their findings specific for the function of the AI product. During a truthing task, radiologists should not be restricted to make observations on the case which might be incidental but which could be very important for managing biases in the AI algorithm. The resulting imaging and reporting data comprise a powerful and valuable dataset the manufacturer can now use to complete regulatory activities.  Our client Medcase now can utilize this service for other customers.