Discover the secrets of FDA-compliant Data Labeling
In 2020, an analysis revealed that 70% of diagnostic AI systems based on visual data are based on data from only three US states. This lack of diversity is a real problem in the health technology sector.
With the evolution of medical data labeling requirements, the Food and Drug Administration (or FDA) has taken concrete steps. In January 2021, FDA released its ”AI/ML-based Software as a Medical Device Action Plan“, establishing for the first time a federal framework in the United States, to regulate AI and Machine Learning in medical devices.
To ensure that these technologies remain safe and effective, for both patients and healthcare professionals, it is essential to fully understand why data labeling is an important part of this system, and what standards should be respected.
💡 In this guide, we'll take a detailed look at FDA requirements, its approach to using training data for AI, and the key steps for implementing an FDA-compliant labeling process in your organization.
-hand-innv.png)
The fundamentals of medical data labeling
In the medical field, we are facing a major challenge: 80% of health data is not structured, making them difficult to exploit. This is precisely where Data Labeling or the implementation of a LabelOps process or DataPrepOps for medical data comes into play.
Medical Data Labeling is a meticulous process of annotating images, videos, and other medical data with accurate and relevant information (also called “metadata”). Indeed, this practice allows AI algorithms to understand and interpret medical images, in particular to identify:
- Anatomical structures
- Specific pathologies
- Potential anomalies
- Important clinical signs
In order to ensure the accuracy of medical diagnoses, data annotation specialists (like Innovatiana) use a variety of annotation methods, including:
- Bounding boxes to demarcate areas of interest
- Polygons to mark precise contours
- Landmarks to identify specific structures
In addition, the quality of labeling is essential because it has a direct impact on the performance of artificial intelligence models, and therefore on diagnostics. At Innovatiana, we use medical experts who can accurately identify symptoms, illnesses, and treatments. These specialists use specialized software such as V7, Encord, Radiant DICOM Viewer or OsiriX to ensure annotations that comply with clinical standards.
FDA regulatory requirements for medical data labeling
To ensure the compliance of Data Labeling in medical artificial intelligence, it is essential to follow the FDA's strict guidelines for labeling and data management. The objective is to ensure that the datasets used to train AI models meet the security, traceability and quality requirements applicable to medical devices.
📌 Classification of medical devices and impact on Data Labeling
The FDA classifies medical devices into three categories based on their level of risk, which directly influences the level of data labeling requirements:
- Class I (low risk): General controls, adapted to non-critical annotation systems (e.g. diagnostic support tools without automated decision-making).
- Class II (moderate risk): Stricter requirements, requiring rigorous validation of the annotations used to train AI models (e.g. automated detection of anomalies on medical imagery).
- Class III (high risk): Requires prior FDA approval, as these systems can have a direct impact on medical decision-making (e.g. AI interpreting scans to diagnose serious conditions).
✅ Implementation of a quality assurance program for Data Labeling
The FDA requires players in the sector to set up a quality assurance program integrating good practices specific to Data Labeling, in particular:
- Data annotation and structuring standards : Compliance with DICOM (medical imaging), HL7 FHIR (data interoperability), and GxP (good manufacturing and data management practices).
- Validating annotations : Rigorous process including cross-reviews by medical experts and AI specialists to ensure the accuracy of the labels applied.
- Complete documentation and traceability : All Data Labeling steps must be recorded in a validation file to prove compliance with regulatory requirements.
📑 Controls and audits to ensure compliance
To meet FDA standards, the Data Labeling process must include strict controls, including:
- Systematic verification of the consistency of annotations via quality metrics (e.g. inter-annotator agreement, statistical analyses).
- Save changes : Any changes in labels or annotation algorithms must be documented and approved prior to application.
- Maintaining version histories : Each dataset used to train a model must be stored with clear versioning in order to trace the origin and evolution of the annotations.
🔍 Validation and acceptance of datasets by the FDA
Before using an annotated dataset to train medical AI, the FDA requires a formal assessment to ensure that it meets quality and safety criteria. This validation includes:
- Performance tests on annotated samples to ensure the robustness of the model.
- FDA compliance checks and the guidelines on AI/ML-based Software as a Medical Device.
- Data Labeling Methods Audit to avoid bias and ensure the representativeness of the data.
💡 By integrating these regulatory requirements as soon as the Data Labeling process is in place, we contribute to the compliance of medical AI models and facilitate their approval by the FDA, thus ensuring reliable and safe solutions for patients and healthcare professionals.
Implementation of a compliant Labeling process
To ensure Data Labeling that complies with regulatory requirements in the context of the development of medical AI, it is essential to structure a rigorous process integrating both human expertise and advanced technological tools.
The first step is to define accurate annotation protocols, aligned with FDA recommendations and industry standards, such as DICOM for medical images and HL7 FHIR for data exchange. We use specialized annotation platforms such as V7 or Encord that ensure consistency and high quality through cross-validations by medical experts and machine learning specialists.
The integration of multi-level annotations is also key:
- Automatic pre-annotation (for images, text, videos, ...) using AI models to speed up processing.
- Human validation by radiologists or clinicians to ensure accuracy.
- Audit and correction through iterations based on reliability metrics.
To process large volumes of data while respecting patient confidentiality, we implement pseudonymization protocols and apply HIPAA or GDPR requirements. In addition, the use of synthetic data is an interesting alternative to overcome the limitations of real medical datasets, especially in terms of diversity and the protection of sensitive data.
💡 Looking for datasets to experiment and develop medical AI models? Do not hesitate to check out our Top 15 !
Finally, a continuous monitoring system ensures that annotation models and datasets evolve in accordance with the latest FDA regulations and industry best practices, thus guaranteeing reliable and usable labeling for training AI models.
Conclusion
FDA requirements for medical data labeling may seem complex at first. However, our analysis shows that a methodical and structured approach makes it possible to achieve the required compliance.
The success of a compliant labeling program is based on three essential pillars. First, the expertise of specialized medical annotators ensures the accuracy of the data. Second, a robust validation and traceability system ensures the quality of the process. Finally, rigorous security protocols protect sensitive patient information.
The future of medical data labeling will depend on the ability to maintain these high standards while adapting to technological developments. We believe that companies that invest in processes that meet FDA requirements today are well positioned for tomorrow's medical innovations...
