NMPA released the “Draft Guideline on Clinical Evaluation of AI-Assisted Diagnostic Medical Devices” on June 17, 2026. Feedbacks need to be submitted by July 10, 2026.
This draft document provides recommendations for clinical evaluation of AI-assisted diagnostic medical devices, specifically those classified as Class III (software or embedded software, code 21-04-02) that aid in characterizing lesions (e.g., benign/malignant status) in medical images. It targets products like those assisting in pulmonary nodule, thyroid nodule, breast nodule, or gastrointestinal polyp evaluation. The guidance is not mandatory but serves as a reference for applicants and reviewers, allowing for alternative scientifically justified approaches.
This guideline forms an integral part of China’s Digital Health regulatory framework and serves as a supplement to other software & AI related documents, such as those on medical device software, cybersecurity, AI-assisted software, and human factors engineering.
Please click HERE for our technical review on AI-aided Software Guideline. The article was published on BioWorld, a Hong Kong-based biotech magazine.
Click HERE for our webinar on “SaMD Registration Requirements: China & US Perspectives”
Click HERE for our webinar on “Key Takeaways and Best Practices of China Human Factor/Usability”
Scope and General Principles
The draft guideline applies to AI devices that provide diagnostic decision support (e.g., recommending intervention or providing tumor staging/phenotyping). It explicitly excludes: (1) AI for lesion detection (computer-aided detection); (2) devices predicting disease probability; (3) multi-class classifiers differentiating more than two lesion types; (4) triage/referral AI that flags suspected cases for further review; and (5) software paired with in vitro diagnostics.
Products requiring decision support typically necessitate clinical trials to validate core algorithm performance. Non-decision functions (e.g., measurement, segmentation, report generation) may be evaluated via secondary endpoints in the same trial or through other clinical evidence methods. If the core AI algorithm has already been clinically validated in another product by the same manufacturer, and is found to be substantially equivalent (e.g., same algorithm, independent function), a new trial may not be needed for that specific algorithm.
Clinical Trial Design Considerations
The primary objective is to evaluate the diagnostic accuracy of the AI-assisted device in its intended use setting, along with safety and usability. Controlled trials are generally recommended; designs may include parallel-group, crossover self-controlled, or Multi-Reader Multi-Case (MRMC) designs. The test group typically involves physicians using AI assistance, compared against a control group of physicians reading independently.
Study Subjects
– Image Samples: Retrospective collection from real-world clinical data is acceptable, with strict inclusion/exclusion criteria and consecutive sampling to avoid selection bias. For real-time imaging (e.g., ultrasound), prospective data collection is recommended. Samples must be independent of the product’s training and test sets. A diverse disease spectrum should be represented, and for retrospective data, demographic, disease-related (e.g., stage, size, location), and diagnostic confirmation information must be thoroughly collected.
– Readers: In MRMC designs, multiple readers of varying experience levels are included to account for reader variability. The number of readers must be justified.
Evaluation Metrics
Primary efficacy endpoints should be diagnostic accuracy measures robust to prevalence, such as sensitivity, specificity, and Area Under the Receiver Operating Characteristic Curve (AUC). Trials generally should be designed for superiority, e.g., superiority in AUC or superiority in sensitivity with non-inferiority in specificity. Secondary endpoints can include patient-level sensitivity, predictive values, Kappa agreement, diagnostic time, workflow optimization, and device usability/reliability.
Reference Standard
A robust clinical reference standard, demonstrating superior diagnostic performance to the test method, must be established and justified. Histopathology (preferably surgical pathology over biopsy) is the preferred gold standard, supplemented by imaging, clinical history, laboratory results, and long-term follow-up. For benign lesions where biopsy is unethical, a composite standard can be used: positive cases confirmed by biopsy, negative cases confirmed by biopsy or long-term imaging follow-up (e.g., ≥1 year for stable pulmonary nodules). The methods for establishing the reference standard must be detailed.
Sample Size and Statistical Analysis
Sample size calculations depend on the study design, chosen primary endpoint, and statistical assumptions (e.g., power, alpha level, expected effect size, non-inferiority margin). For MRMC designs, methods like Obuchowski-Rockette (OR) or Dorfman-Berbaum-Metz-Hillis (DBMH) analysis guide sample size determination, considering the number of readers and the estimated variance. The guideline provides example calculations for MRMC designs, emphasizing the need to justify all parameters. All enrolled subjects/readers should be included in the primary analysis, with point estimates and 95% confidence intervals provided for primary metrics.
Additional Trial Considerations
– Training: Pre-trial reader training (including case studies not part of the trial set) is essential to minimize bias and should mirror real-world application training.
– Blinding: Readers must be blinded to clinical data and reference standard results. Crossover designs may incorporate washout periods (typically 4-6 weeks) to mitigate memory effects.
Non-Decision Support Functions
Functions like segmentation, registration, 3D reconstruction, and measurement can be evaluated through secondary trial endpoints or other clinical evidence (e.g., algorithmic stress tests, database testing). Standard metrics for accuracy may include Dice coefficients for segmentation, target registration error for fusion, and subjective evaluation of reconstruction quality.
Labeling Requirements
The Instructions for Use must include a clinical trial summary, clear intended use (e.g., specific anatomy, imaging modality), and warnings. Warnings should state that the software is an aid only, does not replace pathological diagnosis or clinician judgment, and that there are risks of false positives/negatives. Applicable guidelines and standards used in development should be cited, along with a note that users must assess risks associated with guideline updates.
Key Clinical Evaluation Examples
The guideline provides two practical appendices illustrating specific design considerations for AI-assisted diagnostic products. These examples are not prescriptive but serve as reference models for similar devices.
Appendix 1 – CT Image Pulmonary Nodule AI-Assisted Diagnosis
For products aiding in the benign/malignant characterization of pulmonary nodules on CT, an MRMC (multi‑reader, multi‑case) or crossover self‑controlled design is recommended. In an MRMC design, readers are split into two groups and undergo two phases: with and without AI assistance, separated by a washout period. Each reader interprets all image samples in both conditions, ensuring blinding to clinical and pathological data.
Primary endpoints typically combine ROC‑derived metrics (e.g., FROC, LROC) with lesion‑level sensitivity and specificity. Secondary endpoints include patient‑level sensitivity, predictive values, Kappa agreement, reading time, and subjective usability measures (e.g., Likert scales). If non‑decision functions (e.g., segmentation, measurement) are evaluated, additional secondary metrics are defined.
Sample size estimation is illustrated using the MRMC framework. Assuming an effect size of 0.06 for AUC, 11 readers, a superiority margin of 0, and 80% power, at least 98 total cases (1:1 positive/negative) are required. For lesion‑level sensitivity (effect size 0.08, superiority margin 0), 128 positive nodules (approx. 64 patients with 2 nodules each) are needed. For patient‑level specificity (effect size –0.06, non‑inferiority margin –0.1), 176 negative cases are required. Combining these, the minimum sample is 240 cases; accounting for 10% dropout and 10% exclusion, the final target is 265 cases (71 positive, 194 negative).
Appendix 2 – Ultrasound Thyroid Nodule AI‑Assisted Diagnosis
For ultrasound‑based thyroid nodule characterization, a parallel‑group crossover design is proposed. Patients are randomized into two groups; each group undergoes reads with and without AI by the same physician, with a washout period (e.g., 4 weeks) between phases. Blinding is maintained.
The reference standard is generally surgical histopathology, especially for challenging cases where biopsy is indicated. Inclusion criteria may enrich for more diagnostically difficult nodules (e.g., size >5 mm with clinical suspicion, or >20 mm benign nodules with patient willingness for surgery), ensuring that the reference standard is uniformly pathological.
Primary endpoints may be ROC‑based or a composite of lesion‑level sensitivity and patient‑level specificity. Sample size calculations are detailed using superiority for sensitivity and non‑inferiority for specificity. For sensitivity, assuming a control rate of 78%, an AI‑assisted rate of 85%, superiority margin 0, α=0.05, β=0.2, the required malignant cases are 479. For specificity, assuming control and test rates both 84%, a non‑inferiority margin of –0.1, the required benign cases are 211. Accounting for 20% dropout, the total sample size becomes 863 cases.
Both appendices emphasize the importance of pre‑trial reader training using independent case sets, robust blinding, and clear justification of all statistical parameters. They also highlight the need to tailor the choice of endpoints and reference standards to the specific clinical context and disease prevalence. These illustrative examples provide a concrete framework that applicants can adapt, with appropriate justifications, to their own product’s intended use and technological characteristics.