We're building a better ClinicalTrials.gov. Check it out and tell us what you think!
Working…
ClinicalTrials.gov
ClinicalTrials.gov Menu

Artificial Intelligence and Radiologists at Prostate Cancer Detection in MRI: The PI-CAI Challenge

The safety and scientific validity of this study is the responsibility of the study sponsor and investigators. Listing a study does not mean it has been evaluated by the U.S. Federal Government. Read our disclaimer for details.
 
ClinicalTrials.gov Identifier: NCT05489341
Recruitment Status : Active, not recruiting
First Posted : August 5, 2022
Last Update Posted : August 5, 2022
Sponsor:
Collaborators:
Ziekenhuisgroep Twente
University Medical Center Groningen
Norwegian University of Science and Technology
Information provided by (Responsible Party):
Radboud University Medical Center

Brief Summary:
The PI-CAI challenge aims to validate the diagnostic performance of artificial intelligence (AI) and radiologists at clinically significant prostate cancer (csPCa) detection/diagnosis in MRI, with respect to histopathology and follow-up (≥ 3 years) as reference. The study hypothesizes that state-of-the-art AI algorithms, trained using thousands of patient exams, are non-inferior to radiologists reading bpMRI. As secondary end-points, it investigates the optimal AI model for csPCa detection/diagnosis, and the effects of dynamic contrast-enhanced imaging and reader experience on diagnostic accuracy and inter-reader variability.

Condition or disease Intervention/treatment
Prostate Cancer Diagnostic Test: Histopathology and Magnetic Resonance Imaging with Follow-Up Diagnostic Test: Histopathology and Magnetic Resonance Imaging

Detailed Description:

Prostate cancer (PCa) is one of the most prevalent cancers in men worldwide. One million men receive a diagnosis and 300,000 die from clinically significant PCa (csPCa) (defined as ISUP≥2), each year, worldwide. Multiparametric magnetic resonance imaging (mpMRI) is playing an increasingly important role in the early diagnosis of prostate cancer, and has been recommended by the European Association of Urology (EAU), prior to biopsies. However, current guidelines for reading prostate mpMRI (i.e. PI-RADS v2.1) follow a semi-quantitative assessment that mandates substantial expertise for proper usage. This can lead to low inter-reader agreement (<50%), sub-optimal interpretation and overdiagnosis.

Modern artificial intelligence (AI) algorithms have paved the way for powerful computer-aided detection and diagnosis (CAD) systems that rival human performance in medical image analysis. Clinical trials are the gold standard for assessing new medications and interventions in a controlled and comparative manner, and the equivalent for developing AI algorithms are international competitions or "grand challenges", where increasingly large datasets are released to public to solve clinically relevant tasks with AI. Grand challenges can address the lack of trust, scientific evidence and adequate validation among AI solutions, by providing the means to compare algorithms against each other using common datasets and a unified experimental setup.

PI-CAI (Prostate Imaging: Cancer AI) is an all-new grand challenge, with over 10,000 carefully-curated prostate MRI exams to validate modern AI algorithms and estimate radiologists' performance at csPCa detection and diagnosis. Key aspects of the study design have been established in conjunction with an international, multi-disciplinary scientific advisory board (16 experts in prostate AI, radiology and urology) -to unify and standardize present-day guidelines, and to ensure meaningful validation of prostate-AI towards clinical translation.

The 2022 edition of PI-CAI will focus on validating AI at automated 3D detection and diagnosis of csPCa in bpMRI. PI-CAI primarily consists of two sub-studies:

  • AI Study (Grand Challenge): An annotated multi-center, multi-vendor dataset of 1500 bpMRI exams (including their clinical and acquisition variables) is made publicly available for all participating teams and the research community at large. Teams can use this dataset to develop AI models, and submit their trained algorithms (in Docker containers) for evaluation. At the end of this open development phase, all algorithms are ranked, based on their performance on a hidden testing cohort of 1000 unseen scans. In the closed testing phase, organizers retrain the top-ranking 5 AI algorithms using a larger dataset of 7500-9500 bpMRI scans (including additional training scans from a private dataset). Finally, their performance is re-evaluated on the hidden testing cohort (with rigorous statistical analyses), to determine the top 3 AI algorithms for automated 3D detection and diagnosis of csPCa in bpMRI (i.e. the winners of the grand challenge).
  • Reader Study: 50+ international prostate radiologists perform a reader study using a subset of 400 scans from the hidden testing cohort. For each case, radiologists complete their assessments in two rounds. At first, using clinical and acquisition variables plus bpMRI sequences only, enabling head-to-head comparisons against AI trained on the same. And then, using clinical and acquisition variables plus full mpMRI sequences, enabling comparisons between AI and current clinical practice (PI-RADS v2.1). Overall, the goal of this study is to estimate the performance of the average radiologist at detection and diagnosis of csPCa in MRI.

In the end, PI-CAI aims to benchmark state-of-the-art AI algorithms developed in the grand challenge, against prostate radiologists participating in the reader study -to evaluate the clinical viability of modern prostate-AI solutions at csPCa detection and diagnosis in MRI.

Layout table for study information
Study Type : Observational
Estimated Enrollment : 5000 participants
Observational Model: Cohort
Time Perspective: Retrospective
Official Title: Artificial Intelligence and Radiologists at Prostate Cancer Detection in MRI: The PI-CAI Challenge
Actual Study Start Date : February 1, 2022
Estimated Primary Completion Date : February 2023
Estimated Study Completion Date : May 2023

Resource links provided by the National Library of Medicine

MedlinePlus related topics: Prostate Cancer

Group/Cohort Intervention/treatment
Public Training and Development Set (1500 cases)
Available for all participants and researchers, to train and develop AI models. Includes multi-vendor (Siemens Healthineers, Philips Medical Systems) prostate bpMRI cases from three Dutch centers (Radboud University Medical Center, Ziekenhuisgroep Twente, University Medical Center Groningen), acquired between 2012-2021. All data is fully anonymized and made available under a non-commercial CC BY-NC 4.0 license. Includes 328 cases from the PROSTATEx challenge (prostatex.grand-challenge.org). Imaging data has been released via: zenodo.org/record/6624726 (DOI: 10.5281/zenodo.6624726). Lesion annotations of csPCa have been released and are maintained via: github.com/DIAGNijmegen/picai_labels.
Diagnostic Test: Histopathology and Magnetic Resonance Imaging
Reference standard establishes histologically-confirmed (ISUP ≥ 2) cases of csPCa as positives, and histopathology- (ISUP ≤ 1) or MRI- (PI-RADS ≤ 2) confirmed cases of indolent PCa or benign tissue as negatives.

Private Training Set (7500-9500 cases)
Used exclusively by the organizers to retrain the top-ranking 5 AI algorithms, with large-scale data. Includes multi-vendor (Siemens Healthineers, Philips Medical Systems) prostate bpMRI cases from three Dutch centers (Radboud University Medical Center, Ziekenhuisgroep Twente, University Medical Center Groningen), acquired between 2012-2021.
Diagnostic Test: Histopathology and Magnetic Resonance Imaging
Reference standard establishes histologically-confirmed (ISUP ≥ 2) cases of csPCa as positives, and histopathology- (ISUP ≤ 1) or MRI- (PI-RADS ≤ 2) confirmed cases of indolent PCa or benign tissue as negatives.

Hidden Validation and Tuning Cohort (100 cases)
Used for a live, public leaderboard that enables AI model selection and tuning throughout the open development phase of the challenge. Includes multi-vendor (Siemens Healthineers, Philips Medical Systems) prostate bpMRI cases from three Dutch centers (Radboud University Medical Center, Ziekenhuisgroep Twente, University Medical Center Groningen), acquired between 2012-2021, that remain fully hidden throughout the course of the challenge.
Diagnostic Test: Histopathology and Magnetic Resonance Imaging with Follow-Up
Reference standard establishes histologically-confirmed (ISUP ≥ 2) cases of csPCa as positives, and histopathology- (ISUP ≤ 1) or MRI- (PI-RADS ≤ 2) with follow-up (≥ 3 years) confirmed cases of indolent PCa or benign tissue as negatives.

Hidden Testing Cohort (1000 cases)
Used to benchmark AI, radiologists, and test all hypotheses at the end of the PI-CAI challenge. A subset of 400 cases from this cohort is used to facilitate the PI-CAI: Reader Study. Includes multi-vendor (Siemens Healthineers, Philips Medical Systems) internal testing data (unseen prostate bpMRI cases from three seen Dutch centers {Radboud University Medical Center, Ziekenhuisgroep Twente, University Medical Center Groningen}) and external testing data (unseen prostate bpMRI cases from one unseen Norwegian center {Norwegian University of Science and Technology}), acquired between 2012-2021.
Diagnostic Test: Histopathology and Magnetic Resonance Imaging with Follow-Up
Reference standard establishes histologically-confirmed (ISUP ≥ 2) cases of csPCa as positives, and histopathology- (ISUP ≤ 1) or MRI- (PI-RADS ≤ 2) with follow-up (≥ 3 years) confirmed cases of indolent PCa or benign tissue as negatives.




Primary Outcome Measures :
  1. AI vs Radiologists from Reader Study [ Time Frame: 6 months ]
    Diagnostic performance of the top 5 AI models from the grand challenge and 50+ radiologists from the reader study, at csPCa detection/diagnosis in prostate bpMRI, with respect to histopathology and MRI with follow-up (≥ 3 years) as reference, to assess the clinical viability of present-day AI solutions.

  2. AI vs Radiologists from Clinical Routine [ Time Frame: 6 months ]
    Diagnostic performance of the top 5 AI models from the grand challenge and the historical reads of radiologists from clinical routine, at csPCa detection/diagnosis in prostate bpMRI, with respect to histopathology and MRI with follow-up (≥ 3 years) as reference, to assess the clinical viability of present-day AI solutions.


Secondary Outcome Measures :
  1. AI vs AI [ Time Frame: 6 months ]
    Diagnostic performance of the top 5 AI models from the grand challenge, at csPCa detection/diagnosis in prostate bpMRI, with respect to histopathology and MRI with follow-up (≥ 3 years) as reference, to deduce the optimal AI model architecture for this given task.

  2. Radiologists vs Radiologists from Reader Study [ Time Frame: 6 months ]
    Diagnostic performance and inter-reader variability of 50+ radiologists from the reader study, at csPCa detection/diagnosis in prostate mpMRI, with respect to histopathology and MRI with follow-up (≥ 3 years) as reference, to deduce the effects of dynamic contrast-enhanced imaging and reader experience.


Biospecimen Retention:   Samples With DNA
Biospecimen is not stored beyond the timeframe of this study, for the purpose of this study. However, biospecimen is stored beyond the timeframe of this study, for the purpose of regular clinical care. In this case, biospecimen refers to histopathology tissue acquired from confirmatory prostate biopsies and prostatectomies. Within the scope of clinical routine, storing such specimen can facilitate reassessments through the future, e.g. for comparisons if the patient presents new findings or metastasis of their initial findings, for comparisons against histopathology findings of the original patient's offspring, or even for legal purposes in the case of misdiagnosis.


Information from the National Library of Medicine

Choosing to participate in a study is an important personal decision. Talk with your doctor and family members or friends about deciding to join a study. To learn more about this study, you or your doctor may contact the study research staff using the contacts provided below. For general information, Learn About Clinical Studies.


Layout table for eligibility information
Ages Eligible for Study:   18 Years and older   (Adult, Older Adult)
Sexes Eligible for Study:   Male
Accepts Healthy Volunteers:   Yes
Sampling Method:   Non-Probability Sample
Study Population
All patient exams are of men suspected of harboring csPCa, with elevated levels of prostate-specific antigen (≥ 3 ng/mL) and/or abnormal findings on digital rectal exam, and without a history of treatment or any prior positive histopathology (ISUP ≥ 2) findings. Patients underwent prostate MRI, and were primarily examined at one of three Dutch centers (Radboud University Medical Center, Ziekenhuisgroep Twente, University Medical Center Groningen) or one Norwegian center (Norwegian University of Science and Technology) during regular clinical routine, between 2012-2021.
Criteria

Inclusion Criteria:

  • Men suspected of harboring csPCa, with elevated levels of prostate-specific antigen (≥ 3 ng/mL) and/or abnormal findings on digital rectal exam, who subsequently underwent prostate MRI.

Exclusion Criteria:

  • Patients who opted-out or did not give permission to reuse clinical data.
  • Patients with a history of prior prostate treatment.
  • Patients with a history of prior positive csPCa findings in histopathology (ISUP ≥ 2).
  • Patients whose prostate MRI exhibit severe artifacts (e.g. heavy warping due to rectal air, metal artifacts from hip prostheses, heavy motion blur), thereby impeding their usage.
  • Patients, whose positive histopathology findings (ISUP ≥ 2) cannot be reliably localized on MRI (e.g. MRI-invisible lesions, systematic biopsy diagnostic reports with ambiguous, "random" or missing location information).

Information from the National Library of Medicine

To learn more about this study, you or your doctor may contact the study research staff using the contact information provided by the sponsor.

Please refer to this study by its ClinicalTrials.gov identifier (NCT number): NCT05489341


Locations
Layout table for location information
Netherlands
RadboudUMC
Nijmegen, Gelderland, Netherlands, 6525 GA
Sponsors and Collaborators
Radboud University Medical Center
Ziekenhuisgroep Twente
University Medical Center Groningen
Norwegian University of Science and Technology
Investigators
Layout table for investigator information
Principal Investigator: Henkjan Huisman, PhD Radboud University Medical Center
Additional Information:
Study Data/Documents: Individual Participant Data Set  This link exits the ClinicalTrials.gov site
Identifier: 10.5281/zenodo.6624726
Imaging for the PI-CAI: Public Training and Development Dataset, containing 1500 fully-anonymized prostate bpMRI scans from 1476 patients, acquired between 2012-2021, at three Dutch centers (Radboud University Medical Center, University Medical Center Groningen, Ziekenhuisgroep Twente).
Study Protocol  This link exits the ClinicalTrials.gov site
Identifier: 10.5281/zenodo.6667655
Preregistration of the PI-CAI challenge study design, in compliance with BIAS reporting guidelines (https://www.equator-network.org/reporting-guidelines/bias-transparent-reporting-of-biomedical-image-analysis-challenges/).
Analytic Code  This link exits the ClinicalTrials.gov site
Source code for preprocessing prostate MRI data archives.
Analytic Code  This link exits the ClinicalTrials.gov site
Source code for training baseline diagnostic AI models.
Analytic Code  This link exits the ClinicalTrials.gov site
Source code for evaluating csPCa detection and diagnosis performance, and performing all statistical tests with respect to the same.
Individual Participant Data Set  This link exits the ClinicalTrials.gov site
Annotations for the PI-CAI: Public Training and Development Dataset, containing basic clinical and acquisition variables, csPCa annotations and outcomes for 1500 fully-anonymized prostate bpMRI exams from 1476 patients, acquired between 2012-2021, at three Dutch centers (Radboud University Medical Center, University Medical Center Groningen, Ziekenhuisgroep Twente).

Layout table for additonal information
Responsible Party: Radboud University Medical Center
ClinicalTrials.gov Identifier: NCT05489341    
Other Study ID Numbers: CMO2016-3045-Project-20011
First Posted: August 5, 2022    Key Record Dates
Last Update Posted: August 5, 2022
Last Verified: July 2022
Individual Participant Data (IPD) Sharing Statement:
Plan to Share IPD: Yes
Plan Description: To facilitate open and transparent science, our end-to-end study protocol and our source code for preprocessing prostate MRI data archives, training baseline diagnostic AI models, evaluating lesion detection/diagnosis performance, and implementing statistical tests for AI/radiologists vs AI/radiologists comparisons, have been publicly released. Furthermore, a fully-anonymized dataset of 1500 prostate bpMRI scans from the PI-CAI challenge, and their outcomes, have been released to promote further research.
Supporting Materials: Study Protocol
Statistical Analysis Plan (SAP)
Clinical Study Report (CSR)
Analytic Code
Time Frame: Individual Participant Data Set (PI-CAI: Public Training and Development Set), Study Protocol, Statistical Analysis Plan (SAP) and Analytic Code has been shared with all participants of the PI-CAI challenge and the research community at large, towards the start of the challenge (June 2022). Clinical Study Report (CSR) will be released in the form of multiple publications after the completion of the challenge (tentatively May 2023). All of the aforementioned IPD will remain publicly accessible perpetually.
Access Criteria: Please refer to the "References" section of this protocol.
URL: http://pi-cai.grand-challenge.org/

Layout table for additional information
Studies a U.S. FDA-regulated Drug Product: No
Studies a U.S. FDA-regulated Device Product: No
Keywords provided by Radboud University Medical Center:
Magnetic Resonance Imaging
Prostate Cancer
Artificial Intelligence
Computer-Aided Detection and Diagnosis
Additional relevant MeSH terms:
Layout table for MeSH terms
Prostatic Neoplasms
Genital Neoplasms, Male
Urogenital Neoplasms
Neoplasms by Site
Neoplasms
Prostatic Diseases