Working… Menu

Validation of an Artificial Intelligence-based Algorithm for Skeletal Age Assessment

The safety and scientific validity of this study is the responsibility of the study sponsor and investigators. Listing a study does not mean it has been evaluated by the U.S. Federal Government. Read our disclaimer for details. Identifier: NCT03530098
Recruitment Status : Active, not recruiting
First Posted : May 21, 2018
Last Update Posted : October 11, 2019
Information provided by (Responsible Party):
Safwan S Halabi, MD, Stanford University

Brief Summary:
The purpose of this study is to understand the effects of using a Artificial Intelligence algorithm for skeletal age estimation as a computer-aided diagnosis (CADx) system. In this prospective real-time study, the investigators will send de-identified hand radiographs to the Artificial Intelligence algorithm and surface the output of this algorithm to the radiologist, who will incorporate this information with their normal workflows to make a diagnosis of the patient's bone age. All radiologists involved in the study will be trained to recognize the surfaced prediction to be the output of the Artificial Intelligence algorithm. The radiologists' diagnosis will be final and considered independent to the output of the algorithm.

Condition or disease Intervention/treatment Phase
Bone Age Device: BoneAgeModel Not Applicable

Detailed Description:

The investigators are targeting to study the effect of their Artificial Intelligence algorithm on the radiologists' estimation of skeletal age. Currently, radiologists make the estimation using only the patients' radiographic images and health records. As part of this study, the radiologists will make diagnoses about their patients using the patients' radiographic images, health records, and the output of the CADx algorithm. The investigators wish to understand how radiologists using the Artificial Intelligence algorithm compare to radiologists who do not for the specific task of estimating skeletal age.

This study is organized as a multi-institutional randomized control trial with two arms - experiment (receiving the Artificial Intelligence algorithm's output) and control (no intervention). Both of these arms will be compared to a clinical reference standard ("gold standard") composed of a panel of radiologists. The metric of comparison will be Mean Absolute Distance (MAD) and Root-Mean-Square (RMS) error. The investigators plan to use statistical tests such as the t-test to determine any statistically-significant difference in skeletal age estimation between the two groups.

The investigators hope to recruit and analyze data from a sample size of 1000 patients.The patients will not undergo any research procedures that deviate from the current standard practices.

Layout table for study information
Study Type : Interventional  (Clinical Trial)
Estimated Enrollment : 1200 participants
Allocation: Randomized
Intervention Model: Parallel Assignment
Intervention Model Description: A hand radiograph will be randomly assigned to one of two groups - control and experiment. In the control group, participating radiologists will diagnose the case using the current standard of care (no intervention). In the experiment group, the radiologists will factor in the output of the Artificial Intelligence algorithm in their diagnosis. In all cases, the decision of the radiologist will be considered final.
Masking: None (Open Label)
Primary Purpose: Diagnostic
Official Title: Prospective Clinical Validation of an Artificial Intelligence-based Algorithm for Skeletal Age Assessment
Actual Study Start Date : July 12, 2018
Estimated Primary Completion Date : October 2019
Estimated Study Completion Date : November 2019

Arm Intervention/treatment
No Intervention: Control
This is the control arm where no intervention is provided; represents current standard of care.
Experimental: Experiment
This is the experiment arm where the intervention, "BoneAgeModel", is provided. The participating radiologists in this arm will receive the output of the Artificial Intelligence algorithm. They will be asked to incorporate this new information with their normal workflows to make a diagnosis. The radiologists' diagnosis will be considered final.
Device: BoneAgeModel
BoneAgeModel is an Artificial Intelligence tool that takes in a pediatric patient's hand radiograph and gender, and outputs their skeletal (bone) age. The intervention involves using this tool as a factor in the clinical decision making process of the participating radiologists. The radiologist's decision will be considered final.

Primary Outcome Measures :
  1. Paired Difference of Skeletal Age Estimate [ Time Frame: Through study completion, an average of 8 months ]
    The skeletal age estimates from radiologists in the experiment arm will compared to that from the radiologists in the control group. The metrics used for comparison will be Mean Absolute Difference (MAD) and Root Mean Squared (RMS), both of which will be computed against a clinical reference standard composed of expert radiologists. Intuitively, this outcome compares the "accuracy" of the radiologists when they are using the BoneAgeModel to when they are not.

Secondary Outcome Measures :
  1. Time Saved [ Time Frame: Through study completion, an average of 8 months ]
    Amount of time taken by radiologists when using the BoneAgeModel as compared to when they are not.

Information from the National Library of Medicine

Choosing to participate in a study is an important personal decision. Talk with your doctor and family members or friends about deciding to join a study. To learn more about this study, you or your doctor may contact the study research staff using the contacts provided below. For general information, Learn About Clinical Studies.

Layout table for eligibility information
Ages Eligible for Study:   1 Year to 18 Years   (Child, Adult)
Sexes Eligible for Study:   All
Accepts Healthy Volunteers:   No

Inclusion Criteria:

  • Patients between the age of 1 years and 18 years
  • Patients referred to get a hand radiograph taken for skeletal age assessment

Exclusion Criteria:

  • Patients with age less than 1 year or greater than 18 years
  • Patients not referred to get a hand radiograph taken for skeletal age assessment

Information from the National Library of Medicine

To learn more about this study, you or your doctor may contact the study research staff using the contact information provided by the sponsor.

Please refer to this study by its identifier (NCT number): NCT03530098

Layout table for location information
United States, California
Stanford University
Stanford, California, United States, 94305
United States, Connecticut
Yale New Haven Hospital
New Haven, Connecticut, United States, 06519
United States, District of Columbia
MedStar Georgetown University Hospital
Washington, District of Columbia, United States, 20007
United States, Georgia
Emory University
Atlanta, Georgia, United States, 30322
United States, Kansas
University of Kansas Medical Center
Kansas City, Kansas, United States, 66160
United States, Massachusetts
Boston Children's Hospital
Boston, Massachusetts, United States, 02115
United States, New York
New York University
New York, New York, United States, 10016
United States, Ohio
Cincinnati Children's Hospital Medical Center
Cincinnati, Ohio, United States, 45229
United States, Pennsylvania
Children's Hospital of Philadelphia
Philadelphia, Pennsylvania, United States, 19104
Sponsors and Collaborators
Stanford University
Layout table for investigator information
Study Chair: Curtis Langlotz, M.D. Ph.D. Stanford University
Study Director: David Eng, B.S. Stanford University
Study Director: Nishith Khandwala, B.S. Stanford University
Principal Investigator: Safwan Halabi, M.D. Stanford University

Layout table for additonal information
Responsible Party: Safwan S Halabi, MD, Principal Investigator, Stanford University Identifier: NCT03530098     History of Changes
Other Study ID Numbers: IRB #44764
First Posted: May 21, 2018    Key Record Dates
Last Update Posted: October 11, 2019
Last Verified: October 2019
Individual Participant Data (IPD) Sharing Statement:
Plan to Share IPD: Yes
Plan Description: Individual participant data that underlie the results reported in this article after deidentification (text, tables, figures and appendices).
Supporting Materials: Study Protocol
Statistical Analysis Plan (SAP)
Analytic Code
Time Frame: Beginning 3 months and ending 5 years following article publication.
Access Criteria: Researchers who provide a methodologically sound proposal.

Layout table for additional information
Studies a U.S. FDA-regulated Drug Product: No
Studies a U.S. FDA-regulated Device Product: Yes
Device Product Not Approved or Cleared by U.S. FDA: Yes
Pediatric Postmarket Surveillance of a Device Product: No
Product Manufactured in and Exported from the U.S.: Yes
Keywords provided by Safwan S Halabi, MD, Stanford University:
Age Determination by Skeleton
Machine Learning
Deep Learning
Artificial Intelligence
Clinical Validation