We use a set of 100 patients with sequential LDCT (total of 200 scans), including equal number of cancer and non-cancer cases. Additional information will be provided to contestants that includes nodule location and time interval between the sequential scans.

Training (Calibration) and Testing Data

We plan to use small training set or calibration data set and larger test data sets (i.e., 30 patients for Train and larger for Test cohort), and the cohorts will have equal number of cancer and non-cancer cases. The subjects LDCT scans are provided with one identified nodule location information, as a screen capture and slice location. The training dataset includes screening time interval. Participants are allowed to use outside training data, but asked to disclose the source with details including number of samples as supplemental information to their submission. Training data will be available on Jan 8, 2018. Participants will have eight weeks to train their algorithms and two weeks to apply them to the test data.

Quantitative Descriptors

No restrictions on the type of descriptors. But we will limit entries to quantitative metrics (computer generated) with minimal manual input like initialization of algorithms (semi-manual). 

Some examples include:

• Size or volume metrics
• Change in size or volume
• Radiomic metrics (or delta radiomics)
• Transfer Learning
• Deep Learning methods

Details on the methodology need to appear in the training report (see below) along with an abstract, due on March 15, 2018.

Data Details

The challenge will provide total of 200 LDCT scans for 100 subjects.

• Training dataset will contain 60 LDCT scans (30 subjects) that include scans from two sequential time intervals. Pathological diagnosis (cancer or non cancer status) will be provided.
  
  
• Test dataset will contain 140 LDCT scans (70 subjects) that include scans from two sequential time intervals. Final pathological diagnosis will be withheld.

Model Description Documentation

Participants are required to submit abstract and analysis description document that contain details on model description, processing details, including any pre/post processing, inference methodology proposed for test data set. Details on use of any supplemental training data other than challenge training set must be included. The report should contain all pertinent details, up to 3 pages in length.

The deadline for abstract submission has been extended to March 23.

You may your abstract either via the submission method detailed in "How to Submit" below, or you can upload a file to this abstract submission link: https://moffitt.sharefile.com/r-rdd6492d8b5643d3a.

Nodule Measurements and Malignancy Probability:

The participants are expected to provide nodules size, volume measurements (for identified nodule) along with probability of malignancy. This information will be required for both train and test data. The CSV or excel formatted file should contain following information:

The NLST study provides approximate nodule location for malignant cases. We have identified benign nodules for non-cancer cases and mapped the nodules across the screening time interval with the help of an expert radiologist.

Evaluation Metrics

We will use the participant probability of malignancy score (0 to 1) at diagnostic time to compute the receiver operator characteristics using the nodules cancer diagnosis. The area under the curve (AUC) of the Receiver Operating Characteristic (ROC) curve will be used to evaluate participant’s methods ability to predict malignancy. The nodules cancer status was histologically verified by the NLST study team.

How to Submit

You can submit results using the 'Participate' tab. You will need to compress your abstract and analysis descirption document (.pdf) and submission file (.csv) into a .zip file before you can submit it via the Codalab interface. Note that your submission file must be titled "submission.csv" to process properly in the evaluation system. You are not required to include your abstract in every submission during the Training phase, and we will accept your most recently submitted abstract as your final submission.

To test our system, you can download a reference .zip file at this link for the Training dataset: http://isbichallenges.cloudapp.net/my/datasets/download/5768025f-f6bf-4b85-a511-fae73b3c41e6. This reference file should receive an AUC of 0.773 in the Training phase.

At the conclusion of the challenge, organizers will develop an overview manuscript, describing the challenge and its results, for submission to a peer-reviewed journal, such as IEEE Transactions in Medical Imaging, Medical Image Analysis, or Journal of Medical Imaging. Select members of the top-ranked teams will be invited to participate as co-authors in the overview manuscript. Rest of the teams who wish the results to be part of the manuscripts supplemental section will be acknowledged.

The overall challenge timeline will be as follows:

Training Phase
Jan 8 – March 1

Abstract (Training model report - see Submission details tab)
EXTENDED: March 23
 
Test Phase
NEW: March 12 – March 21 

Announcement of results
March 26

Workshop
April 4, ISBI Conference
Winners and other participants will briefly present results.
http://biomedicalimaging.org/2018/

Organizing Team

• Yoganand Balagurunathan, PhD, H.L.Moffitt Cancer Center, Tampa, FL
• Keyvan Farahani, PhD, National Cancer Institute, Bethesda, MD
• Dmitry Goldgof, PhD, University of South Florida, Tampa, FL
• Lubomir Hadjiyski, PhD, University of Michigan, Ann Arbor, MI
• Jayashree Kalpathy-Cramer, PhD, Massachusetts General Hospital, Cambridge, MA
• Michael McNitt-Gray, PhD, University of California, Los Angeles, CA
• Sandy Napel, PhD, Stanford University, Palo Alto, CA

Data Quality

• Ying Liu, MD, Tianjin Medical University Cancer Institute and Hospital
• Qian Li, MD, Tianjin Medical University Cancer Institute and Hospital
• Matthew Schabath, PhD, H.L.Moffitt Cancer Center (MCC), Tampa, FL
• Paul Pinsky, PhD, National Cancer Institute, Bethesda, MD

Dear ISBI Challenge Participants:Thank you all so much for your patience as we wrap up the ISBI Lung Challenge. With close to 107 participants from all over the world, this challenge exceeded all expectations. Despite some bumps along the road, we are grateful to all of the organizers and participants for their patience and engagement.

CONGRATULATIONS TO THE ISBI CHALLENGE PARTICIPANTS!

Sincerely,

ISBI Lung Challenge Organizers

Contacts

• Keyvan Farahani (Farahani@nih.gov)
• Yoganand Balagurunathan (yoganand.balagurunathan@moffitt.org)

1. Acceptable entries to the challenge competitions must be algorithms that are based on a single nodule for each subject.  

2. Participants are allowed up to three submissions for the test results. At conclusion of the test phase we will take the best submission from each team as their official entry.

3. Only the area under the curve will used in ranking of entries for the challenge. However, we ask participants to also provide measurements of size and volume for identified nodules. These parameters will be used in producing the final report for the lung malignancy prediction challenge, but will not affect the rankings.

LIMITATION ON CHALLENGE DATA USAGE: Due to NCI/NLST data usage restrictions, participants would mandatorily agree to comply with the following:

(a) The challenge data and segmentations cannot be shared in part or in full through any public domain or redistributed in any other form.

(b)  The challenge data (patient images, segmentations and status flag) must to be deleted after the end of the challenge

(c) The challenge data (Raw images, segmentation) cannot be distributed in a publication, however results of the findings can be disseminated for scholarly purpose, d) challenge data (images, segmentation) cannot be used for commercial purpose.