ISBI 2018 - Lung Nodule Malignancy Prediction, Based on Sequential CT Scans

Organized by artem - Current server time: May 27, 2018, 12:52 a.m. UTC

First phase

Jan. 8, 2018, 5 p.m. UTC


Competition Ends
March 22, 2018, 6:59 a.m. UTC

ISBI 2018

Lung Nodule Malignancy Prediction, Based on Sequential CT Scans

Challenge Description

This challenge intends to advance methods development on the current clinical impediment to assess nodules status for lung cancer screening subjects with consecutive scans.

We invite ISBI 2018 participants to develop algorithms or re-package computational methods with potential clinical utility to identify malignancy. We will provide sequential low-dose CT (LDCT) scans at two screening intervals from the National Lung Screening Trial (NLST), with matched identified nodules from the same subject. We would like the participating teams to provide estimated nodules dimensions (longest diameter, volume) in the screening interval and the probability of malignancy. The teams are open to use any radiomic descriptors for nodules across time points and or change in size measurements including doubling time (DT) toward their assessment. If teams prefer to use doubling time (DT) metric (measured in days), following formulation is stated for reference.

DT = (t2 - t1) * (log 2 /(log(V2) - log(V1)))

Where V1, and V2 are the nodules volume (or size) measured at two screening intervals; in our study, t1 and t2 are baseline and diagnostic scan time respectively. Participants may use any other preferred formulation, any variant formulation, need to be described with reasoning in their respective training summary report.

Specifically, participants are asked to submit files that include nodule size (longest diameter), volume, and a probability of malignancy score (range from 0 to 1, for absence or presence of cancer, respectively) for both the train and blinded test cases. The participants will be evaluated on the test data performance.

See more details about the ISBI 2018 conference at

We use a set of 100 patients with sequential LDCT (total of 200 scans), including equal number of cancer and non-cancer cases. Additional information will be provided to contestants that includes nodule location and time interval between the sequential scans.

Training (Calibration) and Testing Data

We plan to use small training set or calibration data set and larger test data sets (i.e., 30 patients for Train and larger for Test cohort), and the cohorts will have equal number of cancer and non-cancer cases. The subjects LDCT scans are provided with one identified nodule location information, as a screen capture and slice location. The training dataset includes screening time interval. Participants are allowed to use outside training data, but asked to disclose the source with details including number of samples as supplemental information to their submission. Training data will be available on Jan 8, 2018. Participants will have eight weeks to train their algorithms and two weeks to apply them to the test data.

Quantitative Descriptors

No restrictions on the type of descriptors. But we will limit entries to quantitative metrics (computer generated) with minimal manual input like initialization of algorithms (semi-manual). 

Some examples include:

  • Size or volume metrics
  • Change in size or volume
  • Radiomic metrics (or delta radiomics)
  • Transfer Learning
  • Deep Learning methods

Details on the methodology need to appear in the training report (see below) along with an abstract, due on March 15, 2018.

Data Details

The challenge will provide total of 200 LDCT scans for 100 subjects.

  • Training dataset will contain 60 LDCT scans (30 subjects) that include scans from two sequential time intervals. Pathological diagnosis (cancer or non cancer status) will be provided.

  • Test dataset will contain 140 LDCT scans (70 subjects) that include scans from two sequential time intervals. Final pathological diagnosis will be withheld.

Model Description Documentation

Participants are required to submit abstract and analysis description document that contain details on model description, processing details, including any pre/post processing, inference methodology proposed for test data set. Details on use of any supplemental training data other than challenge training set must be included. The report should contain all pertinent details, up to 3 pages in length.

The deadline for abstract submission has been extended to March 23.

You may your abstract either via the submission method detailed in "How to Submit" below, or you can upload a file to this abstract submission link:

Nodule Measurements and Malignancy Probability:

The participants are expected to provide nodules size, volume measurements (for identified nodule) along with probability of malignancy. This information will be required for both train and test data. The CSV or excel formatted file should contain following information:

1 Subject-ID
2 Cohort Label (Indicate: Train or Test)
3 Nodule Size at time T1
4 Nodule Size at time T2
5 Nodule volume at time T1
6 Nodule volume at time T2
7 Malignancy Probability at Train or Test (0 to 1).
8 Descriptor-Type-Used (text field, limit to 3 words, examples are: Radiomics/delta radiomics/Deep learning/others)
9 Optional: DICOM UIDs at T1
10 Optional: DICOM UIDs at T2
11 Optional: Comments (if any)

The NLST study provides approximate nodule location for malignant cases. We have identified benign nodules for non-cancer cases and mapped the nodules across the screening time interval with the help of an expert radiologist.

Evaluation Metrics

We will use the participant probability of malignancy score (0 to 1) at diagnostic time to compute the receiver operator characteristics using the nodules cancer diagnosis. The area under the curve (AUC) of the Receiver Operating Characteristic (ROC) curve will be used to evaluate participant’s methods ability to predict malignancy. The nodules cancer status was histologically verified by the NLST study team.

How to Submit

You can submit results using the 'Participate' tab. You will need to compress your abstract and analysis descirption document (.pdf) and submission file (.csv) into a .zip file before you can submit it via the Codalab interface. Note that your submission file must be titled "submission.csv" to process properly in the evaluation system. You are not required to include your abstract in every submission during the Training phase, and we will accept your most recently submitted abstract as your final submission.

To test our system, you can download a reference .zip file at this link for the Training dataset: This reference file should receive an AUC of 0.773 in the Training phase.

At conclusion of the challenge organizers will develop an overview manuscript, describing the challenge and its results, for submission to a peer-reviewed journal, such as IEEE Transactions in Medical Imaging, Medical Image Analysis, or Journal of Medical Imaging. The top three ranked teams will be invited to participate as co-authors in the overview manuscript. Rest of the teams who wish the results to be part of the manuscripts supplemental section will be acknowledged.

The overall challenge timeline will be as follows:

Training Phase
Jan 8 – March 1

Abstract (Training model report - see Submission details tab)
EXTENDED: March 23
Test Phase
NEW: March 12 – March 21 

Announcement of results
March 26

April 4, ISBI Conference
Winners and other participants will briefly present results.

Organizing Team

  • Yoganand Balagurunathan, PhD, H.L.Moffitt Cancer Center, Tampa, FL
  • Keyvan Farahani, PhD, National Cancer Institute, Bethesda, MD
  • Dmitry Goldgof, PhD, University of South Florida, Tampa, FL
  • Lubomir Hadjiyski, PhD, University of Michigan, Ann Arbor, MI
  • Jayashree Kalpathy-Cramer, PhD, Massachusetts General Hospital, Cambridge, MA
  • Michael McNitt-Gray, PhD, University of California, Los Angeles, CA
  • Sandy Napel, PhD, Stanford University, Palo Alto, CA

Data Quality

  • Ying Liu, MD, Tianjin Medical University Cancer Institute and Hospital
  • Qian Li, MD, Tianjin Medical University Cancer Institute and Hospital
  • Matthew Schabath, PhD, H.L.Moffitt Cancer Center (MCC), Tampa, FL
  • Paul Pinsky, PhD, National Cancer Institute, Bethesda, MD

Dear ISBI Challenge Participants:Thank you all so much for your patience as we wrap up the ISBI Lung Challenge. With close to 107 participants from all over the world, this challenge exceeded all expectations. Despite some bumps along the road, we are grateful to all of the organizers and participants for their patience and engagement.


User AUC
gaperezs  0.913
mehrtash 0.897
qmia2018 0.879
liu3xing3long 0.873
Ricard 0.868
MIL 0.867
liangzhao 0.857
xuefeng 0.855
adityabagari 0.854
benVZ 0.848
huzq 0.825
WiemSafta 0.809
superman 0.797
JosephEnguehard 0.789
pranjalsahu 0.742
SilvanaC 0.698
uthoff 0.251


ISBI Lung Challenge Organizers


1. Acceptable entries to the challenge competitions must be algorithms that are based on a single nodule for each subject.  

2. Participants are allowed up to three submissions for the test results. At conclusion of the test phase we will take the best submission from each team as their official entry.

3. Only the area under the curve will used in ranking of entries for the challenge. However, we ask participants to also provide measurements of size and volume for identified nodules. These parameters will be used in producing the final report for the lung malignancy prediction challenge, but will not affect the rankings.


LIMITATION ON CHALLENGE DATA USAGE: Due to NCI/NLST data usage restrictions, participants would mandatorily agree to comply with the following:

(a) The challenge data and segmentations cannot be shared in part or in full through any public domain or redistributed in any other form.

(b)  The challenge data (patient images, segmentations and status flag) must to be deleted after the end of the challenge

(c) The challenge data (Raw images, segmentation) cannot be distributed in a publication, however results of the findings can be disseminated for scholarly purpose, d) challenge data (images, segmentation) cannot be used for commercial purpose.



Start: Jan. 8, 2018, 5 p.m.

Description: For submission instructions, please see the "Submission Details" tab on the main page. Please submit a .zip file with your .csv file submission inside.


Start: March 12, 2018, midnight

Competition Ends

March 22, 2018, 6:59 a.m.

You must be logged in to participate in competitions.

Sign In