This challenge intends to advance methods development on the current clinical impediment to assess nodules status for lung cancer screening subjects with consecutive scans.
We invite ISBI 2018 participants to develop algorithms or re-package computational methods with potential clinical utility to identify malignancy. We will provide sequential low-dose CT (LDCT) scans at two screening intervals from the National Lung Screening Trial (NLST), with matched identified nodules from the same subject. We would like the participating teams to provide estimated nodules dimensions (longest diameter, volume) in the screening interval and the probability of malignancy. The teams are open to use any radiomic descriptors for nodules across time points and or change in size measurements including doubling time (DT) toward their assessment. If teams prefer to use doubling time (DT) metric (measured in days), following formulation is stated for reference.
DT = (t2 - t1) * (log 2 /(log(V2) - log(V1)))
Where V1, and V2 are the nodules volume (or size) measured at two screening intervals; in our study, t1 and t2 are baseline and diagnostic scan time respectively. Participants may use any other preferred formulation, any variant formulation, need to be described with reasoning in their respective training summary report.
Specifically, participants are asked to submit files that include nodule size (longest diameter), volume, and a probability of malignancy score (range from 0 to 1, for absence or presence of cancer, respectively) for both the train and blinded test cases. The participants will be evaluated on the test data performance.
See more details about the ISBI 2018 conference at http://biomedicalimaging.org/2018/
We use a set of 100 patients with sequential LDCT (total of 200 scans), including equal number of cancer and non-cancer cases. Additional information will be provided to contestants that includes nodule location and time interval between the sequential scans.
Training (Calibration) and Testing Data
We plan to use small training set or calibration data set and larger test data sets (i.e., 30 patients for Train and larger for Test cohort), and the cohorts will have equal number of cancer and non-cancer cases. The subjects LDCT scans are provided with one identified nodule location information, as a screen capture and slice location. The training dataset includes screening time interval. Participants are allowed to use outside training data, but asked to disclose the source with details including number of samples as supplemental information to their submission. Training data will be available on Jan 8, 2018. Participants will have eight weeks to train their algorithms and two weeks to apply them to the test data.
Quantitative Descriptors
No restrictions on the type of descriptors. But we will limit entries to quantitative metrics (computer generated) with minimal manual input like initialization of algorithms (semi-manual).
Some examples include:
Details on the methodology need to appear in the training report (see below) along with an abstract, due on March 15, 2018.
Data Details
The challenge will provide total of 200 LDCT scans for 100 subjects.
Model Description Documentation
Participants are required to submit abstract and analysis description document that contain details on model description, processing details, including any pre/post processing, inference methodology proposed for test data set. Details on use of any supplemental training data other than challenge training set must be included. The report should contain all pertinent details, up to 3 pages in length.
The deadline for abstract submission has been extended to March 23.
You may your abstract either via the submission method detailed in "How to Submit" below, or you can upload a file to this abstract submission link: https://moffitt.sharefile.com/r-rdd6492d8b5643d3a.
Nodule Measurements and Malignancy Probability:
The participants are expected to provide nodules size, volume measurements (for identified nodule) along with probability of malignancy. This information will be required for both train and test data. The CSV or excel formatted file should contain following information:
Column | Contents |
---|---|
1 | Subject-ID |
2 | Cohort Label (Indicate: Train or Test) |
3 | Nodule Size at time T1 |
4 | Nodule Size at time T2 |
5 | Nodule volume at time T1 |
6 | Nodule volume at time T2 |
7 | Malignancy Probability at Train or Test (0 to 1). |
8 | Descriptor-Type-Used (text field, limit to 3 words, examples are: Radiomics/delta radiomics/Deep learning/others) |
9 | Optional: DICOM UIDs at T1 |
10 | Optional: DICOM UIDs at T2 |
11 | Optional: Comments (if any) |
The NLST study provides approximate nodule location for malignant cases. We have identified benign nodules for non-cancer cases and mapped the nodules across the screening time interval with the help of an expert radiologist.
Evaluation Metrics
We will use the participant probability of malignancy score (0 to 1) at diagnostic time to compute the receiver operator characteristics using the nodules cancer diagnosis. The area under the curve (AUC) of the Receiver Operating Characteristic (ROC) curve will be used to evaluate participant’s methods ability to predict malignancy. The nodules cancer status was histologically verified by the NLST study team.
How to Submit
You can submit results using the 'Participate' tab. You will need to compress your abstract and analysis descirption document (.pdf) and submission file (.csv) into a .zip file before you can submit it via the Codalab interface. Note that your submission file must be titled "submission.csv" to process properly in the evaluation system. You are not required to include your abstract in every submission during the Training phase, and we will accept your most recently submitted abstract as your final submission.
To test our system, you can download a reference .zip file at this link for the Training dataset: http://isbichallenges.cloudapp.net/my/datasets/download/5768025f-f6bf-4b85-a511-fae73b3c41e6. This reference file should receive an AUC of 0.773 in the Training phase.
At the conclusion of the challenge, organizers will develop an overview manuscript, describing the challenge and its results, for submission to a peer-reviewed journal, such as IEEE Transactions in Medical Imaging, Medical Image Analysis, or Journal of Medical Imaging. Select members of the top-ranked teams will be invited to participate as co-authors in the overview manuscript. Rest of the teams who wish the results to be part of the manuscripts supplemental section will be acknowledged.
The overall challenge timeline will be as follows:
Training Phase
Jan 8 – March 1
Abstract (Training model report - see Submission details tab)
EXTENDED: March 23
Test Phase
NEW: March 12 – March 21
Announcement of results
March 26
Workshop
April 4, ISBI Conference
Winners and other participants will briefly present results.
http://biomedicalimaging.org/2018/
Dear ISBI Challenge Participants:Thank you all so much for your patience as we wrap up the ISBI Lung Challenge. With close to 107 participants from all over the world, this challenge exceeded all expectations. Despite some bumps along the road, we are grateful to all of the organizers and participants for their patience and engagement.
CONGRATULATIONS TO THE ISBI CHALLENGE PARTICIPANTS!
User | AUC |
gaperezs | 0.913 |
mehrtash | 0.897 |
qmia2018 | 0.879 |
liu3xing3long | 0.873 |
Ricard | 0.868 |
MIL | 0.867 |
liangzhao | 0.857 |
xuefeng | 0.855 |
adityabagari | 0.854 |
benVZ | 0.848 |
huzq | 0.825 |
WiemSafta | 0.809 |
superman | 0.797 |
JosephEnguehard | 0.789 |
pranjalsahu | 0.742 |
SilvanaC | 0.698 |
uthoff | 0.251 |
Sincerely,
ISBI Lung Challenge Organizers
1. Acceptable entries to the challenge competitions must be algorithms that are based on a single nodule for each subject.
2. Participants are allowed up to three submissions for the test results. At conclusion of the test phase we will take the best submission from each team as their official entry.
3. Only the area under the curve will used in ranking of entries for the challenge. However, we ask participants to also provide measurements of size and volume for identified nodules. These parameters will be used in producing the final report for the lung malignancy prediction challenge, but will not affect the rankings.
LIMITATION ON CHALLENGE DATA USAGE: Due to NCI/NLST data usage restrictions, participants would mandatorily agree to comply with the following:
(a) The challenge data and segmentations cannot be shared in part or in full through any public domain or redistributed in any other form.
(b) The challenge data (patient images, segmentations and status flag) must to be deleted after the end of the challenge
(c) The challenge data (Raw images, segmentation) cannot be distributed in a publication, however results of the findings can be disseminated for scholarly purpose, d) challenge data (images, segmentation) cannot be used for commercial purpose.
Start: Jan. 8, 2018, 5 p.m.
Description: For submission instructions, please see the "Submission Details" tab on the main page. Please submit a .zip file with your .csv file submission inside.
Start: March 12, 2018, midnight
Start: March 13, 2018, 1:32 p.m.
March 22, 2018, 6:59 a.m.
You must be logged in to participate in competitions.
Sign In