Deep Learning Model

Analysis of Deep Learning Model

Notebook:

• Deep learning model.ipynb

Data:

• training_data.pkl from ../data/
• feature_import.pkl from ../data/deep_learning/
• sclc_train.pkl from ../data/independent_test/
• sclc_call.pkl from ../data/independent_test/

Summary of Analysis:

Training data was read into the notebook. Validation data was removed from the training data and germline variants were relabeled as fail. Diseases were condensed by types. The features and labels were obtained from the training data and a random seed was set for reproducibility. The training data was split for cross-validation and the 2/3 of the data were used to train the model. The deep forest was implemented using the keras library as a feed-forward neural network with the input layer equaling the number of features, four hidden layers with 20-node hidden layers, and an output layer equaling the three outputs. The input and hidden layers used a hyperbolic tangent (tanh) activation function, the output layer used a softmax activation function. Categorical cross-entropy was used as a loss function and the Adam optimizer was used over 700 epochs with a batch size of 2,000. L2 regularization was used with a weight of 0.001. Accuracy profile, ROC curves, and reliability diagrams were created.

Feature importance for the deep learning model was calculated by training a model on the cross-validation dataset, independently shuffling each of the 71 features, and determining the change in average AUC, by comparing baseline and shuffled performance.

The hold-out test set was analyzed using the deep learning model. Accuracy profile, ROC curves, and reliability diagrams were created.

To determine how disease type played a role in feature importance, we collapsed the diseases types into 'solid' vs. 'blood' . The model was re-trained using these collapsed features and the accuracy profile, ROC curve, and reliability diagram were plotted for the internal validation and the hold-out test set.

To simulate performance on a de-novo test set where the reviewer is unknown, we removed the reviewer feature and re-performed the analysis. Accuracy profile, AUC curves and reliability diagrams were generated for predictions without the reviewer feature. This model was employed on the independent test set (orthogonal validation) and the SCLC cases (manual review validation). The accuracy profile, AUC curves and reliability diagrams were generated.

Finally, we performed an analysis to determine how much manual review was required to over come batch effects associated with this model. Specifically, increments of 5% of the new SCLC data was randomly selected and used to re-train the model. Performance was evaluated and plotted. Results from this analysis were stored in ../data/deep_learning/sclc_batch_effect_correction.pkl.