In order to develop the Clinical Trial Risk Tool, we had to conduct a quality control exercise on the components. Each parameter which is fed into the complexity model is trained and evaluated independently. For an overview of how the tool works, please read this blog post.
I used two datasets to train and evaluate the tool:
- Manual dataset – this was a set of between 100 and 300 protocols which I read through individually and annotated key parameters such as the sample size. The number annotated per parameter varied between 100 and 300.
- ClinicalTrials.gov dataset – this was a much larger dataset of 11925 protocols downloaded from ClinicalTrials.gov. These came together with NCT ID, phase, pathology, SAP, number of arms and number of subjects, but the data was voluntarily provided by the researchers and in many cases is out of date or inaccurate.
By combining the two datasets I was able to get some of the advantages of a large dataset and some of the advantages of a smaller, more accurate dataset.
For validation on the manual dataset, I used cross-validation. For validation on the ClinicalTrials.gov dataset, I took the third digit of the trial’s NCT ID. Trials with values 0-7 were used for training, with value 8 were used for validation, and those with value 9 are held out as a future test set.
Validation scores for manual dataset
Validation scores on small manually labelled dataset (about 100 protocols labelled, but 300 labelled for number of subjects). You can reproduce my experiments using the notebooks from this folder.
|Component||Accuracy – manual validation dataset||AUC – manual validation dataset||Technique|
|Condition (Naive Bayes)||88%||100%||Naive Bayes|
|SAP (Naive Bayes)||85%||87%||Naive Bayes|
|Effect Estimate||73%||95%||Naive Bayes|
|Number of Subjects||69% (71% within 10% margin)||N/A||Rule based combined with Random Forest|
Validation scores for ClinicalTrials.gov dataset
You can reproduce my experiments using the notebooks from this folder. As a sanity check I also trained a Naive Bayes classifier for some of these components to check that our models are outperforming a reasonable baseline.
|Component||Accuracy – ClinicalTrials.gov validation dataset||Baseline Accuracy (Naive Bayes) – ClinicalTrials.gov validation dataset||Technique|
|Phase||75%||45%||Ensemble – rule based + random forest|
|Number of Subjects||13%||6%||Rule based combined with Random Forest|
|Number of Arms||58%||52%||Ensemble|
|Countries of Investigation||AUC 87%||N/A||Ensemble – rule based + random forest + Naive Bayes|
In particular I found that the ClinicalTrials.gov value for the sample size was particularly inaccurate, hence the very low performance of the model on that value.
Other results of the validation
By far the most difficult model was the number of subjects (sample size) classifier.
I designed this component as a stage of manually defined features to identify candidate sample sizes (numeric values in the text), combined with a random forest using these features to identify the most likely candidate. Here is an output of the feature importances of the random forest model.
Similarly, the simulation classifier is a random forest that uses manually defined features of key words:
For any of the components, we also plotted a Confusion Matrix.
Since each component is designed differently, it has been complex to validate the performance.
However I have provided some Jupyter notebooks in the repository to run the validation and reproduce my results.
There is still much scope for improvement of several features, especially sample size.
Some parameters, such as simulation, were not available in the ClinicalTrials.gov dataset and so could only be trained and validated manually. We hope to be able to annotate more data for these areas.