How accurate is the Clinical Trial Risk Tool?

In order to develop the Clinical Trial Risk Tool, we had to conduct a quality control exercise on the components. Each parameter which is fed into the complexity model is trained and evaluated independently. For an overview of how the tool works, please read this blog post.

Datasets

I used two datasets to train and evaluate the tool:

Manual dataset – this was a set of between 100 and 300 protocols which I read through individually and annotated key parameters such as the sample size. The number annotated per parameter varied between 100 and 300.
ClinicalTrials.gov dataset – this was a much larger dataset of 11925 protocols downloaded from ClinicalTrials.gov. These came together with NCT ID, phase, pathology, SAP, number of arms and number of subjects, but the data was voluntarily provided by the researchers and in many cases is out of date or inaccurate.

By combining the two datasets I was able to get some of the advantages of a large dataset and some of the advantages of a smaller, more accurate dataset.

Validation

For validation on the manual dataset, I used cross-validation. For validation on the ClinicalTrials.gov dataset, I took the third digit of the trial’s NCT ID. Trials with values 0-7 were used for training, with value 8 were used for validation, and those with value 9 are held out as a future test set.

Validation scores for manual dataset

Validation scores on small manually labelled dataset (about 100 protocols labelled, but 300 labelled for number of subjects). You can reproduce my experiments using the notebooks from this folder.


Component	Accuracy – manual validation dataset	AUC – manual validation dataset	Technique
Condition (Naive Bayes: HIV vs TB)	88%	100%	Naive Bayes
SAP (Naive Bayes)	85%	87%	Naive Bayes
Effect Estimate	73%	95%	Naive Bayes
Number of Subjects	69% (71% within 10% margin)	N/A	Rule based combined with Random Forest
Simulation	94%	98%	Naive Bayes

Validation scores for ClinicalTrials.gov dataset

You can reproduce my experiments using the notebooks from this folder. As a sanity check I also trained a Naive Bayes classifier for some of these components to check that our models are outperforming a reasonable baseline.


Component	Accuracy – ClinicalTrials.gov validation dataset	Baseline Accuracy (Naive Bayes) – ClinicalTrials.gov validation dataset	Technique
Phase	75%	45%	Ensemble – rule based + random forest
SAP		82%	Naive Bayes
Number of Subjects	13%	6%	Rule based combined with Random Forest
Number of Arms	58%	52%	Ensemble
Countries of Investigation	AUC 87%	N/A	Ensemble – rule based + random forest + Naive Bayes

In particular I found that the ClinicalTrials.gov value for the sample size was particularly inaccurate, hence the very low performance of the model on that value.

Other results of the validation

By far the most difficult model was the number of subjects (sample size) classifier.

I designed this component as a stage of manually defined features to identify candidate sample sizes (numeric values in the text), combined with a random forest using these features to identify the most likely candidate. Here is an output of the feature importances of the random forest model.

Similarly, the simulation classifier is a random forest that uses manually defined features of key words:

For any of the components, we also plotted a Confusion Matrix.

Confusion matrix for the baseline (Naive Bayes) phase extractor on the ClinicalTrials.gov validation dataset

Conclusions

Since each component is designed differently, it has been complex to validate the performance for clinical trial risk assessment.

However I have provided some Jupyter notebooks in the repository to run the validation and reproduce my results.

There is still much scope for improvement of several features, especially sample size.

Some parameters, such as simulation, were not available in the ClinicalTrials.gov dataset and so could only be trained and validated manually. We hope to be able to annotate more data for these areas.