In order to develop the Clinical Trial Risk Tool, we had to conduct a quality control exercise on the components. Each parameter which is fed into the complexity model is trained and evaluated independently. For an overview of how the tool works, please read this blog post.
I used two datasets to train and evaluate the tool:
By combining the two datasets I was able to get some of the advantages of a large dataset and some of the advantages of a smaller, more accurate dataset.
For validation on the manual dataset, I used cross-validation. For validation on the ClinicalTrials.gov dataset, I took the third digit of the trial’s NCT ID. Trials with values 0-7 were used for training, with value 8 were used for validation, and those with value 9 are held out as a future test set.
Validation scores on small manually labelled dataset (about 100 protocols labelled, but 300 labelled for number of subjects). You can reproduce my experiments using the notebooks from this folder.
| Component | Accuracy – manual validation dataset | AUC – manual validation dataset | Technique |
| Condition (Naive Bayes: HIV vs TB) | 88% | 100% | Naive Bayes |
| SAP (Naive Bayes) | 85% | 87% | Naive Bayes |
| Effect Estimate | 73% | 95% | Naive Bayes |
| Number of Subjects | 69% (71% within 10% margin) | N/A | Rule based combined with Random Forest |
| Simulation | 94% | 98% | Naive Bayes |
You can reproduce my experiments using the notebooks from this folder. As a sanity check I also trained a Naive Bayes classifier for some of these components to check that our models are outperforming a reasonable baseline.
| Component | Accuracy – ClinicalTrials.gov validation dataset | Baseline Accuracy (Naive Bayes) – ClinicalTrials.gov validation dataset | Technique |
| Phase | 75% | 45% | Ensemble – rule based + random forest |
| SAP | 82% | Naive Bayes | |
| Number of Subjects | 13% | 6% | Rule based combined with Random Forest |
| Number of Arms | 58% | 52% | Ensemble |
| Countries of Investigation | AUC 87% | N/A | Ensemble – rule based + random forest + Naive Bayes |
In particular I found that the ClinicalTrials.gov value for the sample size was particularly inaccurate, hence the very low performance of the model on that value.
By far the most difficult model was the number of subjects (sample size) classifier.
I designed this component as a stage of manually defined features to identify candidate sample sizes (numeric values in the text), combined with a random forest using these features to identify the most likely candidate. Here is an output of the feature importances of the random forest model.
Similarly, the simulation classifier is a random forest that uses manually defined features of key words:
For any of the components, we also plotted a Confusion Matrix.
Confusion matrix for the baseline (Naive Bayes) phase extractor on the ClinicalTrials.gov validation dataset
Since each component is designed differently, it has been complex to validate the performance for clinical trial risk assessment.
However I have provided some Jupyter notebooks in the repository to run the validation and reproduce my results.
There is still much scope for improvement of several features, especially sample size.
Some parameters, such as simulation, were not available in the ClinicalTrials.gov dataset and so could only be trained and validated manually. We hope to be able to annotate more data for these areas.
You may have been tasked with creating a clinical trial site budget. This is a budget itemising all the costs that will be incurred at the study site. The site budget may be needed To estimate the total cost of that site running part of the trial, as part of a bid to the CRO or sponsor To identify who needs to be reimbursed for each cost item To ascertain whether or not it is feasible to run the trial at that site To ensure that the site is reimbursed for the costs that they incur while running the trial If you are building a site budget, the most important document is the study protocol.
Fast Data Science are pleased to announce that the Clinical Trial Risk Tool, has been accepted as a supplier on the UK Government’s G-Cloud 15 framework. The G-Cloud 15 framework allows public sector bodies to buy cloud-based computing services such as AI, hosting, software and support directly without lengthy, costly traditional tender processes. What does the Clinical Trial Risk Tool do? The Clinical Trial Risk Tool helps users to analyse clinical trial protocols and documents.
Estimating the total cost of a clinical trial before it runs is challenging. Public data on past trial costs can be hard to come by, as many companies guard this information carefully. Trials in high income countries and low and middle income countries have very different costs. Upload your clinical trial protocol and create a cost benchmark with AI Protocol to cost benchmark The Clinical Trial Risk Tool uses AI and Natural Language Processing (NLP) to estimate the cost of a trial using the information contained in the clinical trial protocol.
