In order to develop the Clinical Trial Risk Tool, we had to conduct a quality control exercise on the components. Each parameter which is fed into the complexity model is trained and evaluated independently. For an overview of how the tool works, please read this blog post.
I used two datasets to train and evaluate the tool:
By combining the two datasets I was able to get some of the advantages of a large dataset and some of the advantages of a smaller, more accurate dataset.
For validation on the manual dataset, I used cross-validation. For validation on the ClinicalTrials.gov dataset, I took the third digit of the trial’s NCT ID. Trials with values 0-7 were used for training, with value 8 were used for validation, and those with value 9 are held out as a future test set.
Validation scores on small manually labelled dataset (about 100 protocols labelled, but 300 labelled for number of subjects). You can reproduce my experiments using the notebooks from this folder.
| Component | Accuracy – manual validation dataset | AUC – manual validation dataset | Technique |
| Condition (Naive Bayes: HIV vs TB) | 88% | 100% | Naive Bayes |
| SAP (Naive Bayes) | 85% | 87% | Naive Bayes |
| Effect Estimate | 73% | 95% | Naive Bayes |
| Number of Subjects | 69% (71% within 10% margin) | N/A | Rule based combined with Random Forest |
| Simulation | 94% | 98% | Naive Bayes |
You can reproduce my experiments using the notebooks from this folder. As a sanity check I also trained a Naive Bayes classifier for some of these components to check that our models are outperforming a reasonable baseline.
| Component | Accuracy – ClinicalTrials.gov validation dataset | Baseline Accuracy (Naive Bayes) – ClinicalTrials.gov validation dataset | Technique |
| Phase | 75% | 45% | Ensemble – rule based + random forest |
| SAP | 82% | Naive Bayes | |
| Number of Subjects | 13% | 6% | Rule based combined with Random Forest |
| Number of Arms | 58% | 52% | Ensemble |
| Countries of Investigation | AUC 87% | N/A | Ensemble – rule based + random forest + Naive Bayes |
In particular I found that the ClinicalTrials.gov value for the sample size was particularly inaccurate, hence the very low performance of the model on that value.
By far the most difficult model was the number of subjects (sample size) classifier.
I designed this component as a stage of manually defined features to identify candidate sample sizes (numeric values in the text), combined with a random forest using these features to identify the most likely candidate. Here is an output of the feature importances of the random forest model.
Similarly, the simulation classifier is a random forest that uses manually defined features of key words:
For any of the components, we also plotted a Confusion Matrix.
Confusion matrix for the baseline (Naive Bayes) phase extractor on the ClinicalTrials.gov validation dataset
Since each component is designed differently, it has been complex to validate the performance for clinical trial risk assessment.
However I have provided some Jupyter notebooks in the repository to run the validation and reproduce my results.
There is still much scope for improvement of several features, especially sample size.
Some parameters, such as simulation, were not available in the ClinicalTrials.gov dataset and so could only be trained and validated manually. We hope to be able to annotate more data for these areas.
Guest post by Safeer Khan, Lecturer at Department of Pharmaceutical Sciences, Government College University, Lahore, Pakistan Multi-Arm & Multi-Stage (MAMS) Clinical Trials Design Tips The design of clinical trials is increasingly challenged by the Rising Costs, limited availability of eligible patient populations, and the growing demand for timely therapeutic evaluation. Traditional parallel-group designs, which typically compare a single intervention to a control, are often insufficient to meet these pressures in terms of speed, efficiency, and resource utilization.
You can use the t-test when you want to compare the means (averages) of continuous data between two groups, such as blood pressure or maximum concentration of a drug in urine (Cmax). If you have data with a dichotomous outcome, you can use the Chi-Squared test instead - please try our Chi-Squared sample size calculator. The calculator below will calculate the minimum sample size for you. Your expected effect size d is the standardised effect size according to Cohen’s definition.
You can use the Chi-Squared test to analyse your trial data or A/B test data if you have two groups with a dichotomous outcome. For example, you have two arms in your trial: the placebo and the intervention arm, and your endpoint is either yes or no, such as “did the subject experience an adverse event during the trial”. The calculator below will calculate the minimum sample size for you. Your expected effect size w is the standardised effect size according to Cohen’s definition.
