In order to develop the Clinical Trial Risk Tool, we had to conduct a quality control exercise on the components. Each parameter which is fed into the complexity model is trained and evaluated independently. For an overview of how the tool works, please read this blog post.
I used two datasets to train and evaluate the tool:
By combining the two datasets I was able to get some of the advantages of a large dataset and some of the advantages of a smaller, more accurate dataset.
For validation on the manual dataset, I used cross-validation. For validation on the ClinicalTrials.gov dataset, I took the third digit of the trial’s NCT ID. Trials with values 0-7 were used for training, with value 8 were used for validation, and those with value 9 are held out as a future test set.
Validation scores on small manually labelled dataset (about 100 protocols labelled, but 300 labelled for number of subjects). You can reproduce my experiments using the notebooks from this folder.
| Component | Accuracy – manual validation dataset | AUC – manual validation dataset | Technique |
| Condition (Naive Bayes: HIV vs TB) | 88% | 100% | Naive Bayes |
| SAP (Naive Bayes) | 85% | 87% | Naive Bayes |
| Effect Estimate | 73% | 95% | Naive Bayes |
| Number of Subjects | 69% (71% within 10% margin) | N/A | Rule based combined with Random Forest |
| Simulation | 94% | 98% | Naive Bayes |
You can reproduce my experiments using the notebooks from this folder. As a sanity check I also trained a Naive Bayes classifier for some of these components to check that our models are outperforming a reasonable baseline.
| Component | Accuracy – ClinicalTrials.gov validation dataset | Baseline Accuracy (Naive Bayes) – ClinicalTrials.gov validation dataset | Technique |
| Phase | 75% | 45% | Ensemble – rule based + random forest |
| SAP | 82% | Naive Bayes | |
| Number of Subjects | 13% | 6% | Rule based combined with Random Forest |
| Number of Arms | 58% | 52% | Ensemble |
| Countries of Investigation | AUC 87% | N/A | Ensemble – rule based + random forest + Naive Bayes |
In particular I found that the ClinicalTrials.gov value for the sample size was particularly inaccurate, hence the very low performance of the model on that value.
By far the most difficult model was the number of subjects (sample size) classifier.
I designed this component as a stage of manually defined features to identify candidate sample sizes (numeric values in the text), combined with a random forest using these features to identify the most likely candidate. Here is an output of the feature importances of the random forest model.
Similarly, the simulation classifier is a random forest that uses manually defined features of key words:
For any of the components, we also plotted a Confusion Matrix.
Confusion matrix for the baseline (Naive Bayes) phase extractor on the ClinicalTrials.gov validation dataset
Since each component is designed differently, it has been complex to validate the performance for clinical trial risk assessment.
However I have provided some Jupyter notebooks in the repository to run the validation and reproduce my results.
There is still much scope for improvement of several features, especially sample size.
Some parameters, such as simulation, were not available in the ClinicalTrials.gov dataset and so could only be trained and validated manually. We hope to be able to annotate more data for these areas.
A number of documents are needed to produce a complete and accurate clinical trial budget. The necessary documents typically include: the study protocol (generally the most important document for building a budget) the informed consent form the clinical trial agreement or sponsor contract any laboratory and pharmacy manuals the charge master or schedule of fees the sponsor’s budget. However, sometimes a sponsor will send only the synopsis to a CRO when requesting a quote.
Are you wondering how you can build a detailed clinical trial budget from the protocol, whether for a site, CRO, or sponsor? This may appear an intimidating task. You have to read the protocol carefully, find the schedule of events, calculate how many times each activity occurs during the trial, and slowly create an itemised budget spreadsheet. There are cost items buried in footnotes that you need to look for.
Guest post by Youssef Soliman, medical student at Assiut University and biostatistician In 2025, the outsourcing of clinical trials has become a common strategy for pharmaceutical and biotechnology sponsors. Facing rising R&D costs and complicated studies, sponsors turn to Contract Research Organizations (CROs) and other external partners to manage clinical trials. This practice, known as outsourcing clinical trials, is adopted as a best practice for containing costs and enhancing efficiency in drug development [1].
