{"id":1688,"date":"2022-12-17T16:06:52","date_gmt":"2022-12-17T16:06:52","guid":{"rendered":"http:\/\/clinicaltrialrisk.org\/?p=1688"},"modified":"2023-01-23T12:57:28","modified_gmt":"2023-01-23T12:57:28","slug":"accurate-clinical-trial-risk-tool","status":"publish","type":"post","link":"https:\/\/clinicaltrialrisk.org\/accurate-clinical-trial-risk-tool\/","title":{"rendered":"How accurate is the Clinical Trial Risk Tool?"},"content":{"rendered":"\n
In order to develop the Clinical Trial Risk Tool, we had to conduct a quality control exercise on the components. Each parameter which is fed into the complexity model is trained and evaluated independently. For an overview of how the tool works, please read this blog post<\/a>.<\/p>\n\n\n\n I used two datasets to train and evaluate the tool:<\/p>\n\n\n\n By combining the two datasets I was able to get some of the advantages of a large dataset and some of the advantages of a smaller, more accurate dataset.<\/p>\n\n\n\n For validation on the manual dataset, I used cross-validation. For validation on the ClinicalTrials.gov dataset, I took the third digit of the trial’s NCT ID. Trials with values 0-7 were used for training, with value 8 were used for validation, and those with value 9 are held out as a future test set.<\/p>\n\n\n\n Validation scores on small manually labelled dataset (about 100 protocols labelled, but 300 labelled for number of subjects). You can reproduce my experiments using the notebooks from this folder<\/a>.<\/p>\n\n\n\nDatasets<\/h2>\n\n\n\n
\n
Validation<\/h2>\n\n\n\n
Validation scores for manual dataset<\/h2>\n\n\n\n
Component<\/td> Accuracy – manual validation dataset<\/td> AUC – manual validation dataset<\/td> Technique<\/td><\/tr> Condition (Naive Bayes)<\/td> 88%<\/td> 100%<\/td> Naive Bayes<\/td><\/tr> SAP (Naive Bayes)<\/td> 85%<\/td> 87%<\/td> Naive Bayes<\/td><\/tr> Effect Estimate<\/td> 73%<\/td> 95%<\/td> Naive Bayes<\/td><\/tr> Number of Subjects<\/td> 69% (71% within 10% margin)<\/td> N\/A<\/td> Rule based combined with Random Forest<\/td><\/tr> Simulation<\/td> 94%<\/td> 98%<\/td> Naive Bayes<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n Validation scores for ClinicalTrials.gov dataset<\/h2>\n\n\n\n