SciScore, a tool that can measure rigor criteria presence or absence in a biomedical study

While reproducibility of a scientific study is possible to measure generally as another lab is attempting to replicate the study, a labor intensive and difficult proposition, determining whether there is sufficient information about reagents to track these down, or whether the study was undertaken in a rigorous way, is less difficult. SciScore is a software tool that can detect ~55 rigor criteria known to impact the reproducibility of a study and identify key research resources, which must be tracked down prior to attempting a replication attempt. Because the tool can detect the presence (+1) or absence (+0) of these key rigor criteria and the ability track down research resources, it has been used to assign an overall “rigor score” to each manuscript that it parses. This score is a whole number between 1 and 10 (best). The number is based on how many criteria are found vs the number that are expected for the type of study, rounded to the nearest whole number. Generally higher numbers mean that more rigor items, such as statements about blinding during the experiment and RRIDs for cell lines, are found in the paper. To determine how the literature overall was performing with regards to these rigor items, we ran the tool on ~2.5M papers licensed as “text mining permitted” in the Open Access subset of PubMed Central. This revealed that the average number of rigor criteria addressed by the average paper is 4/10 in 2020. The average ability to find organisms, cell lines, or antibodies is about 30% while software tools fare better with about 80% average findability. About 50% of papers address sex of subjects, and nearly 40% describe how subjects were randomized into groups. The study also reveals that the biomedical literature is getting better, but we collectively have a way to go.