334
views
1
recommends
+1 Recommend
2 collections
    1
    shares
      scite_
       
      • Record: found
      • Abstract: found
      • Conference Proceedings: found
      Is Open Access

      SciScore, a tool that can measure rigor criteria presence or absence in a biomedical study

      Published
      proceedings-article
        1 , , 1
      ScienceOpen
      RExPO22
      2-3 September, 2022
      text-mining, artificial intelligence, rigor, reproducibility, transparency, RRID, persistent identifiers
      Bookmark

            Abstract

            While reproducibility of a scientific study is possible to measure generally as another lab is attempting to replicate the study, a labor intensive and difficult proposition, determining whether there is sufficient information about reagents to track these down, or whether the study was undertaken in a rigorous way, is less difficult. SciScore is a software tool that can detect ~55 rigor criteria known to impact the reproducibility of a study and identify key research resources, which must be tracked down prior to attempting a replication attempt. Because the tool can detect the presence (+1) or absence (+0) of these key rigor criteria and the ability track down research resources, it has been used to assign an overall “rigor score” to each manuscript that it parses. This score is a whole number between 1 and 10 (best). The number is based on how many criteria are found vs the number that are expected for the type of study, rounded to the nearest whole number. Generally higher numbers mean that more rigor items, such as statements about blinding during the experiment and RRIDs for cell lines, are found in the paper. To determine how the literature overall was performing with regards to these rigor items, we ran the tool on ~2.5M papers licensed as “text mining permitted” in the Open Access subset of PubMed Central. This revealed that the average number of rigor criteria addressed by the average paper is 4/10 in 2020. The average ability to find organisms, cell lines, or antibodies is about 30% while software tools fare better with about 80% average findability. About 50% of papers address sex of subjects, and nearly 40% describe how subjects were randomized into groups. The study also reveals that the biomedical literature is getting better, but we collectively have a way to go.

            Content

            Author and article information

            Conference
            ScienceOpen
            15 July 2022
            Affiliations
            [1 ] SciCrunch Inc.
            Author notes
            Author information
            https://orcid.org/0000-0002-5497-0243
            Article
            10.14293/S2199-1006.1.SOR-.PPPXBQN6.v1
            8f3db553-1e98-4439-8bfa-d14b7051c339

            This work has been published open access under Creative Commons Attribution License CC BY 4.0 , which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Conditions, terms of use and publishing policy can be found at www.scienceopen.com .

            RExPO22
            Maastricht, Netherlands
            2-3 September, 2022
            History
            : 15 July 2022

            Data sharing not applicable to this article as no datasets were generated or analysed during the current study.
            transparency,artificial intelligence,text-mining,rigor,reproducibility,persistent identifiers,RRID

            Comments

            Comment on this article