Adam Russell, an anthropologist and program manager with the Defense Advanced Research Projects Agency (DARPA) was recently interviewed in Nature Index on the agency’s pursuit of tools that would assign confidence levels to published research in the social and behavioral sciences through its SCORE initiative (Systematizing Confidence in Open Research and Evidence). These tools will look for a mix of endogenous (internal) and exogenous (external) signals to assign a kind of credit score to the degree of reproducibility and replicability of claims. Endogenous signals might include sample sizes, pre-registration of research, open data, shared code, etc., while exogenous signals might include author and institution reputation, impact factors, peer review networks, social media commentary, post-publication review, funding sources, conflicts of interest, rates of retraction, and more.

While DARPA intends to use these tools in tandem with human experts to help the military calibrate the reliability of data it uses to inform its decisions, it knows this won’t be foolproof. When asked if he believes the issue of research irreproducibility will ever by completely solved, Russell says, “No, I don’t think irreproducibility will ever not be a feature of research. Science is really difficult and messy. But I do think the research community more generally can make improvements in where, how, and why irreproducibility occurs, and help limit the extent to which it impedes important scientific progress.”