This paper investigates the conceptual relationship between openness and reproducibility using a model-centric approach, heavily informed by probability theory and statistics. We first clarify the concepts of reliability, auditability, replicability, and reproducibility–each of which denotes a potential scientific objective. Then we advance a conceptual analysis to delineate the relationship between open scientific practices and these objectives. Using the notion of an idealized experiment, we identify which components of an experiment need to be reported and which need to be repeated to achieve the relevant objective. The model-centric framework we propose aims to contribute precision and clarity to the discussions surrounding the so-called reproducibility crisis.

Psychology is designed to meet scope and sequence requirements for the single-semester introduction to psychology course. The book offers a comprehensive treatment of core concepts, grounded in both classic studies and current and emerging research. The text also includes coverage of the DSM-5 in examinations of psychological disorders. Psychology incorporates discussions that reflect the diversity within the discipline, as well as the diversity of cultures and communities across the globe.Senior Contributing AuthorsRose M. Spielman, Formerly of Quinnipiac UniversityContributing AuthorsKathryn Dumper, Bainbridge State CollegeWilliam Jenkins, Mercer UniversityArlene Lacombe, Saint Joseph's UniversityMarilyn Lovett, Livingstone CollegeMarion Perlmutter, University of Michigan

By the end of this section, you will be able to:Explain what a correlation coefficient tells us about the relationship between variablesRecognize that correlation does not indicate a cause-and-effect relationship between variablesDiscuss our tendency to look for relationships between variables that do not really existExplain random sampling and assignment of participants into experimental and control groupsDiscuss how experimenter or participant bias could affect the results of an experimentIdentify independent and dependent variables

The analysis of longitudinal observational data can take many forms and requires many decisions, with research findings and conclusions often found to differ across independent longitudinal studies addressing the same question. Differences in measurements, sample composition (e.g., age, cohort, country/culture), and statistical models (e.g., change/time function, covariate set, centering, treatment of incomplete data) can affect the replicability of results. The central aim of the Integrative Analysis of Longitudinal Studies of Aging (IALSA) research network (NIH/NIA P01AG043362) is to optimize opportunities for replication and cross-validation across heterogeneous sources of longitudinal data by evaluating comparable conceptual and statistical models at the construct-level. We will provide an overview of the methodological challenges associated with comparative longitudinal and international research, including the comparability of alternative models of change, measurement harmonization and construct-level comparison, retest effects, distinguishing and contrasting between-person and within-person effects across studies, and evaluation of alternative models for change over time. These methodological challenges and recommended approaches will be discussed within the context of reproducible and replication research focused on longitudinal studies.

By the end of this section, you will be able to:Explain what a correlation coefficient tells us about the relationship between variablesRecognize that correlation does not indicate a cause-and-effect relationship between variablesDiscuss our tendency to look for relationships between variables that do not really existRecognize how correlations coefficients tell us about the relationships between variables specific to selection methods.

Replicability and Reproducibility in Comparative Psychology Psychology faces a replication crisis. The Reproducibility Project: Psychology sought to replicate the effects of 100 psychology studies. Though 97% of the original studies produced statistically significant results, only 36% of the replication studies did so (Open Science Collaboration, 2015). This inability to replicate previously published results, however, is not limited to psychology (Ioannidis, 2005). Replication projects in medicine (Prinz et al., 2011) and behavioral economics (Camerer et al., 2016) resulted in replication rates of 25 and 61%, respectively, and analyses in genetics (Munafò, 2009) and neuroscience (Button et al., 2013) question the validity of studies in those fields. Science, in general, is reckoning with challenges in one of its basic tenets: replication. Comparative psychology also faces the grand challenge of producing replicable research. Though social psychology has born the brunt of most of the critique regarding failed replications, comparative psychology suffers from some of the same problems faced by social psychology (e.g., small sample sizes). Yet, comparative psychology follows the methods of cognitive psychology by often using within-subjects designs, which may buffer it from replicability problems (Open Science Collaboration, 2015). In this Grand Challenge article, I explore the shared and unique challenges of and potential solutions for replication and reproducibility in comparative psychology.

In science, replication is the process of repeating research to determine the extent to which findings generalize across time and across situations. Recently, the science of psychology has come under criticism because a number of research findings do not replicate. In this module we discuss reasons for non-replication, the impact this phenomenon has on the field, and suggest solutions to the problem.

This is a clearinghouse for resources related to open science in Special Education. If you find a good resource that has not been included, please email it to marcy@cos.io.

Interpreting the first results from the Reproducibility Project: Cancer Biology requires a highly nuanced approach. Reproducibility is a cornerstone of science, and the development of new drugs and medical treatments relies on the results of preclinical research being reproducible. In recent years, however, the validity of published findings in a number of areas of scientific research, including cancer research, have been called into question (Begley and Ellis, 2012; Baker, 2016). One response to these concerns has been the launch of a project to repeat selected experiments from a number of high-profile papers in cancer biology (Morrison, 2014; Errington et al., 2014). The aim of the Reproducibility Project: Cancer Biology, which is a collaboration between the Center for Open Science and Science Exchange, is two-fold: to provide evidence about reproducibility in preclinical cancer research, and to identify the factors that influence reproducibility more generally.

Confirmation through competent replication is a founding principle of modern science. However, biomedical researchers are rewarded for innovation, and not for confirmation, and confirmatory research is often stigmatized as unoriginal and as a consequence faces barriers to publication. As a result, the current biomedical literature is dominated by exploration, which to complicate matters further is often disguised as confirmation. Only recently scientists and the public have begun to realize that high-profile research results in biomedicine can often not be replicated. Consequently, confirmation has become central stage in the quest to safeguard the robustness of research findings. Research which is pushing the boundaries of or challenges what is currently known must necessarily result in a plethora of false positive results. Thus, since discovery, the driving force of scientific progress, is unavoidably linked to high false positive rates and cannot support confirmatory inference, dedicated confirmatory investigation is needed for pivotal results. In this chapter I will argue that the tension between the two modes of research, exploration and confirmation, can be resolved if we conceptually and practically separate them. I will discuss the idiosyncrasies of exploratory and confirmatory studies, with a focus on the specific features of their design, analysis, and interpretation.

Current debates about the replication crisis in psychology take it for granted that direct replication is valuable, largely focusing on its role in uncovering questionable statistical practices. This article takes a broader look at the notion of replication in psychological experiments. It is argued that all experimentation/replication involves individuation judgments and that research in experimental psychology frequently turns on probing the adequacy of such judgments. In this vein, I highlight the ubiquity of conceptual and material questions in research, arguing that replication has its place but is not as central to psychological research as it is sometimes taken to be.

For a given research question, there are usually a large variety of possible analysis strategies acceptable according to the scientific standards of the field, and there are concerns that this multiplicity of analysis strategies plays an important role in the non-replicability of research findings. Here, we define a general framework on common sources of uncertainty arising in computational analyses that lead to this multiplicity, and apply this framework within an overview of approaches proposed across disciplines to address the issue. Armed with this framework, and a set of recommendations derived therefrom, researchers will be able to recognize strategies applicable to their field and use them to generate findings more likely to be replicated in future studies, ultimately improving the credibility of the scientific process.

It is widely believed that research that builds upon previously published findings has reproduced the original work. However, it is rare for researchers to perform or publish direct replications of existing results. The Reproducibility Project: Cancer Biology is an open investigation of reproducibility in preclinical cancer biology research. We have identified 50 high impact cancer biology articles published in the period 2010-2012, and plan to replicate a subset of experimental results from each article. A Registered Report detailing the proposed experimental designs and protocols for each subset of experiments will be peer reviewed and published prior to data collection. The results of these experiments will then be published in a Replication Study. The resulting open methodology and dataset will provide evidence about the reproducibility of high-impact results, and an opportunity to identify predictors of reproducibility.