This webinar addresses questions related to writing, reviewing, editing, or funding a study using the Registered Report format, featuring Chris Chambers and ...
Interpreting the first results from the Reproducibility Project: Cancer Biology requires a highly nuanced approach. Reproducibility is a cornerstone of science, and the development of new drugs and medical treatments relies on the results of preclinical research being reproducible. In recent years, however, the validity of published findings in a number of areas of scientific research, including cancer research, have been called into question (Begley and Ellis, 2012; Baker, 2016). One response to these concerns has been the launch of a project to repeat selected experiments from a number of high-profile papers in cancer biology (Morrison, 2014; Errington et al., 2014). The aim of the Reproducibility Project: Cancer Biology, which is a collaboration between the Center for Open Science and Science Exchange, is two-fold: to provide evidence about reproducibility in preclinical cancer research, and to identify the factors that influence reproducibility more generally.
Currently, there is a growing interest in ensuring the transparency and reproducibility of the published scientific literature. According to a previous evaluation of 441 biomedical journals articles published in 2000–2014, the biomedical literature largely lacked transparency in important dimensions. Here, we surveyed a random sample of 149 biomedical articles published between 2015 and 2017 and determined the proportion reporting sources of public and/or private funding and conflicts of interests, sharing protocols and raw data, and undergoing rigorous independent replication and reproducibility checks. We also investigated what can be learned about reproducibility and transparency indicators from open access data provided on PubMed. The majority of the 149 studies disclosed some information regarding funding (103, 69.1% [95% confidence interval, 61.0% to 76.3%]) or conflicts of interest (97, 65.1% [56.8% to 72.6%]). Among the 104 articles with empirical data in which protocols or data sharing would be pertinent, 19 (18.3% [11.6% to 27.3%]) discussed publicly available data; only one (1.0% [0.1% to 6.0%]) included a link to a full study protocol. Among the 97 articles in which replication in studies with different data would be pertinent, there were five replication efforts (5.2% [1.9% to 12.2%]). Although clinical trial identification numbers and funding details were often provided on PubMed, only two of the articles without a full text article in PubMed Central that discussed publicly available data at the full text level also contained information related to data sharing on PubMed; none had a conflicts of interest statement on PubMed. Our evaluation suggests that although there have been improvements over the last few years in certain key indicators of reproducibility and transparency, opportunities exist to improve reproducible research practices across the biomedical literature and to make features related to reproducibility more readily visible in PubMed.
This module dwells on a number of methods (including old and new) available for research evaluation. The module comprises the following four units:
Unit 1. Introduction to Research Evaluation Metrics and Related Indicators.
Unit 2. Innovations in Measuring Science and Scholarship: Analytical Tools and Indicators in Evaluation Scholarship Communications.
Unit 3. Article and Author Level Measurements, and
Unit 4. Online Citation and Reference Management Tools.
Brief overviews of the units are presented below.
Unit 1 encompassed and discussed citation analysis, use of citation-based indicators for research evaluation, common bibliometric indicators, classical bibliometric laws, author level indicators using authors' public profiles, article level metrics using altmetric tools. It is to be noted that author level indicators and article level metrics are new tools for research evaluation. Author level indicators encompasses h index, citations count, i10 index, g index, articles with citation, average citations per article, Eigenfactor score, impact points, and RG score. Article level metrics or altmetrics are based on Twitter, Facebook, Mendeley, CiteULike, and Delicious which have been discussed. All technical terms used in the Unit have been defined.
Unit 2 deals with analytical tools and indicators used in evaluating scholarly communications. The tools covered are The Web of Science, Scopus, Indian Citation Index (ICI), CiteSeerX, Google Scholar and Google Scholar Citations. Among these all the tools except Indian Citation Index (ICI) are international in scope. ICI is not very much known outside India. It is a powerful tool as far Indian scholarly literature is concerned. As Indian journals publish a sizable amount of foreign literature, the tool will be useful for foreign countries as well. The analytical products with journal performance metrics Journal Citation Reports (JCR®) has also been described. In the chapter titled New Platforms for Evaluating Scholarly Communications three websites i.e. SCImago Journal & Country Rank (SJR) [ScimagoJR.com], eigenFACTOR.org, JournalMetrics.com and one software called Publish or Perish (POP) Software have been discussed.
Article and author level measurements have been discussed in Unit 3. Author and researcher identifiers are absolutely essential for searching databases in the WWW because a name like D Singh can harbour a number of names such as Dan Singh, Dhan Singh, Dhyan Singh, Darbara Singh, Daulat Singh, Durlabh Singh and more. The ResearcherID.com, launched by Thomson Reuters, is a web-based global registry of authors and researchers that individualises each and every name. Open Researcher and Contributor ID (ORCID) is also a registry that uniquely identifies an author or researcher. Both have been discussed in this Unit. Article Level Metrics (Altmetrics) has been treated in this Unit with the discussion as to how altmetrics can be measured with Altmetric.com and ImpactStory.org. Altmetrics for Online Journals has also been touched. There are a number of academic social networks of which ResearchGate.net, Academia.edu, GetCited.org, etc. have been discussed. Regional journal networks with bibliometric indicators are also in existence. Two networks of this type such as SciELO – Scientific Electronic Library Online, and Redalyc have been dealt with.
The last unit (Unit 4) is on online citation and reference management tools. The tools discussed are Mendeley, CiteULike, Zotero, Google Scholar Library, and EndNote Basic. The features of all the management tools have been discussed with figures, tables, and text boxes.
This is Module Four of the UNESCO's Open Access Curriculum for Researchers.
Full-Text is available at http://unesdoc.unesco.org/images/0023/002322/232210E.pdf
This list of resources consists of resources for researchers, editors, and reviewers interested in practicing open science principles, particularly in education research. This list is not exhaustive but meant as a starting point for individuals wanting to learn more about doing open science work specifically for qualitative research.This list was compiled by the following contributors: Rachel Renbarger, Sondra Stegenga, Thomas, Sebastian Karcher, and Crystal Steltenpohl. This resource list grew out of a hackathon at the Virtual Unconference on Open Scholarship Practices in Education Research.
As part of the Swiss Open Research Data Grants, the Swiss Reproducibility Network (SwissRN) organized two half-day workshops for researchers in all empirical disciplines and at all levels at SwissRN institutional members in Switzerland: one about preregistration and registered reports (presented by Evie Vergauwe and Caro Hautekiet) and one about data and research management (presented by Eva Furrer and Rachel Heyard). The two half-day workshops were held at four different locations: the University of Zurich and ETH Zurich (May 6th), the University of Bern (May 31st) and the University of Geneva (June 7th).
In the preregistration and registered report workshop, we covered questions such as (1) Why and how to preregister a study?, (2) What is the difference between a study preregistration and a registered report, and (3) How to deal with potential obstacles regarding study preregistration. In the practical part, we discussed situations one can encounter when preregistering a study or submitting a registered report, and how to deal with these situations. Additionally, participants got the opportunity to preregister a simplified example study to get a first, hands-on experience with preregistration.
In the data management workshop, we covered questions such as (1) How to best manage your data and research projects?, (2) What are the FAIR principles?, and (3) How can good meta data and documentation improve your research output? In the practical part, participants got a first taste of version control using Gitlab.
Contact Preregistration & RR workshop:
Caro - caro.hautekiet@uzh.ch
Evie - evie.vergauwe@unige.ch
Contact Data Management Workshop
Rachel - rachel.heyard@uzh.ch
Eva - eva.furrer@uzh.ch
Researchers, scholars and scientists main business is scholarly communication. We communicate about our work to others, as we push the boundaries of what we know and the society knows. We question established notions and truths about science. We share our findings with others, and in a way that is popularly known as scholarly communication which emerged with the publication of first journal in 1665. However, the term gained popularity only in the 1970s, as access to peer reviewed and scholarly communication became difficult. This module has four units covering introduction to scholarly communication, peer reviewed journals, electronica journals and databases and the Serials Crisis. At the end of this module, the learner is expected to be able to:
- Explain philosophy, mission, and objectives of scholarly communication
- Describe the process of scholarly communication
- Identify different channels of scholarly communication
- Discuss the dysfunctioning of the scholarly communication
In Unit 1, Introduction to scholarly communication, we have discussed different aspects of scholarly communication – particularly its genesis, importance and ethics of academic publishing, and different communication channels available in academic publishing. Some of these channels are commonly described as primary sources as they provide first-hand testimony or direct evidence concerning a topic under investigation. Historically, scientific journals were initiated by learned societies and other scholarly communities for reporting results of concluded research works or scientific discoveries. Now many forprofit publishers have started publishing research journals.
Unit 2, Communicating with Peer Review Journals, covers two important academic publishing channels, namely peer reviewed journals, conferences and their proceedings. This Unit also highlights different methods and procedures of peer reviewing for publishing primary literature emanated from research studies. The peer reviewing is essential for validating quality of research findings conveyed by researchers, which are subject to fulfilment of ethical standards and appropriate research design, sampling and other methodological issues.
In Unit 3, Electronic journals and databases, we have discussed the emergence of electronic journals in academic and research environment due to wide proliferation of information and communication technologies (ICT) in research communications and academic publishing. Scientific communities and scientific communications from the global South are getting substantive attentions through adaptation of electronic journals and electronic academic databases in the process of research communications.
In Unit 4, the Serials Crisis, we discuss the cost of peer reviewed publications and the problems faced by researchers in developing countries. The focus of this unit is on highlighting the problems and discusses possible solutions including the emergence of open access as one of the solutions. Open access journal publishing helps in mitigating some of the problems associated with serials crisis.
This is Module One of the UNESCO's Open Access Curriculum for Researchers.
Full-Text is available at http://unesdoc.unesco.org/images/0023/002319/231938e.pdf
Preregistration is the process of specifying project details, such as hypotheses, data collection procedures, and analytical decisions, prior to conducting a study. It is designed to make a clearer distinction between data-driven, exploratory work and a-priori, confirmatory work. Both modes of research are valuable, but are easy to unintentionally conflate. See the Preregistration Revolution for more background and recommendations.
For research that uses existing datasets, there is an increased risk of analysts being biased by preliminary trends in the dataset. However, that risk can be balanced by proper blinding to any summary statistics in the dataset and the use of hold out datasets (where the "training" and "validation" datasets are kept separate from each other). See this page for specific recommendations about "split samples" or "hold out" datasets. Finally, if those procedures are not followed, disclosure of possible biases can inform the researcher and her audience about the proper role any results should have (i.e. the results should be deemed mostly exploratory and ideal for additional confirmation).
This project contains a template for creating your preregistration, designed specifically for research using existing data. In the future, this template will be integrated into the OSF.
Preregistration is the process of specifying project details, such as hypotheses, data collection procedures, and analytical decisions, prior to conducting a study. It is designed to make a clearer distinction between data-driven, exploratory work and a-priori, confirmatory work. Both modes of research are valuable, but are easy to unintentionally conflate. See the Preregistration Revolution for more background and recommendations.
For research that uses existing datasets, there is an increased risk of analysts being biased by preliminary trends in the dataset. However, that risk can be balanced by proper blinding to any summary statistics in the dataset and the use of hold out datasets (where the "training" and "validation" datasets are kept separate from each other). See this page for specific recommendations about "split samples" or "hold out" datasets. Finally, if those procedures are not followed, disclosure of possible biases can inform the researcher and her audience about the proper role any results should have (i.e. the results should be deemed mostly exploratory and ideal for additional confirmation).
This project contains a template for creating your preregistration, designed specifically for research using existing data. In the future, this template will be integrated into the OSF.
The open science movement is rapidly changing the scientific landscape. Because exact definitions are often lacking and reforms are constantly evolving, accessible guides to open science are needed. This paper provides an introduction to open science and related reforms in the form of an annotated reading list of seven peer-reviewed articles, following the format of Etz, Gronau, Dablander, Edelsbrunner, and Baribault (2018). Written for researchers and students – particularly in psychological science – it highlights and introduces seven topics: understanding open science; open access; open data, materials, and code; reproducible analyses; preregistration and registered reports; replication research; and teaching open science. For each topic, we provide a detailed summary of one particularly informative and actionable article and suggest several further resources. Supporting a broader understanding of open science issues, this overview should enable researchers to engage with, improve, and implement current open, transparent, reproducible, replicable, and cumulative scientific practices.
This is the last Module of the course on Open Access for researchers. So far you have studied about Open Access, its history, advantages, initiatives, copyrights and licensing, evaluation matrix for research – all in the context of scholarly communication. In this Module with just two units, we would like to help you share your work in Open Access though repositories and journals. At the end of this module, you are expected to be able to:
- Understand the publication process involved in dissemination of scholarly works;
- Choose appropriate Open Access journals and repositories for sharing research results;
- Use social media to promote personal research work and build reputation.
In Unit 1, we discuss the research publication process at five stages – planning stage, preparing stage, pre-publication stage, publication stage and postpublication stage. We emphasize the importance of social media in sharing and making your work visible to the target groups.
In Unit 2, we focus on sharing your research through OA repositories and Journals. First we discussed the different types of repositories to select and highlighted the steps that you may consider including deposit in your own institutional repositories or in global open repositories. We then discuss the sources of finding and deciding on OA journals. This unit also provides guidance on choosing the right OA journals, as the quality of OA journals is often questioned.
This is Module Five of the UNESCO's Open Access Curriculum for Researchers.
Full-Text is available at http://unesdoc.unesco.org/images/0023/002322/232211E.pdf
Open Science accelerates the discovery of cures and advances new knowledge by improving the rigor and transparency of research and resusabilty of resulting data, materials, and code. But acceleration requires efficiency in managing the research lifecycle with integrated tools that work together to reduce the burden on investigators to manage, collaborate, and share as they work.
OSF provides the interface for research collaboration, with increased efficiency for the research team through integrated storage provider tools and citation managers.
Get familiar with the OSF project interface, understand how it can accelerate collaboration, transparency, sharing, and reuse of research outputs, and take a deep dive into integrations that connect directly into OSF for efficiency in researcher workflows.
Visit osf.io and help.osf.io to learn more.
The past decade has seen rapid growth in conversations around and progress towards fostering a more transparent, open, and cumulative science. Best practices are being codified and established across fields relevant to gerontology from cancer science to psychological science. Many of the areas currently under development are of particular relevance to gerontologists such as best practices in balancing open science with participant confidentiality or best practices for preregistering archival, longitudinal data analysis. The present panel showcases one of the particular strengths of the open science movement - the contribution that early career researchers are making to these ongoing conversations on best practices. Early career researchers have the opportunity to blend their expertise with technology, their knowledge of their disciplines, and their vision for the future in shaping these conversations. In this panel, three early career researchers share their insights. Pfund presents an introduction to preregistration and the value of preregistration from the perspective of “growing up” within the open science movement. Seaman discusses efforts in and tools for transparency and reproducibility in neuroimaging of aging research. Ludwig introduces the idea of registered reports as a particularly useful form of publication for researchers who use longitudinal methods and/or those who work with hard-to-access samples. The symposium will include time for the audience to engage the panel in questions and discussion about current efforts in and future directions for transparent, open, and cumulative science efforts in gerontology.
Students are introduced to concepts of sampling distributions, p-values, and hypothesis testing. Using both simulated and real data for methylmercury level in fish populations, students will determine whether observations fall within government safety guidelines for safe consumption.
In this webinar, a panel discusses licensing options, fundamentals in choosing a license for your research, and answers questions about licensing scholarship. The panel consists of moderator Joanna Schimizzi, Professional Learning Specialist at the Institute for the Study of Knowledge Management in Education, along with panelists Brandon Butler, Director of Information Policy, University of Virginia Library and Becca Neel, Assistant Director for Resource Management & User Experience, University of Southern Indiana for an informative discussion on licensing your research. Accessible and further resources for this event are available on OSF: https://osf.io/s4wdf/
A key component of scientific communication is sufficient information for other researchers in the field to reproduce published findings. For computational and data-enabled research, this has often been interpreted to mean making available the raw data from which results were generated, the computer code that generated the findings, and any additional information needed such as workflows and input parameters. Many journals are revising author guidelines to include data and code availability. This work evaluates the effectiveness of journal policy that requires the data and code necessary for reproducibility be made available postpublication by the authors upon request. We assess the effectiveness of such a policy by (i) requesting data and code from authors and (ii) attempting replication of the published findings. We chose a random sample of 204 scientific papers published in the journal Science after the implementation of their policy in February 2011. We found that we were able to obtain artifacts from 44% of our sample and were able to reproduce the findings for 26%. We find this policy—author remission of data and code postpublication upon request—an improvement over no policy, but currently insufficient for reproducibility.
Growth of the open science movement has drawn significant attention to data sharing and availability across the scientific community. In this study, we tested the ability to recover data collected under a particular funder-imposed requirement of public availability. We assessed overall data recovery success, tested whether characteristics of the data or data creator were indicators of recovery success, and identified hurdles to data recovery. Overall the majority of data were not recovered (26% recovery of 315 data projects), a similar result to journal-driven efforts to recover data. Field of research was the most important indicator of recovery success, but neither home agency sector nor age of data were determinants of recovery. While we did not find a relationship between recovery of data and age of data, age did predict whether we could find contact information for the grantee. The main hurdles to data recovery included those associated with communication with the researcher; loss of contact with the data creator accounted for half (50%) of unrecoverable datasets, and unavailability of contact information accounted for 35% of unrecoverable datasets. Overall, our results suggest that funding agencies and journals face similar challenges to enforcement of data requirements. We advocate that funding agencies could improve the availability of the data they fund by dedicating more resources to enforcing compliance with data requirements, providing data-sharing tools and technical support to awardees, and administering stricter consequences for those who ignore data sharing preconditions.
From January 2014, Psychological Science introduced new submission guidelines that encouraged the use of effect sizes, estimation, and meta-analysis (the “new statistics”), required extra detail of methods, and offered badges for use of open science practices. We investigated the use of these practices in empirical articles published by Psychological Science and, for comparison, by the Journal of Experimental Psychology: General, during the period of January 2013 to December 2015. The use of null hypothesis significance testing (NHST) was extremely high at all times and in both journals. In Psychological Science, the use of confidence intervals increased markedly overall, from 28% of articles in 2013 to 70% in 2015, as did the availability of open data (3 to 39%) and open materials (7 to 31%). The other journal showed smaller or much smaller changes. Our findings suggest that journal-specific submission guidelines may encourage desirable changes in authors’ practices.
It is widely believed that research that builds upon previously published findings has reproduced the original work. However, it is rare for researchers to perform or publish direct replications of existing results. The Reproducibility Project: Cancer Biology is an open investigation of reproducibility in preclinical cancer biology research. We have identified 50 high impact cancer biology articles published in the period 2010-2012, and plan to replicate a subset of experimental results from each article. A Registered Report detailing the proposed experimental designs and protocols for each subset of experiments will be peer reviewed and published prior to data collection. The results of these experiments will then be published in a Replication Study. The resulting open methodology and dataset will provide evidence about the reproducibility of high-impact results, and an opportunity to identify predictors of reproducibility.
Reproducibility and open scientific practices are increasingly being requested or required of scientists and researchers, but training on these practices has not kept pace, especially in biomedical research. This course intends to help bridge that gap and covers the fundamentals and workflow of data analysis in R.
The linked website contains the lesson, lecture, and assignment material for the course, as well as additional instructions to other potential instructors of this course material.