Framework for
Open and
Reproducible
Research
Training
We have written a manuscript about documenting and tracking replication efforts. In psychological science, replicability — repeating a study with a new sample achieving consistent results — is critical for affirming the validity of scientific findings. Despite its importance, replication efforts are few and far between in psychological science with many attempts failing to corroborate past findings. This scarcity, compounded by the difficulty in accessing replication data, jeopardizes the efficient allocation of research resources and impedes scientific advancement. Addressing this crucial gap, we present the Replication Database, a novel platform hosting >1,200 original findings paired with replication findings. The infrastructure of this database allows researchers to submit, access, and engage with replication findings. The database makes replications visible, easily findable via a graphical user interface, and tracks replication rates across various factors, such as publication year or journal. This will facilitate future efforts to evaluate the robustness of psychological research. More info about the project and other FORRT replication efforts can be found on the Replication Hub. This effort was also highlighted by the journal Science in their news at a glance!
Röseler, L., Kaiser, L., Doetsch, C., Klett, N., Seida, C., Schütz, A., Aczel, B., Adelina, N., Agostini, V., Alarie, S., Albayrak-Aydemir, N., Aldoh, A., Al-Hoorie, A. H., Azevedo, F., Baker, B. J., Barth, C. L., Beitner, J., Brick, C., Brohmer, H., Chandrashekar, S. P., Chung, K. L., Cockcroft, J. P., Cummins, J., Diveica, V., Dumbalska, T., Efendic, E., Elsherif, M., Evans, T., Feldman, G., Fillon, A., Förster, N., Frese, J., Genschow, O., Giannouli, V., Gjoneska, B., Gnambs, T., Gourdon-Kanhukamwe, A., Graham, C. J., Hartmann, H., Haviva, C., Herderich, A., Hilbert, L. P., Holgado, D., Hussey, I., Ilchovska, Z. G., Kalandadze, T., Karhulahti, V.-M., Kasseckert, L., Klingelhöfer-Jens, M., Koppold, A., Korbmacher, M., Kulke, L., Kuper, N., LaPlume, A., Leech, G., Lohkamp, F., Lou, N. M., Lynott, D., Maier, M., Meier, M., Montefinese, M., Moreau, D., Mrkva, K., Nemcova, M., Oomen, D., Packheiser, J., Pandey, S., Papenmeier, F., Paruzel-Czachura, M., Pavlov, Y. G., Pavlović, Z., Pennington, C. R., Pittelkow, M.-M., Plomp, W., Plonski, P. E., Pronizius, E., Pua, A. A., Pypno-Blajda, K., Rausch, M., Rebholz, T. R., Richert, E., Röer, J. P., Ross, R., Schmidt, K., Skvortsova, A., Sperl, M. F. J., Tan, A. W. M., Thürmer, J. L., Tołopiło, A., Vanpaemel, W., Vaughn, L. A., Verheyen, S., Wallrich, L., Weber, L., Wolska, J. K., Zaneva, M., & Zhang, Y. (2024). The Replication Database: Documenting the Replicability of Psychological Science. Journal of Open Psychology Data, 12: 8, pp. 1–23. DOI: https://doi.org/10.5334/jopd.101
The preprint can be found here (osf.io/me2ub).
We have written a manuscript entitled Teaching open and reproducible scholarship: a critical review of the evidence base for current pedagogical methods and their outcomes. This manuscript describes the available (empirical) evidence of the impact (and importance) of integrating open scholarship into higher education, its benefits and challenges on three specific areas: students’ (a) scientific literacy; (b) engagement with science; and (c) attitudes towards science. This paper was borne out of colleagues reporting a need for something tangible to take to Heads of Departments to show pedagogical rationale for open scholarship (above and beyond the moral/‘good for science’ case that wasn’t always being heard on the ground). Cite as:
Pownall, M., Azevedo, F., König, L. M., Slack, H. R., Evans, T. R., Flack, Z., … & FORRT. (2023). Teaching open and reproducible scholarship: a critical review of the evidence base for current pedagogical methods and their outcomes. Royal Society Open Science, 10(5), Article 221255. https://doi.org/10.1098/rsos.221255
The preprint can be found here (osf.io/9e526).
We have written a manuscript entitled The replication crisis has led to positive structural, procedural, and community changes. This manuscript reviews how research structures, procedures and communities have changed in response to the replication crisis, showing that the replication “crisis” has been a positive credibility revolution. We also outline how improvements can continue to be made by accounting for all three levels. This manuscript was born out of the fact that much of the literature on the Replication Crisis has focused on understanding the potential underlying causes of low replicability, and framing solutions as a matter of individual researcher reform, and our community wanted to provide a wider perspective on what has been learnt since the crisis by highlighting the role of structural, procedural factors, and evidencing the rapid progress made by social scientists as a response. Cite as:
Korbmacher, M., Azevedo, F., Pennington, C. R., Hartmann, H., Pownall, M., Schmidt, K., Elsherif, M., Breznau, N., Robertson, O., Kalandadze, T., Yu, S., Baker, B. J., O’Mahony, A., Olsnes, J. Ø.-S., Shaw, J. J., Gjoneska, B., Yamada, Y., Röer, J. P., Murphy, J., Alzahawi, S., Grinschgl, S., Oliveira, C. M., Wingen, T., Yeung, S. K., Liu, M., König, L. M., Albayrak-Aydemir, N., Lecuona, O., Micheli, L., & Evans, T. (2023). The replication crisis has led to positive structural, procedural, and community changes. Communications Psychology, 1, 3. https://doi.org/10.1038/s44271-023-00003-2.
The preprint can be found here (osf.io/r6cvx).
We have written a manuscript entitled Opening up understanding of neurodiversity: A call for applying participatory and open scholarship practices. This manuscript gives a brief overview of what participatory research methods are and why they are important for promoting neurodiversity in academia. Cite as:
Gourdon-Kanhukamwe, A., Kalandadze, T., Yeung, S., Azevedo, F., Iley, B. J., Phan, J. M., … Elsherif, M. M. (2023). Opening up understanding of neurodiversity: A call for applying participatory and open scholarship practices. The Cognitive Psychology Bulletin, 8, 23-27. https://doi.org/10.53841/bpscog.2023.1.8.23
The preprint can be found here (osf.io/jq23s).
In response to the varied and plural new terminology introduced by the open scholarship movement, which has transformed academia’s lexicon, we have produced a community and consensus-based Glossary to facilitate education and effective communication between experts and newcomers. FORRT members have written a manuscript entitled A Community-Sourced Glossary of Open Scholarship Terms presenting the beta 0.1 version of our glossary of open scholarship terms. Cite as:
Parsons, S., Azevedo, F., Elsherif, M. M., Guay, S., Shahim, O. N., Govaart, G. H., Norris, E., O’Mahony, A., Parker, A. J., Todorovic, A., Pennington, C. R., Garcia-Pelegrin, E., Lazić, A., Robertson, O. M., Middleton, S. L., Valentini, B., McCuaig, J., Baker, B. J., Collins, E., … Aczel, B. (2022). A Community-Sourced Glossary of Open Scholarship Terms. Nature Human Behaviour, 6(3), 312-318. https://doi.org/10.1038/s41562-021-01269-4.
The preprint can be found here (osf.io/kdqcw) and here is the copy-edited version at the publisher’s website.
We have written a manuscript entitled Towards a culture of open scholarship: The role of pedagogical communities describing (a) the the need to integrate open scholarship principles into research training within higher education; (b) the benefit of pedagogical communities and the role they play in fostering an inclusive culture of open scholarship; and (c) call for greater collaboration with pedagogical communities, paving the way for a much needed integration of top-down and grassroot open scholarship initiatives. Cite as:
Azevedo, F., Liu, M., Pennington, C. R., Pownall, M., Evans, T. R., Parsons, S., Elsherif, M. M., Micheli, L., Westwood, S., & FORRT. (2021). Towards a culture of open scholarship: The role of pedagogical communities. BMC Research Notes, 15(1), 1-5. https://doi.org/10.1186/s13104-022-05944-1
Despite the growing awareness of the benefits of training reproducible methods within undergraduate and postgraduate curricula, there remains a lack of systematic incorporation of open scholarship practices in taught courses across Higher Education. While the potential reasons for this are diverse, one reason may be the dearth of available ready-to-use educational resources. To support instructors aiming to bridge that gap, FORRT’s community built open educational resources that can be integrated into taught courses ‘out of the box’. As creating or changing course content can be onerous and time-consuming, we aimed to make evidence-based, high-quality lesson plans and activities available to teaching faculty, thus reducing the labour required to develop and implement open scholarship content. We compiled lesson plans and activities, and categorized them based on their theme, learning outcome, and method of delivery, which are made publicly available here: FORRT’s Lesson Plans. Cite as:
Pownall, M., Azevedo, F., Aldoh, A., Elsherif, M. M., Vasilev, M. R., Pennington, C. R., … Parsons, S. (2021). Embedding open and reproducible science into teaching: A bank of lesson plans and resources. Scholarship of Teaching and Learning in Psychology. Advance online publication. https://doi.org/10.1037/stl0000307.
The preprint can be found here (osf.io/fgv79).
We have written a manuscript, entitled International Initiatives to Enhance Awareness and Uptake of Open Research in Psychology: A Systematic Narrative Review, which outlines the global landscape of Open Science initiatives within psychology. This preprint provides a systematic review of 187 initiatives, categorized into procedural, structural, and community-based changes, highlighting the roles they play in advancing transparency, replicability, and collaboration in research practices. The review aims to increase awareness, adoption, and coordination of these initiatives across psychology and beyond, serving as a roadmap for disciplines experiencing similar challenges.
Status: Our paper is preprinted 🎉🥳 in PsyArxiv. Cite as:
Skubera, M., Korbmacher, M., Evans, T.R., Azevedo, F., Pennington, C.R. (2024). International Initiatives to Enhance Awareness and Uptake of Open Research in Psychology: A Systematic Narrative Review. https://doi.org/10.31234/osf.io/rakmb.
We have written a manuscript entitled Bridging Neurodiversity and Open Scholarship: How Shared Values Can Guide Best Practices for Research Integrity, Social Justice, and Principled Education explaining that Neurodiversity is fundamental to the understanding of human behaviour and cognition and that neurodivergent individuals are often stigmatised, devalued, and objectified. The authors describe systematic barriers, issue with disclosure, directions on prevalence and stigma, intersection of neurodiversity and open scholarship, and provide recommendations that can lead to personal and systematic changes to improve acceptance of neurodivergent individuals. We also present the perspectives of neurodivergent authors, the majority of whom have personal lived experiences of neurodivergence(s), and provide recommendations for how research and academia can and should be improved in terms of research integrity, inclusivity and diversity. We end with discussing systematic barriers, issue with disclosure, directions on prevalence and stigma, intersection of neurodiversity and open scholarship, and provide recommendations that can lead to personal and systematic changes to improve acceptance of neurodivergent individuals.
Status: Our paper is preprinted 🎉🥳 in MetaArxiv. Cite as:
Elsherif, M. M., Middleton, S. L., Phan, J. M., Azevedo, F., Iley, B. J., Grose-Hodge, M., … Dokovova, M. (2022, June 20). Bridging Neurodiversity and Open Scholarship: How Shared Values Can Guide Best Practices for Research Integrity, Social Justice, and Principled Education. https://doi.org/10.31222/osf.io/k7a9p.
We have written a manuscript entitled Introducing a Framework for Open and Reproducible Research Training (FORRT) describing the importance of integrating open scholarship into higher education, its benefits and challenges, as well as about FORRT initiatives aiming to support educators in this endeavor.
The preprint can be found here (osf.io/bnh7p).
Status: Our manuscript is currently under review. Cite as:
Azevedo, F., Parsons, S., Micheli, L., Strand, J., Rinke, E., … & FORRT (2019, December 13). Introducing a Framework for Open and Reproducible Research Training (FORRT). https://doi.org/10.31219/osf.io/bnh7p
📄 For the Official PLoS Blog, on creating a comprehensive, dynamic database cataloging scientific claims and subsequent replication attempts across various disciplines of social, cognitive, and behavioral sciences.
📜 For The Psychologist, on open scholarship pedagogical communities.
📄 For the Observer, of the Association for Psychological Science, on Navigating Academia as Neurodivergent Researchers.
📃 For In-Mind Magazine, on What is Open Science and Why Does It Need a Glossary.
📜 For the Center of Open Science Blog, on Integrating Open and Reproducible Science Principles into Higher Education.
FORRT was asked by the UK’s Quality Assurance Agency for Higher Education (QAA) to produce a short briefing note in collaboration with the UK Reproducibility Network (UKRN) about the importance of embedding open scholarship into higher education. In the note, we argued that students need to understand and experience open and rigorous research processes to become effective and responsible consumers of evidence. We structured this around three themes (Thinking About Research, Doing Research, and Engaging With Research) and showed how this can be implemented across different fields.
The briefing note can be found here.
Status: Our briefing note is being used for QAA’s work around standards and quality in Higher Education. Cite as:
Azevedo, F., DeBruine, L., Evans, T. R. (2023). Open Scholarship in QAA Briefing. Framework for Open and Reproducible Research Training & UK Reproducibility Network. https://forrt.org/publications
FORRT responded to the the call from the UK House of Commons Science and Technology Committee on for evidence on reproducibility and research integrity, and the roles different institutions play in this arguing that one important component of research integrity that is often absent from discussion: the pedagogical consequences of how we teach, mentor, and supervise students through open scholarship. Our argument boils down to ‘If you want to go fast, go alone. If you want to go far, go together.’
The submission can be found here.
Status: Our submission is currently being considered by the UK’s Parliament Science and Technology Committee. Cite as:
Azevedo, F., Liu, M., Pennington, C. R., Pownall, M., Evans, T. R., Parsons, S., Elsherif, M. M., Micheli, L., Moreau, D., & FORRT. (2021). Towards a culture of open scholarship: The role of pedagogical communities. (Report No. RRE0080). Written evidence on reproducibility and research integrity. UK Parliament. [ UK Parliament, PDF, html]
We have written to UKRN (UK’s Reproducibility Network) and the British Psychological Society (BPS) a short note on our organization and goals, as well as providing 3 (three) recommendations:
The briefing note can be found here.
Status: Our briefing note was informative to UKRN & the British Psychological Society. Cite as:
Parsons, S., Azevedo, F., & FORRT (2020). Briefing note for degree accrediting societies. Framework for Open and Reproducible Research Training. https://forrt.org/publications
We are currently preparing additional manuscripts on a range of topics. To find out more about what we’re working on and how you can contribute, visit our Get Involved page.