Open and Reproducible Science walks into a classroom

Insights from teaching a course on Psychology’s Credibility Revolution

Syllabus Course Materials Teaser Video Course technicals Interview (pdf)

Welcome to FORRT’s first Pedagogy. And what better way to kickstart this initiative than hosting a researcher and educator who has been widely recognized by her peers for her outstanding course and materials on theory of Open and Reproducible Science called Psychology’s Credibility Revolution?

FORRT is delighted to partner up with Julia Strand for its first Pedagogy!

Check out her course announcement (and teaser!)

Julia Strand is Associate Professor of Psychology at Carleton College and she teaches courses such as Introduction to Psychology, The Psychology of Spoken Words and Sensation and Perception. Julia has also recently prepared and taught a course on Psychology’s Credibility Revolution ( link) which excels in all aspects (e.g., content, creativity, visuals, & functionality).

FORRT has reached out to the academic community on social media to ask which questions other scholars would like Julia to answer about her teaching materials on the Psychology’s Credibility Revolution course (on both technical and non-technical spheres). Below you can find Julia’s answers to these questions. And you can also find in this OSF repository Julia’s complete teaching materials, from which everyone can learn, adapt, or repurpose.

Kudos to Julia for preparing such an amazing course and many thanks to her for being so open to partner up with FORRT to share her pedagogies with the wider community. We hope this can serve as an inspiration and help scholars, instructors and educational institutions interested in integrating open and reproducible scholarship tenets in their teaching and mentoring.

What is your teaching philosophy behind the course “Psychology’s Credibility Revolution”?

Julia: I’m trying to do two things with this class. The first is to familiarize students with concepts related to the replication crisis and subsequent credibility revolution. The second is to prepare my students to write their senior honors theses. At Carleton, all students complete a capstone project for their major (“comps”). In the Psychology department, the foundation of that comps is the term paper written in a seminar students take in the fall of their senior year. My Credibility Revolution course is one of those seminars. So in this course, I’m balancing teaching content with helping provide students with the tools they need to write their comps. The comps papers from my section are on a research topic of the student’s choosing, but written through the lens of something we’ve discussed. I’ve had students conduct systematic reviews, trace how findings have replicated (or not) on a particular topic, and evaluate the construct validity of measurement scales, among many others.

I want students to come away from this course with an appreciation for how research is actually conducted, how the incentive structure of academia can be at odds with scientific rigor, and how we can change our field to encourage science that is rigorous, transparent, and reproducible. I’ve found that teaching open science is a great way to help students be more critical consumers of the literature. I know that many of my students won’t go on to do research in Psychology themselves, but I want them all to have an appreciation for how to evaluate research.

Could you share your thoughts on how well the students grappled with the material and assignment in this course? Do you have an impression of whether this course influenced your students’ approaches to their own research projects?

Julia: Students can really get into this content! I’ve found that talking about open science has a relatively low barrier to entry, even for undergrads. Although it may take time to build up enough subject area expertise to evaluate psychological theories or models, students can readily understand why misplaced incentives will affect behavior, why questionable research practices lead to findings that can’t replicate, and so on. For example, I’ll describe two studies that are intended to test the same question (i.e., studies on age-related priming conducted by John Bargh vs Stéphane Doyen) but differ in methodology. Students then brainstorm all the reasons why the two studies may have reached different conclusions. They are incredibly thoughtful and thorough in identifying all the ways the choices that researchers make can influence the outcome of the study.

I ask my students for their impressions of how the course has changed how they engage with the literature, and they almost uniformly say they read more critically. They also regularly say (to my great delight!) that they pay much more attention to the methods sections.

Did you face any barriers to teaching about the credibility revolution? And if so, how did you overcome these?

Julia: I haven’t, really! My department is very supportive of letting faculty choose their own topics in these seminars and has been quite receptive to my incorporating open science principles in my teaching and research.

How many times have you taught this course? Could you give an indication of how much the course has developed over time?

Julia: I’ve taught it twice and will teach it again next year. The course has changed somewhat as I’ve looked for the right balance between teaching content and preparing my students for working on their senior theses. It also changes a bit each time because open science moves fast, so I need to update the content! But the basic bones of the course have been fairly consistent.

Is there anything that you didn’t have time to cover in this course but wish you did?

Julia: So many things. Given that Carleton is on trimesters and our terms are a short 9.5 weeks, and we also spend a substantial portion of that doing things related to comps, there are many topics I’d love to be able to include: the alpha wars, philosophy of science, computational and statistical reproducibility, meta-analyses, Bayesian vs frequentist approaches, and more. I ended up cutting content this year to take off pressure around the U.S election, as well.

What aspects of the course material do you think are the most important? For example if you only had 6 weeks to deliver this course.

Julia: It can be very easy for courses of this nature to be very pessimistic. The last thing I want is for students to come away with the sense “science is broken and everything is doomed.” So I spend a lot of time and energy helping them to understand all the ways our discipline, and science generally, has figured out to improve the practice of how we do science! I named this course Psychology’s Credibility Revolution rather than Psychology’s Replication Crisis because I really wanted to emphasize that the reforms of the last decade are changing how we do science, for the better.

Do you have a favorite part of the course? Do you think your students have a favorite too?

Julia: I love thinking about and teaching measurement and construct validation. I also love Halloween, and it works very nicely that my lessons on those topics tend to be around Halloween time. I created a fake scale that is intended to measure a construct called “Halloweenophilia” (an overwhelming love of all things halloween related) and ask students how they’d assess the construct validation of the instrument. It’s both spooky and fun.

Did you develop your materials from other teaching materials you found? And, what are your thoughts on adapting other’s teaching materials and making these available?

Julia: One reason I was keen to make my course publicly available is that others who teach similar courses have been so generous in sharing their materials! When I started designing this course, I combed through all the syllabi I could find online and made lots of notes about strategies I wanted to emulate and readings to adopt. So I’m happy to facilitate others doing the same.

A thing about teaching I wish I’d known earlier is that teaching doesn’t have to include a high proportion of invention. I adopt ideas from other teachers all the time. In fact, even if we don’t intend to, all of our teaching philosophies and approaches must be shaped by the courses we’ve taken and strategies we’ve seen others use.

Instructors need to curate the available resources/readings/assignments/approaches to achieve their course goals, fit their student population, etc, but they don’t need to design every single thing from scratch. In fact, adopting some materials from others frees you up to put more thought and energy into the aspects of your class that you create from scratch.

Do you have any general advice for a new(ish) lecturer wanting to organise a course similar to this?

Julia: Don’t be afraid to take inspiration from and borrow from others (see answer to question 8 above)!

In addition to the materials themselves (which are awesome), what should other teachers know or bear in mind when teaching this/similar material?

Julia: One challenge I faced is that much of this content is very interconnected—it’s hard to talk about low rates of replication without discussing publication bias, which is hard to talk about without mentioning null hypothesis significance testing, which then gets you into talking about alternatives to p-values….Which is all great, but it can be hard to wait weeks to talk about some topic that seems integral to understanding earlier content. So I think teaching this content requires coming to terms with the fact that you can’t do everything, and can’t do everything at once.

I’d also recommend using concrete examples from content areas you know well. My research is on spoken word recognition and listening effort (the cognitive resources necessary to understand speech), so when I teach content on measurement for example, I talk a lot about different ways of measuring listening effort. Working through examples that you are very familiar with can help students get an appreciation of the complexities of decisions about experiment design.

A few final, quick-fire technical questions: how did you record the lectures? How did you edit the lectures (e.g. to have your face and slides on screen)? How do you make the cartoon character of yourself?

See Julia’s response to all these technical questions in this great video she prepared!

All links

Suggested Citation

Strand, J. (2020) Open and Reproducible Science walks into a classroom: Insights from teaching a course on Psychology’s Credibility Revolution. FORRT Pedagogies. https://doi.org/10.17605/OSF.IO/ZPB8A

Team-Pedagogy Contributors:

Leticia Micheli, Leiden University, the Netherlands.

Flavio Azevedo, Cambridge University, UK.