The Generalizability Crisis

The Generalizability Crisis 

Subject: Research Methods

Duration: 60 minutes

Learning Objectives:

• Understand the concept of generalizability in research
• Recognize the factors that contribute to the generalizability crisis
• Discuss the implications of the generalizability crisis in various fields

Introduction

What is the generalizability crisis?

What factors contribute to it?

Activity 1

Activity 2 - Case studies

Discuss problems relating to design that you can identify in the following case studies.

1. A study examining the efficacy of a new heart medication conducted in a sample of primarily young, healthy individuals, with little representation from older adults and those with pre-existing heart conditions.

____________________

2. A study investigating the prevalence of depression in college students, conducted at a single university with a predominantly white student population, and with little representation from students of other races and ethnicities.

____________________

3. A study examining the effectiveness of a new cancer treatment conducted in a single hospital in a major metropolitan area, with little representation from patients living in rural or suburban areas.

____________________

4. A study examining the effects of a new teaching method on student achievement conducted in a single school with a homogeneous student population, and with little consideration given to cultural or linguistic differences.

____________________

5. A study investigating the effectiveness of a new therapy for anxiety conducted using only self-reported measures of anxiety, with no consideration given to other potential outcomes or measures.

____________________

6. A study examining the prevalence of a particular health condition conducted in a single survey administered to participants over the course of a single day, with no follow-up or longitudinal data collected.

____________________

7. A study examining the efficacy of a new mental health intervention that showed significant improvements in a single clinical trial, but was not replicated in subsequent trials or in real-world clinical settings.

Activity 3 - Implications

Reflecting on the case studies and research that you are familiar with - what could be some implications of these design flaws?

Now look at some specific examples. Can you add anything else?

Here are some specific examples of problems caused by the generalizability crisis. Match them to the titles:

• Some groups might be disadvantaged regarding depression treatment
• Lack of representation of diverse populations in clinical trials
• Some groups overlooked in genetic research
• Inaccurate treatment recommendations for women
• Black skin conditions not as easy to treat

1. Some studies on dermatological conditions have found that certain treatments may not be as effective for black patients due to differences in skin biology. For example, a study published in the Journal of the American Academy of Dermatology found that black patients with acne may have more resistant and inflammatory acne than white patients, which may require different treatment approaches. 

2. Many clinical trials have historically excluded certain populations, such as women, racial and ethnic minorities, and older adults. This can limit our understanding of how certain treatments or interventions work in these populations. For example, a study published in the Journal of the National Cancer Institute found that cancer clinical trials often underrepresent racial and ethnic minorities, which can limit our understanding of the effectiveness of cancer treatments in these populations.

3. Many studies on cardiovascular disease have historically focused on men, which can lead to inaccurate treatment recommendations for women. For example, a study published in the Journal of the American College of Cardiology found that women are less likely than men to receive evidence-based treatments for heart attacks, which can result in worse outcomes.

4. Some genetic research has been criticized for being too focused on populations of European ancestry, which can limit our understanding of genetic diversity and disease risk in other populations. For example, a study published in the American Journal of Human Genetics found that many genetic studies have underrepresented or excluded African populations, which can limit our understanding of genetic diversity and disease risk in these populations.

5. Some mental health studies have focused on populations from Western cultures, which can limit our understanding of mental health in other cultures. For example, a study published in the Journal of Cross-Cultural Psychology found that many measures of depression and anxiety may not be appropriate for use in non-Western cultures, which can limit our understanding of mental health in these populations.

Activity 4 - Strategies to address the problem

Look back at the case studies and discuss / reflect on how these flaws could be addressed. Brainstorm how the problem can be mitigated in your field.