Evaluation of Evidence-Based Practices in Online Learning: A Meta-Analysis and Review of Online Learning Studies

Barbara Means
Yukie Toyama
Robert Murphy
Marianne Bakia
Karla Jones


A systematic search of the research literature from 1996 through July 2008 identified more than a thousand empirical studies of online learning. Analysts screened these studies to find those that(a) contrasted an online to a face-to-face condition, (b) measured student learning outcomes, (c) used a rigorous research design, and (d) provided adequate information to calculate an effect size. As a result of this screening, 51 independent effects were identified that could be subjected to meta-analysis. The meta-analysis found that, on average, students in online learning conditions performed better than those receiving face-to-face instruction. The difference between student outcomes for online and face-to-face classes—measured as the difference between treatment and control means, divided by the pooled standard deviation—was larger in those studies contrasting conditions that blended elements of online and face-to-face instruction with conditions taught entirely face-to-face. Analysts noted that these blended conditions often included additional learning time and instructional elements not received by students in control conditions. This finding suggests that the positive effects associated with blended learning should not be attributed to the media, per se. An unexpected finding was the small number of rigorous published studies contrasting online and face-to-face learning conditions for K–12 students. In light of this small corpus, caution is required in generalizing to the K–12 population because the results are derived for the most part from studies in other settings (e.g., medical training, higher education).



Addresses four research questions:
1. How does the effectiveness of online learning compare with that of face-to face instruction?
2. Does supplementing face-to-face instruction with online instruction enhance learning?
3. What practices are associated with more effective online learning?
4. What conditions influence the effectiveness of online learning?
The main finding from the literature review was that
• Few rigorous research studies of the effectiveness of online learning for K–12 students have been published. A systematic search of the research literature from 1994 through 2006 found no experimental or controlled quasi-experimental studies comparing the learning effects of online versus face-to-face instruction for K–12 students that provide sufficient data to compute an effect size. A subsequent search that expanded the time frame through July 2008 identified just five published studies meeting meta-analysis
criteria. The meta-analysis of 51 study effects, 44 of which were drawn from research with older learners,
found that:

• Students who took all or part of their class online performed better, on average, than those taking the same course through traditional face-to-face instruction. Learning outcomes for students who engaged in online learning exceeded those of students receiving face-to-face instruction, with an average effect size of +0.24 favoring online conditions.
Instruction combining online and face-to-face elements had a larger advantage relative to purely face-to-face instruction than did purely online instruction.
Studies in which learners in the online condition spent more time on task than students in the face-to-face condition found a greater benefit for online learning.
The effectiveness of online learning approaches appears quite broad across different content and learner types.
Effect sizes were larger for studies in which the online and face-to-face conditions varied in terms of curriculum materials and aspects of instructional approach in addition to the medium of instruction.
Blended and purely online learning conditions implemented within a single study
generally result in similar student learning outcomes.
Elements such as video or online quizzes do not appear to influence the amount that students learn in online classes.
Online learning can be enhanced by giving learners control of their interactions with media and prompting learner reflection.
Providing guidance for learning for groups of students appears less successful than does using such mechanisms with individual learners.
These studies DO NOT show that online learning is superior as a medium
In addition, although the types of research designs used by the studies in the meta-analysis were strong (i.e., experimental or controlled quasi-experimental), many of the studies suffered from weaknesses such as small sample sizes, failure to report retention rates for students in the
conditions being contrasted, and, in many cases, potential bias stemming from the authors’ dual roles as experimenters and instructors.

APA Citation

Means, B., Toyama, Y., Murphy, R., Bakia, M., & Jones, K. (2009). Evaluation of evidence-based practices in online learning: A meta-analysis and review of online learning studies.

About the Study

Links to Article https://scholar.google.com/scholar?hl=en&as_sdt=5%2C50&sciodt=0%2C50&cites=8281964792081445913&scipsc=&q=evaluation+of+evidence+based+practices+in+online+learning&btnG=
Mode Online
Publication Type Report
In Publication U.S. Department of Education
Type of Research Quantitative
Research Design Meta-analysis
Intervention/Areas of Study
Level of Analysis
Specific Populations Examined Undergraduates
Peer-Reviewed Yes
Specific Institutional Characteristics of Interest
Specific Course or Program Characteristics
Outcome Variables of Interest
Student Sample Size
Citing Articles https://scholar.google.com/scholar?cites=14854806085574797598&as_sdt=5,50&sciodt=0,50&hl=en