Interactive Learning Online at Public Universities: Evidence from randomized trials

Williams Bowen
Matthew Chingos
Kelly Lack
Thomas Nygren


Higher education is facing serious challenges in the United States. There is increasing concern about rising costs, the quality of education, and that the nation is losing its “competitive edge.” Online learning—specifically highly interactive, closed-loop, online learning systems that we call ILO or Interactive Learning Online—holds the promise of broadening access to higher education to more individuals, while also lowering costs for students. But is the quality there? In our first report in this area, “Barriers to Adoption of Online Learning Systems in U. S. Higher Education,” we highlighted a broad, widely held concern about the quality of learning outcomes achieved through online learning. But do we actually know how interactive online learning systems really compare to the in-classroom experience? This second report was designed to help find answers. We used a strictly quantitative methodology to compare the two learning approaches in a rigorous way. In six different public institutions, we arranged for the same introductory statistics course to be taught. In each instance, a “control” group was enrolled in a traditional classroom-based course, then, a “treatment” group took a hybrid course using a prototype machine-guided mode of instruction developed at Carnegie Mellon University in concert with one face-to-face meeting each week. Students were assigned to these two groups by means of a carefully designed randomization methodology. The research we conducted was designed to answer these questions: ● Can sophisticated, interactive online courses be used to maintain or improve basic learning outcomes (mastery of course content, completion rates, and time-to-degree) in introductory courses in basic subjects such as statistics? ● Are these courses as effective, or possibly more effective, for minority and low-socioeconomic-status students and for other groups subject to stereotype threat? Or, are these groups less well suited to an online approach? ● Are such courses equally effective with not-so-well-prepared students and well-prepared students? The results of this study are remarkable, they show comparable learning outcomes for this basic course, with a promise of cost savings and productivity gains over time. More research is needed. Even though the analysis was rigorous, it was a single course. We need to learn more about the adaptability of existing platforms for offering other courses in different environments. Ithaka S+R is committed to continuing this research and sharing our findings broadly.


Interactivity, Minority, Low SES, Randomized trial


Six public university campuses (including two separate courses in two departments on one campus) that agreed to cooperate in a carefully designed research project utilizing random assignment techniques. How did learning outcomes compare across the treatment and control groups?
We first examine the impact of assignment to the hybrid format, relative to the
traditional format, in terms of the rate at which students completed and passed
the course, their performance on a standardized test of statistics (the CAOS
test), and their score on a set of final exam questions that were the same in the
two formats. Hybrid-format students did perform slightly better
than traditional-format students on three outcomes, achieving pass rates that whereabout three percentage points higher. On average, students learned
just as much in the hybrid format as they would have had they instead taken the
course in the traditional format—with “learning” measured in traditional ways,
in terms of course completion, course grades, and performance on a national test
of statistical literacy

About the Study

Links to Article
Publication Type Journal Article
In Publication
Type of Research
Research Design
Intervention/Areas of Study
Level of Analysis
Specific Populations Examined
Specific Institutional Characteristics of Interest
Specific Course or Program Characteristics
Outcome Variables of Interest
Student Sample Size
Citing Articles