Month: March 2013

On the rate of attrition in MOOCs

On the rate of attrition in MOOCs


The Massively Open Online Class, or MOOC, has its high-profile journalistic celebrants,we know, Clay Shirky and Tom Friedman among them. Those more skeptical about the online learning revolution are not as well known, perhaps, but legion: professors, pundits, and higher ed journalists less unequivocally enthusiastic about the process and probable consequences of MOOCification, and whose opinions, as one might expect, run the gamut from mild skepticism to the demonstrably MOOCpocalyptic. Mostly from among this large latter camp, there has been a lot discussion in recent weeks about the attrition rate in most MOOCs that have been offered so far through the current “big three” MOOC providers: Coursera, Udacity, and edX. Commentators regularly observe that on average fewer than 10% of the students who register for a MOOC actually complete the requirements for the class. (For the most comprehensive current data on MOOC completion rates, see Katy Jordan’s website.) The disappointments (beyond a meager average 9% completion rate) have been notable too, including a completion rate of 3% at Duke University’s first MOOC with Coursera on bioelectricity, recently exhaustively anayzed by Yvonne Belanger at the Duke Center for Instructional Technology. In another recent case, a Coursera class at Georgia Tech was canceled to a severe round of criticism in the higher ed press and blogosphere. These failures have been publicized in the new year amidst the the latest round of heavily publicized rollouts of new university participants in one of the three platforms. Nor is the MOOC attrition rate confined as a topic of discussion to a few disgruntled voices in the blogosphere. In a recent editorial the New York Times acknowledged that low completion rates “appear to be a problem even in small-scale online courses when compared with traditional face-to-face classes” (“The Trouble with Online College” ). The low rate of completion has become a sticking point in contemporary debates about the present and future of MOOCs in higher education.

Different causes are offered for possible non-completion, some of the most frequently invoked of which we can enumerate here:

1) The registrant found the course too difficult.

2) The “soft” commitments of the internet may result in people enrolling without any serious intention to take the class. (At a recent presentation at MIT, Daphne Koller observed that 25% of the dropouts generally occur between registration and the first course posting – i.e., a full quarter leave of the class before the class even begins. Sue Gee reports a similar story for 6.002x Circuits and Electronics, an MITx/edX offering:

3) A simple shortage of time forces a drop out.

And other possibilities readily present themselves too as causes of attrition, including one somewhat less likely to be invoked but I think no less likely to be a cause of non-completion, which is that the course simply fails to hold the interest of student who signs up.

I am not so much interested in speculating on the causes of non-completion. Instead, I want to make a simple observation about why the completion rate for MOOCs has become a topic of great and growing concern in discussions of online learning in higher ed. It has emerged as a hot-button topic, I believe, because the completion rate for MOOCs makes particularly visible some stark differences between educators in “traditional” classrooms and the ed-tech entrepreneurs who have been the biggest boosters of MOOCs thus far. If you are in this latter group, a developer of or an investor in a tool for online learning, then the ability to reach many thousands of learners at a single time is going to seem self-evidently as the great advantage of online learning over the traditional classroom format. To an educator, however, one who thinks of their teaching primarily in terms of work with a group of students in a class, the idea of teaching a class that fewer than 5% of registrants complete will almost invariably seem like an unattractive option.

The difference between these populations may be most readily intelligible in terms of a difference between the kind of tool that these groups imagine the MOOC to be. To the ed-tech entrepreneur, the MOOC is a tool aimed to reach and appeal to the largest number of consumers. Viewed in these terms, the ability of the MOOC to reach large numbers of learners will seem as the standard by which success in MOOCs will be measured. (Members of this group do regularly point to the impressive numbers among those who do complete the requirements for a MOOC, still dwarfing the numbers possible in all but the largest lectures among the traditional forms of class instruction.) Approached as educators, however, the MOOC looks rather less like a tool for market and more like one developed specifically for use by the members of a group (the class), which begins on one date, ends on another, and is indeed the only ostensible reason for having a MOOC in the first place (as opposed to posting video lectures online). To this population, the increase by whatever factor of class size and/or of the number of students who complete the class will hardly compensate for the rather precipitous drop-off in student participation.

Among defenders of the MOOC, one frequent response to statistics on course completion is to call into question the value of completion rates as an assessment metric in the first place. While high dropout rates are problematic in traditional learning environments, the argument goes, the same phenomenon has no significant adverse effects on the digital learning environment of the MOOC. Keith Devlin, Professor of Mathematics at Stanford, and a repeat instructor of a MOOC with Coursera, puts the matter in these terms: “applying the traditional metrics of higher education to MOOCs is entirely misleading. MOOCs are a very different kind of educational package, and they need different metrics — metrics that we do not yet know how to construct.”

I take seriously Devlin’s proposition that there are metrics for assessing MOOCs that have yet to be devised, and which may provide measures for determining success more relevant than retention rates alone. Nor do I believe that it’s either productive or necessary to assume that the MOOC will simulate everything about the format of the traditional classroom, or that we should have to consider their successes on the basis of the same merits.

By the same token, however, I think there are compelling reasons to be concerned with existing rates of attrition in MOOCs, and to regard the dropout rate as a serious problem that the institutions responsible for online learning will need to address sooner or later. (One proposal for improving MOOC retention rates is by charging for these classes. The numbers of registrants would almost certainly go down in the event; beyond the obvious attractiveness of such a plan to MOOC providers and their financial backers, the argument in favor is that those who do pay to sign up will have a financial stake in completing the requirements for the class, and may therefore be more likely to do so.) Devlin asks: “is an 85 percent attrition rate really a problem? In fact, is it significantly different from traditional higher education?” The answer to the latter question is categorically yes. The class that 95% of students registered never attend, drop out, or fail in a “traditional learning environment” would be considered an abject failure in almost every scenario imaginable. Surely it is the case that if I taught a class that 95% or more of my students failed to complete, I would need to answer to those in positions of authority at my institution — and to be fair, I suspect, to many students who fell by the way as well.

To Devlin’s first question, then, of whether the MOOC attrition rate is “really a problem”: the answer will depend, as I have suggested, on whether you approach the MOOC from the perspective of an educator or of an ed-tech entrepreneur. Devlin asserts that the MOOC is in fact “a different kind of educational package” from the traditional university course, and so needs different metrics for determining success. The MOOC is indeed unlike most traditional classroom environments in that it is so far free; and 2) it has no admissions requirements, so that anyone can sign up. But as often observed, online learning is not itself a new practice; precedents include the Open University and MIT’s OpenCourseWare, among other past and present ventures.)

A columnist for Inside Higher Ed writes: “Although one interesting aspect of MOOCs is how many people drop out of them, it seems more worthwhile to focus on the hundreds who complete them and what their data teaches us about how people learn.”
But if you think of these questions — of the precipitous drop-out rate and about “how people learn” online — to be intimately connected, then you will reject the false choice presented here. Indeed it is a worrying sign that one could find thousands of course drop-outs entirely unrelated to the learning results that might be derived from those who complete the class. When Sebastian Thrun drew 150,000 registrants for his first Artificial Intelligence course, the experiment from which Udacity emerged, he did so to great acclaim and astonishment. This impressive figure is often cited, along with a similar figure for 6.002x, the first course offered through MITx. The numbers (if not rates) of those who complete requirements for a MOOC are often cited too. No doubt these are exciting and powerful developments, which may well in time revolutionize learning and teaching in higher ed. Between the tens of thousands who sign up for a MOOC and the considerable but nevertheless much smaller number that complete with a course certificate, however, one could probably find thousands of untold stories; this vast and so far mostly silent population may yet have a great deal to teach us about what works and doesn’t in online learning. We should know more than we do at present about the populations that do not complete MOOCs as well as about those who do.

There is much interesting data to be gathered (including and probably beyond that which is being gathered already) about MOOCs and on effective modes of online learning more generally. And there are important questions, well beyond the scope of my inquiry here, to be asked about what ultimately motivates people to stay in a MOOC, and what can be done to improve the rate of completion. In the meantime, pretending that the attrition rate does not present a problem at all for the future of online learning hardly seems an acceptable or a creditable position. It is for instance not clear to me why, if it is possible (even only in rarest situations) to see MOOCs with completion rates of around 20%, one would remain satisfied with completion rates of only 6 or 7%. But one reason why a chronically low completion rate may remain untroubling for ed-tech entrepreneurs and other MOOC boosters is that the “classroom” function of the MOOC has been superseded altogether. What emerges in its place is a system that rewards the individual achievement of high scorers while excluding from consideration the individual needs of learners or of specific learning communities that make up a class.

It seems to me that there are obvious dangers to pretending that teaching a MOOC is not at all like leading a class — for if it is not, what is it exactly that we are doing? MOOC supporters point to genuinely impressive statistics about global participation in online education thus far, and claim on this basis that online education stands to educate and elevate not an elite few but the entire world, regardless of country of residence or socioeconomic background, in a shared learning endeavor. But if MOOC providers do not attend seriously to problems of attrition, then they also have to abandon these high-flown and so far untested claims. If the corporations and institutions that claim for themselves the capacity to revolutionize higher education cannot be bothered to care about how and how many people are reached by their revolution, then what we are witnessing is not a learning revolution at all, but a commercial one.

March 2013

References / links

Katy Jordan, MOOC Completion Rates – the Data
*My thanks to Maryam Yoon for responding to an initial query and referring me to Katy Jordan’s website.

The highest posted completion rate among MOOCs offered by the current big three providers (19.2%), a two-month class on the programming language Scala, conducted at the Ecole Polytechnique fFédérale de Lausanne, with Coursera

Review of Duke University’s first MOOC, Bioelectricity:

Scott Jaschik, “MOOC Mess” (on the canceled MOOC at GA Tech)

Sue Gee, “MITx – the Fallout Rate”

Keith Devlin, “MOOCs and the Myth of Dropout Rate and Certification”

Dayna Catropa,“Big (MOOC) Data”

New York Times “The Trouble with Online College”

“MOOCs and the Emerging Digital Classroom,” Audio recording of MIT Communications Forum for March 21, 2013, with presentations by Daphne Koller, Anant Agarwal, and Alison Byerly; moderated by David Thorburn.