In addition to measures of learner actions, we can also examine participation through the lens of time. Several factors make persistence complex to calculate and articulate in HeroesX. First, participants can enter the course at any point. Second, there are no deadlines, so late registrants can complete all of the work of the course, even those who register on the final day of the course. The official launch of the course was March 12, 2013; course content was released every week or two through July 24, and the all responses to questions for credit were due on August 24. Some participants took the course on the pace that the content was released, completing two lecture units every week from March through May. Participants could also sign up for Heroes in August, and, to borrow a phrase from contemporary television viewing habits, “binge-watch” the videos and complete all assignments over a weekend.
As a result, two scales of time are of interest. Absolute time represents time from the launch of the course until its completion. Week 0 represents the official launch of the course; week -12 represents the first week where participants can register for HeroesX; the exam was launched in week 19; and all graded materials were due in week 23. Relative time is calculated from a participant’s enrollment, beginning with course launch or their registration, whichever is later. For a participant who registers in the week of course launch or before, relative week 0 is the same as absolute week 0, the launch of the course. For a participant who registers in absolute week 2, week 2 becomes their relative week 0. Absolute weeks allow us to ask, for example, whether many participants drop out after the second quiz. Relative weeks allow us to ask whether participants are most likely to drop out in their first or second week in the course.
To assess persistence, we first examined participants by their registration cohort: clustered by the absolute week in which they registered to join the course. We then, for each cohort, created a “hazard function” describing the risk of stopping out of the course with relative time on the x-axis. The “hazard proportion” at each time period takes all the participants who have been active up to that time period and asks, of these participants, what percentage will post their last activity this week, without earning a certificate? In the Figure 14, we plot these hazard functions by registration cohort on relative time. We show every other cohort, from those who registered 12 weeks before course launch (darkest) to those who registered 18 weeks after course launch (lightest), truncated at week 23, when all course materials were due. Cohorts that begin after the course launched (week 2–week 18) have dashed lines.
Figure 14. Hazard functions for every other registration cohort (from absolute week -11 to 19, darkest to lightest) plotted on relative course week, where week 0 is the week of course launch or the week of initial registration, whichever came later. Dashed lines are for cohorts after week 0. The hazard functions are truncated at absolute week 23, when the all graded material was due. Hazard probability represents the proportion of all participants who ceased activity without earning a certificate of the participants remaining in each time period.
Examining these hazard functions, they are remarkably similar. Of the cohorts that registered at or before the course launch (solid lines), where relative time is the same as absolute time, the hazard functions are very similar. There appear to be certain weeks that have higher hazard than others, and the risk of dropping out applies evenly across all cohorts. These weeks should be of great interest to the designers of HeroesX, and the phenomena of interest to all instructional designers. Notice also that the hazard functions of the cohorts registering after launch track the other cohorts over the first few weeks—the phenomena of high hazard in the first few weeks followed by lower levels of hazard applies regardless of registration time.
For clarity on the fluctuation in hazard from absolute week to week, we also plot these hazard functions on absolute time in Figure 15. Here the hazard functions converge in the later weeks and vary earlier on (we refer to the steep lines on the left side of this figure as the “rake;” the rake is on the left when plotted on absolute time and on the right when plotted on relative time.) This figure more clearly shows how hazard fluctuates over the course. HeroesX is particularly interesting to examine in this regard because the overall structure of the course does not vary from week to week. Every week there are videos, readings, close reading exercises, and questions, albeit in varying volume.
Figure 15. Hazard functions for every other registration cohort (from absolute week -11 to 19, darkest to lightest) plotted on absolute course week, where week 0 is the week of course launch. The hazard functions are truncated at week 23, when all graded material was due.
Changing hazard as a function of course structure—common in courses with exams and due dates—can be ruled out here. Changes in hazard from week to week must be from changes in the quality of course content, the calendar (holidays, etc.), issues with the edX platform that appear from time to time, or other variables. In HeroesX, we see spikes in hazard in weeks 5, 8, and 12. This is only speculative at this point, but one explanatory hypothesis is that these weeks correlate with the releases of some of the Hours that had substantially more content (in terms of numbers and hours of video length) than other Hours in the course. More detailed investigation is required, but this one detail suggests how examinations of these kinds of hazard models could help course teams tune and refine their syllabi from one course run to the next.
To summarize these hazard functions and display a survival curve, in Figure 16 we plot the average hazard function over relative course week. From this average function, we then directly calculate an “implied survival function.” As displayed here, the survival function drops sharply in the first week, and then the cohort size decreases more slowly from week to week. One way to summarize the figure is to say that regardless of when a participant registered, hazard was highest in the first two weeks, then fluctuated between 5% and 15% of the cohort dropping out in any given week. Put another way, once participants persisted past the second week, their risk of dropping out decreased considerably.
Figure 16. Hazard function comprised of the average of all registration cohort hazard functions plotted on relative course week (where week 0 is the course launch or initial registration week, whichever is later) and truncated at week 23 when all graded material was due. Implied survival function is calculated directly from the average hazard function.
These hazard models give some sense of how participants persist throughout the whole course, examining the span of time from first action to last action. However, a participant can log in only twice, on the first day and the last day, and be counted as “surviving” through almost the length of the course. This motivates an alternative metric that counts how many discrete days participants view course material. Among all registrants, we find that median number of days of activity is one, and 75% of registrants have four or fewer days of activity.
For a more granular view, therefore, in Figure 17 we examine the daily activity of those who have viewed over half the course (those who “explored”) and certificate earners. In the figure, we see that people make substantial investigations into the course over different time periods, ranging from only a few days to nearly 200 days with activity. There are five participants who earned a certificate in a single day of activity, and one certificate earner with activity on 198 discrete days. Similarly, many participants viewed content from over half of the chapters of the courseware in a single day, and several others engaged with the course on over 150 discrete days without earning a certificate.
Figure 17. Days with activity, the number of discrete days (demarked in UTC time) during the observational period where participants had at least one action, for explorers and certificate earners (n=1,733)