How well does Adapt predict Success: Part 2 of 3

Continued from Part 1.

How Well Does Adapt Predict Success?

To answer this question, let’s first look at the factors that could impact the success of an Adapt user. We’ve identified 3 possible factors. First, the user’s earned level. Second, the number of practice exams the user has completed. Third, the features of Adapt. The first two factors are quantifiable, but the third factor is much harder to quantify. I’m going to talk about each of these factors.

Let’s start with the first factor – the earned level. Is earned level a good predictor of whether or not an Adapt user will pass the exam? One approach to answer this question is to look at our survey results. In 2013, a total

Screen Shot 2015-04-15 at 2.17.20 PMof 3372 Adapt users responded to our surveys. Only 33% of users with an EL of less than 3 passed the exam. This means about two-third of users with an EL of less than 3 failed. We discovered that as the EL increases, the number of users who passed the exam increases. About 90% of the users with an EL of at least 7 passed the exam. This is a very impressive statistic. Imagine there are 10 students in a class and all 10 students achieved an earned level of at least 7. According to our survey results, 9 out of these 10 students will pass the exam.

Survey results should be taken with a grain of salt. It is possible that users who passed the exam are more likely to respond to our surveys. If that were the case, our pass rates Screen Shot 2015-04-15 at 2.17.54 PMwould be inflated. To remove such bias from our results, we considered a second approach – a more conservative approach. We started by matching the names of our Adapt users in 2013 against the names on the SOA’s list of passing candidates in 2013. An exact match is considered a pass. Anything other than an exact match is considered a fail. This approach is conservative because it is not possible to count a fail as a pass but it is possible to count a pass as a fail. This could happen if a user registers for Adapt and the actual exam using different names. For example, suppose Dave registers for Adapt as Dave Kester, but registers for the actual exam as David Kester. Even if Dave passes the exam, he will be counted as a fail because the names don’t match exactly.

 

Because of this conservative assumption, the pass rates based on this approach are relatively lower than the pass rates from the survey results. However, even with the conservative approach, 83% of the users with an EL of at least 7 passed the exams. In other words, 4 out of 5 people who achieved an EL of at least 7 passed the exams. This is a pretty good number! Contrast this to the pass rate of someone who had an EL of less than 7. Screen Shot 2015-04-15 at 2.19.43 PMOnly a mere 42% of that group passed the exams. Getting to an earned level of at least 7 raises your odds of passing by almost 100%!

One undeniable fact in both the survey and the conservative approach is that the higher the earned level of the user, the more likely the user will pass the exam. This fact holds for all exams. So, that’s the first factor – the earned level.