How well does Adapt predict Success: Part 3 of 3

Continued from Part 1 & Part 2.

The first factor of predicting success was discussed in Part 2. Now, let’s examine the second factor – the number of Adapt exams that a user has completed. Is this a good predictor of whether or notScreen Shot 2015-04-15 at 2.35.33 PM an Adapt user will pass? To predict if an Adapt user will pass the actual exam based on the number of practice exams completed, we performed a logistic regression analysis. The dependent variable is 1 if the user passes the exam, and 0 if the user fails the exam. The independent variable is the number of practice exams the user has completed. It turns out that at a 1% significance level, the number of completed exams has a positive association with the pass rate. This means that the more exams that a user completes, the more likely the user is to pass the exam.

This raises an interesting question. Between the earned level and the number of exams completed, which factor is more important in predicting the user’s Screen Shot 2015-04-15 at 2.36.12 PMsuccess? To answer this question, we performed a similar logistic regression analysis but with two independent variables. The first independent variable is the earned level, and the second independent variable is the number of practice exams completed. At a 1% significance level, the earned level has a positive association with the pass rate whereas the number of completed practice exams does not. Therefore, the conclusion is that the earned level is an important factor in predicting whether or not a user will pass the exam, whereas the number of practice exams completed is a relatively less important factor. What does that mean? That means to predict if you will pass the exam, it doesn’t matter as much whether you complete 5 or 15 practice exams. What matters more is your earned level. In other words, a user who Screen Shot 2015-04-15 at 2.36.46 PMcompleted fewer exams but had a higher earned level is more likely to pass than a user who completed more exams but had a lower earned level. So, that’s the second factor – the number of exams completed.

The third factor that could impact the success of an Adapt user is the features of Adapt. This includes the ability to customize exams, the CBT format, and discussion forum. The ability to customize exams allows students to adjust the practice exam difficulty level and choose question type. Users can decide whether to use SOA questions exclusively, original questions exclusively, or both. Adapt’s practice tests are conducted in CBT format which closely simulates how the actual exam is administered. There is a discussion forum where a user can interact with other users or ask questions that our exam specialists will respond to in a timely manner. We believe that the features of Adapt definitely increase the user’s odds of success. Unfortunately, it is hard to quantify this factor. We have, however, received feedback from Adapt users who found these features helpful.

Screen Shot 2015-04-15 at 2.37.05 PMIn conclusion, pass rates based on our survey results and conservative estimate, as well as our regression analysis, all support the fact that Adapt’s earned level is a good predictor of exam success.