Jim Lynch reports that the Casualty Actuarial Society (CAS) is undertaking a review of its examination process. The bottom line: some pass rates are too low.
Here is a snip from a letter from the CAS president on the issue, mentioning that this isn’t their first rodeo:
With the agreement of the Vice President – Admissions and the Chair of the Examination Committee, I have directed CAS staff to engage a professional education consulting firm to perform an independent review of our process. A 2001 audit report of the CAS examination process and procedures led to the implementation of the following: (1) development and publication of the learning objectives for each exam syllabus, (2) training of exam item writers, and (3) a content-based pass mark process. The objective of this review will be to evaluate our current processes against best practices for adult professional education.
Jim adds some fascinating detail from the dark ages preceding that 2001 review:
In those days, there was a syllabus and not much else. You memorized its readings – sometimes down to the footnotes – and took the test. There were no rubrics. There was no idea how many points could be derived from any given paper. Without being plugged into the actuarial network – working for a big company, buying an ACTEX manual – you really couldn’t tell what the hell the reading was for, what was important, and how important it was.
Yikes.
Actuarial exams occupy an interesting place in the adult education market. They, along with the CFA exams, are a graduated testing system in which the vast majority of people taking an exam fail. That’s an amazing fact, considering that all of those taking later exams have passed a filter of at least one before. The reason is that later exams are harder.
But I don’t think that these exams are difficult in a Kafkaesque way, with lots of gotcha questions and ‘unfair’ tricks. They’re just trying to shovel a lot of content. The CAS exams, relative to the CFA exams, have a bit of moral high ground here with each level having a defined theme. I found the CFA levels to be a bit more arbitrarily divided.
In both cases, it’s all about the syllabus and its knowledge statements. If you genuinely pay attention to that and follow it, you should be fine. But who’s going to do that?
Much more importantly, these documents allow an industry of outstanding teach-to-the-test organizations (TTTT) to spring up.
The readings supplied with the CAS and CFA exams are abysmal. They’re incredibly difficult to use to actually LEARN stuff on your own. Now, you might say: “why don’t we just tell the CAS to be more like the TTTT companies?” Nope.
These companies are awesome because they need to compete on teaching prowess. In 2008, the CFA Institute forced candidates to buy its crappy books, using its monopoly power to try to force students into an educational stream.
I nearly failed an exam because I felt guilty about buying the Schweser books when I already had the material. I’d argue that the aggregate amount of knowledge gained by candidates went down that year. Is that not the most heinous of outcomes?
So if I had some advice for the CAS and for the CFA Institute (as a member of both), I’d say this: if you’re genuinely concerned about passing rates, the easiest way of getting them up is to double the number of exams and half their content. But that would be administratively painful and I’m not sure you really want to crank up the grades THAT much.
If you’re only concerned about making the exams fair then just work really really hard on the syllabus (with rubrics and knowledge statements and weightings and all the bells and whistles Jim missed out on) and make sure the test is absolutely true to it.
Then back the *#@^ off and let the market teach us.
Not Unreasonable 