GPT-4 Fails Final Actuarial Exam

1. Can GPT-4 get all the way to a pass? 
2. If it passes, would that mean its capabilities match an FCAS? 

I’d bet GPT-4 could pass some old exams now and with some clever hacking maybe it could get through this one, too. However, the exams are a moving target and they’ve been evolving away from GPT’s strengths for a generation. I think GPT will accelerate that evolution, strengthening the profession’s value proposition. 

Let’s dive in!

Context

These exams are hard, people³. The pass rate on the 2019 Exam 9 was 56% (338/601) and I like to point out that the 601 candidates who sat for this exam had mostly passed almost a dozen other exams with *similar pass rates*. In a world of grade inflation and decadence, most actuaries I speak to agree the exams have gotten harder over time. These are among the ultimate standardized tests that push candidates on analytical depth, domain knowledge and technical skill under a stressful time constraint. For goodness sake grading the exams is a brutal exercise that itself demands all these skills!

My process was to use a pretty simple prompt, starting each question by saying “this is an actuarial exam question” then pasting the question. Each question got a fresh instance of GPT 4. All answers were generated between March 23rd and 27th, 2023. As I was going through the answers I realized that I made some mistakes in transcribing the questions and regenerated the answers⁴. For a couple questions GPT refused to do calculations so I experimented with more prompts like “you are an actuarial student” etc (see notes) but I was unable to get it to show me the calculations so I kept all the answers from the prompt above. I’m sure there are ways of engineering better prompts that would generate dramatically better answers for some questions. But even though there is certainly low hanging fruit available to improve GPT’s score, I also think its successes and failures contain clues to some hard boundaries on its performance.

Performance Analysis

For analysis I’ll focus on questions 1, 6, 7, 18 and 19. 

Starting with problems 6 and 18, which were the two times GPT got full marks. These two problems were very well defined, fairly straightforward formulations of well known analytical models. These are the bread and butter of ‘easy’ exam questions since they don’t really challenge the candidate on comprehension. You have to memorize a technique and notice the problem requires it and work it without error. There are probably lots of examples online for these techniques and they are mostly worthless as analytical tools for a practicing actuary I think. If GPT embarrasses the profession into killing these questions, good riddance.

The next class of problem is one that GPT didn’t get correct but I think the gap to cross is small. Examples here are problems 1 and 7. In both these situations GPT got confused about some complicated modeling process that was presented in a weird way. It missed the “trick”. I think prompt engineering and some kind of plug-in⁵ that gives GPT a formal model structure to “plug and chug” and then leave it to interpret results will quickly improve its scores on these. Even humans don’t necessarily deeply understand the models we use and GPT interestingly often attempted to build a model from first principles when it didn’t recognize some obscure actuarial terminology. It really didn’t work. Which brings me to the last category.

I think the failure on 19 is most instructive of the limits of GPT. Here we have a toy model for calculating the capital requirements of an insurance company and GPT completely whiffed. This model is coherent in the sense that it captures underlying concepts of what a leverage model should cover but the figures and mathematical structure are ridiculous in their simplicity. It is recognizable only to someone who “speaks math” and understands what models are “trying to do”. Novel, ad hoc, toy models are tools of an effective actuary who is distilling a process to capture basic ideas and communicate with colleagues. Mostly these models don’t exist in a formal sense, aren’t studied on the internet and are reflections of the quality of judgment of a very good (or bad) actuary. Building and manipulating these models is creative analytical work at its best. I don’t see how GPT figures these out without practicing as an actuary for a decade. 

Implications for CAS Exams?

I predict the CAS exams will (and should!) continue to get ‘harder’ in the coming years, partly in response to GPT, which frankly is exposing the weakest parts of the exam system for what it is: wasteful memorization. But partly this has been the direction of the exam committee for a generation already and I think this strategy has been deeply vindicated by GPT. Many of our cousin professions are asking deeper existential questions than actuaries need to. It’s hard to see what a lawyer does after we integrate even today’s version of GPT into the economy, much less whatever on earth we have in 2030. For actuaries there is a pretty clear path to continued relevance. My favorite parts of the exams were when a question made me examine my own knowledge and experience and integrate it with the syllabus material to produce something new, which aligns to the differentiating skill of the best the actuaries. GPT won’t do that for a long while.  

So let’s keep focusing there!

Questions and Answers

See google doc

Notes

I gave GPT-4 the 2019 version of CAS Exam 9 which you can see here (including grading rubric). 

• I started a new prompt window for each numbered question. The only prompt I used was to say “this is an actuarial exam” then pasted the text below
• For some answers (Q1), GPT refused to perform calculations. After noticing this I tried a variety of different prompts like: “answer this as an actuarial student” “answer this with a perfect answer and perform all calculations” but it never gave me any calculations. So I kept the answers below since they’re equivalent to the answers I saw when experimenting with different prompts
• I noticed that GPT has some very favorite techniques for analyzing problems, liking VaR and Sharpe Ratio and pulled those in frequently. I think it lacks an ability to differentiate among more nuanced analytical techniques for highly adjacent problem spaces. 
• GPT performs much better in essay type questions that use words instead of analytical techniques to find numerical answers to problems
• Question 4: There is an idiosyncratic definition of market price of risk in the syllabus which was misinterpreted by gpt. This is a bit of a trick the cas pulled to write an exam question I think
• In question 5 the syllabus departs from investopedia’s analysis of CAPM vs APT. Why?
• On Question 6 it got all the calculations right. I made an error and regenerated the response which was really really wacky.. then I discovered another error so regenerated again and got it right.. how do the papers capture this kind of variability?
• In Q 7 “Tricks” in presenting problems really mess up its capabilities. Exams (and life!) are full of problem types that do not present themselves in a way that can be simulated on a test. Very complex interest rate term structures or scrambled problem information that requires a highly generalized model of the operation of a market totally mess it up.
• In a sense it feels like to the extent our specified models are accurate, GPT can navigate problem spaces. 
• In Q8, GPT totally missed the shortcut formulas and did not understand the concept of an insurance book that renews
• In Q9 the questions were pretty rote. The yield curve implications of prepayment risk were lost on gpt here which is kind of a layer deep on the standard terminology. I would bet that this could be improved with prompt engineering. 
• On question 14 we have a solid understanding of the relationship of irr and surplus and investment income even though the calculation mechanics are wonky. It can interpret results it seems, which is pretty neat. 
• On 15 there is a volatile mix of hallucination and real answers. Problems with the time value of money
• On 16, there are some conceptual mistakes in figuring out timings of cash flows and more complex discounting than straight PV
• For 17, it lacks an understanding of the cash flow timing of investment income and loss payments. There’s a kind of real world knowledge that actuaries have there that gpt doesn’t. 
• In 18, this question was right in the wheelhouse and GPT nailed it.
• For 19, this was an ad hoc arbitrary function for leverage and GPT both misunderstood the point of the function and how to use it to calculate capital.