Executive Summary
Monte Carlo analysis has become an increasingly popular arrow in the financial planner's quiver, as an improvement over the oversimplified traditional straight-line projection. Unfortunately, though, use of Monte Carlo analysis has begun to focus excessively on a singular probability of success, that itself can be almost as misleading as straight-line projections when not viewed in proper context. However, this is not a flaw of the Monte Carlo approach itself, but instead of the tools being used by financial planners. Instead, what's ultimately needed is software that shows not just the probability of success, but also the magnitude and consequences of failure, and a sensitivity analysis that helps clients understand the impact of the trade-off decisions they have available. What can ultimately result is a "next generation" of Monte Carlo analysis, that provides a more useful, relevant, and actionable framework to help clients make effective financial planning decisions.
The inspiration for today's blog post is a recent discussion I had with another financial planner, who was very critical of using Monte Carlo analysis, suggesting that it provides results which aren't meaningful to clients because it doesn't help them to understand trade-offs and make decisions.
"But that's not a problem with Monte Carlo analysis," I replied, "that's a problem with bad Monte Carlo software!"
Probability Of Success AND Magnitude Of Failure
It is certainly true that simply presenting a singular "probability of success" for a client's retirement plan is not a very effective tool for making a financial planning decision. Although as a methodology, analyzing a financial planning scenario on a Monte Carlo basis is at least better than simply using a straight line projection, because it accounts for the possibility of a range of outcomes and the impact of volatility and return sequencing on the results, trying to draw a conclusion from a single data point is problematic.
For instance, as I've written in the past, when viewed solely on the basis of the probability of failure, the plan with the lowest risk of failure might not even be the best choice. This is because just looking at the probability of success or failure fails to take into account the magnitude of the failures that do occur. Would you rather have scenario A, an 85% probability of success, or scenario B, a 90% probability? Clearly, the latter. But what if the 15% failure scenario A only required a 5% spending cut to get back on track, and the latter 10% failure scenario B would require a 50% spending cut, including selling your house, to get back on track? Would you still prefer B with a 10% chance of losing your house to A with a 15% chance of just losing a few nights out on the town? Suddenly, the opposite choice emerges; the scenario with the less intense failure scenario is preferable. In fact, it appears that in most situations, when clients select what probability of success they want, they're in fact expressing how intensely concerned they are about the failure scenario.
Monte Carlo Sensitivity Analysis
While better quantifying not just the probabilities of success but also the magnitude of the failures is an improvement to the Monte Carlo process, it still falls short of becoming an effective decision-making tool for clients, because ultimately know the probability of success and the magnitude of failure is still only relevant when it's compared to other scenarios with different results. In other words, it doesn't help to know that scenario A has an 85% probability of success and a 15% probability of a 5% spending cut until you also know the details of scenario B and can compare the two.
While that may be helpful in a subset of situations where a client has two specific "what-if" scenarios to compare, in the more generalized case, clients often don't realize or understand what trade-offs they have available, and the impact of those decisions. For instance, what's the benefit of saving 5% more per year? Is that better or worse than cutting spending 5% in retirement? How does that compare to retiring 1 year later? Or adding 10% in equity exposure to increase growth?
In the ideal world, Monte Carlo analysis software should provide a "sensitivity analysis" that shows the impact of various adjustments to the plan and the associated probability of success. Such an evaluation can also help the client understood how sensitive the plan is to its underlying assumptions - for instance, what happens if long-term returns are lower or inflation is higher in the future? Results could be presented in a table, as shown below:
|
Factor to Change |
Impact on Probability of Success |
|
+/- 5% in retirement spending |
9%
|
|
+/- 5% in pre-retirement savings |
3%
|
|
+/- 1 year in expected retirement date |
6%
|
|
+/- 1 year in expected mortality age |
2%
|
|
+/- 10% in equity exposure |
5%
|
|
+/- 1% in long-term expected bond return |
2%
|
|
+/- 1% in long-term expected stock return |
3%
|
|
+/- 1% in long-term expected inflation rate |
6% |
(Editor's Note: For simplicity, it is assumed for the purposes of Figure 8 that positive and negative changes to the factors have a single positive or negative impact on the probability of success; in point of fact, though, it is possible that a positive change could have a positive impact of greater magnitude than a comparable negative change, or vice versa.)
Bringing It All Together
For the typical client, who likely has little familiarity with Monte Carlo analysis, retirement planning in general, or how various retirement-related decisions can impact each other, a sensitivity analysis can become a launching point for further discussion about various trade-offs - developed specifically for the client's own individual circumstances. For some clients, adding equities might be more appealing than retiring later; for other clients, it might be preferable to save more now if they can clearly see how it impacts their ability to spend more in the future. It can be very helpful to know, per the chart above, that charts to retirement spending have 3x the impact of changes in pre-retirement saving, for a particular client scenario.
Ultimately, a combination of several factors, and/or changes to a greater degree than the sensitivity analysis shows, will still require a more detailed "what-if" scenario comparison. Going deeper into specific scenarios also helps to show the impact on not just the probability of success, but the magnitudes of failure as well. Overall, by providing a starting point for understanding what changes do and do not have the greatest impact on the plan, clients can gain a better perspective on the available trade-offs and decide what's most important to them.
But the bottom line is that while Monte Carlo analysis results are sometimes hard to connect with or act upon for clients, that may be a consequence of poor outputs, not a problem with the underlying approach itself! What's ultimately needed is a "next generation" of Monte Carlo analysis tools that provide a more useful, relevant, and actionable framework to help clients make effective financial planning decisions.
So what do you think? Would a sensitivity analysis help you to explain the consequences of trade-off decisions for clients? Do you think it would be useful for clients trying to make decisions? Does your Monte Carlo software offer this feature? Do you wish it did? Would your clients think differently about probabilities of success if we also more clearly showed the magnitude and consequences of the failures?
Michael:
I agree completely. In fact, in an upcoming paper in August issue of Journal of Financial Planning, I delve into metric that captures the magnitude of failure as part of introduction to an Analysis Framework for Retirement Income Strategies.
Not only that, you are aware that Income Discovery, the software that my firm developed to help advisors illustrate various retirement income strategies and their performance along multiple metrics: probability of success and Years of Income in Bad Market, the second risk metric. Almost all the sensitivity analysis you described in the table in your blog can be accomplished within the tool.
Finally, an advanced tool to do sensitivity analysis of a retirement income plan under Monte Carlo Simulation and present such analysis to the client is available.
Michael,
Hurray! As you know I have been harping on this problem for a long time.
I have three issues I want to bring up.
With respect to sensitivity analysis, I don’t think the success % is the right metric (dependent variable) for comparison (people really don’t have any idea of what % is appropriate). I think most people want to keep the success % fixed (at whatever the planner recommends) and see how these various adjustments would affect their living standard. For example, each year I delay retirement increases my retirement living standard by 5%.
Second, one of the challenges of using Monte Carlo to determine the magnitude of failure is that Monte Carlo doesn’t provide you with a single number. Instead as you push out the probability envelope, the failure gets worse and worse. In other words, the .1% worst case looks worse than the 1% worst case which looks worse than the 5% chance worst case.
So how far out the probability curve do we have to look or do we use some kind of weighted average (e.g., expected shortfall). We probably have to look fairly far out since we often buy life insurance for an event that has less than a 1% chance of occurring.
Which brings me to my final issue. Currently Monte Carlo simulations mostly focus on variations in returns, most of them ignore risk of death, risk of long term care, etc. Instead, these are left to be dealt with on a scenario by scenario basis. Which begs the question of whether Monte Carlo should be expanded to incorporate all these risks or whether we should just drop Monte Carlo and use scenario modeling for return variation as well (e.g., model what happens if PE10 drops to 9).
Great post!
David
Again, you continue to spike a nerve in me. Candidly, many financial firms care only about some form of “output” required to hand to a client and ostensibly support the sale of a fee-based product.
Much of the modeling inspires false hope and fulfills a “planning metric,” or planning goal by the firm. So to me anyway, current modeling is to benefit the firms and compliance departments, not the clients.
Some form of sensitivity analysis is indeed required.
I tackle this subject in a book I wrote coming out in September.
I believe Jim Otar has created a smarter simulator by using actual U.S. stock market history as his guide (www.retirementoptimizer.com)
I’m entertained to think of current Monte Carlo models this way: Monte Carlo is like a bee that swarms in close proximity to a hive and rarely strays too far, or even to another hive. Sort of like a bee on a leash. Attempting to believe with full confidence you can rein in the primal nature of a bee is ridiculous. It’s how a MC simulation attempts to contain risk in the controlled environment of a normal curve.
Please continue your good work. Looking forward to reading Manish’s work in JFP as well.
Michael –
Are you leading us toward Portfolio Pathfinder?
1. BEFORE SELECTION, ZERO IN ON BEST PORTFOLIOS WITH A GOAL FRONTIER GRAPH.
How did you choose the portfolio? Not with a silly short-term-fear questionnaire, I hope. Even Bodie (or one of them) now sneers at that. Of course, do Monte Carlo before the selection. Do it for the client’s plan and goals with each of a series of portfolios along the conservative-to-aggressive range. Then compare them in probability of meeting the goals, on a GOAL frontier graph. Inform the client for zeroing in on the best – instead of diverting his attention to his fear of short-term wobble, aka “risk.”
2. HOW DO THE BEST PORTFOLIOS COMPARE IN HOW FAR ABOVE OR BELOW THE GOAL RESULTS MAY BE?
For this, use a graph comparing any two of the best portfolios in result-probability distributions. Make it scrollable, so clients can see how the portfolios compare in probabilities for various target heights for final value.
3. HOW DO THE BEST PORTFOLIOS COMPARE IN PROBABILITIES FOR VALUE YEAR BY YEAR ALONG THE WAY?
Well, if we recorded the Monte Carlos right, we can show the client how any two best portfolios compare in this way too. And make this graph interactive too. This graph is important for seeing when risk of value falling too low may begin to appear, and for monitoring progress year by year along the way.
4. WHAT ABOUT THAT SHORT-TERM WOBBLE THAT CALLING IT “RISK” MAKES CLIENTS FEAR?
Instead of the fear-word “risk” and that silly questionnaire, let’s inform the client of what the year-by-year path may be with Portfolio A, and compare it with Portfolio B. For each portfolio, show ten individual simulation runs through the life of the client’s plan. Maybe show another ten. She will see what the variations may be – in the context of multi-year progress toward the goal.
5. WHAT ABOUT DAVID’S QUESTION OF HOW MANY YEARS, AND YOURS OF SENSITIVITIES TO OTHER KEY FACTORS IN THE PLAN?
We can graph that too. Show a graph with a curve of goal-meeting probabilities out to various numbers of years, out to very old. Then offer additional curves showing how different that curve would be if retirement were one or two years earlier or later, or if the annual investment or withdrawals were $X greater or smaller.
BUT WAIT. ONE MORE THING. CHANGE THE KEY ASSUMPTIONS AND GO THROUGH IT ALL AGAIN.
Even for whole major asset classes – which of course we’ll stay with for the actual investment — nobody knows with any precision what return-rate means and standard deviations will prove to be right in the future. Go back to the start, click the Conservatizer and the Uncertaintizer to change your assumptions for these and go through it all again. Look for answers that appear close to best under various assumptions.
THANK YOU, MICHAEL!
At last! – Out of the single-year Modern Portfolio Sinkhole that the Markowitz-endorsed AllocationMaster threw us into two decades ago. Out of that hole, advancing to informed pursuit of best choices for the client’s dollar plan, goals, and priorities.
Dick Purcell
Nice post Michael. We go back and forth at my firm about Monte Carlo analysis and how to use it with clients. James Montier from GMO did an amazing paper about financial models back in May that I immediately thought was easiliy applicable to financial planning modelling as well. To alter his statement, “Give a money a [Monte Carlo Simulator] and you’ve got a potential financial disaster on your hands.” He asserts that ‘all financial model underpinnings and assumptions hsould be rigorously reviewed to find their weakest lnks or the elements they deliberately ignore, as these are the most likely source of a model’s faliure.’ This post, as well as a previous post I recall about life expectancy in planning models, does just that. I find the most difficult time using financial planning software because I can’t imagine trying to put together a financial plan without the sensitivity analyses you describe, so I end up creating them myself on Excel usually, and free writing the plan to go with it. Not very efficient, but i can’t imagine putting out anything less.