Financial advisors often discuss how futile it is to try and predict what will happen in the stock market, and yet the reality is that a core component of creating a financial plan involves making a prediction for how markets will behave, at least in the long run. An assumed rate of return is needed, for example, to illustrate how much a client might need to save for retirement, how much savings they'll have when they do retire, and how long their savings will last after they stop working. Which means that advisors need some way to create a reasonable return assumption, even as just a starting 'average' scenario around which the advisor can build other scenarios to illustrate better or worse cases.
One way to predict future market returns is to use the past as a guide, with nearly 100 years of continuous market data since the 1920s covering a wide range of economic conditions showing that, for instance, the S&P 500 has had nearly a 10% average compounding return (7% above inflation) since 1926. And yet the problem with using that long-term average return is that investors generally don't have 100 years to invest – their time horizons are much shorter, usually spanning a few decades at most, and in those instances the conditions at the beginning of the time period (namely the market valuation compared to the underlying business and economic conditions at the time) can significantly impact the actual returns experienced over that period.
The Shilller Cyclically Adjusted Price-to-Earnings (CAPE 10) Ratio is one example that takes into account current market valuations versus company earnings, generally predicting that the higher the valuation at the beginning of a period the lower the expected return for that period. Which, at today's current high valuations, suggests that, for instance, the S&P 500 might 'only' return just over 5% over the next 10 years (as opposed to the 10% that it has averaged over the last nearly 100 years). Additionally, when breaking down current conditions even further to factor in the impact of the returns on cash, dividend yields, expected tax rates, and corporate leverage, U.S. equities could have a very difficult time trying to match or exceed the average returns of the past (and would need a tech bubble-like increase in valuations to come near the market's performance of the last decade)!
The key point is that when projecting future market returns, simply using historical averages without factoring in current evaluations risks painting an overly optimistic return scenario that could lead to under-saving for retirement (which could be exacerbated by a combination of higher life expectancies, increasing long-term care costs, and governmental funding difficulties around Social Security and Medicare). The good news, however, is that there are ways for advisors to address the current issue of high U.S. equity valuations, including diversification with international stocks (which have much lower valuations) to help mitigate the risk of the S&P 500 failing to repeat its past performance in the future, the use of value stocks, and alternative investments.
