21 Why does forecasting fail?
It is easy to become frustrated with forecasting. Sometimes the forecasting process seems to “fail.” Let us discuss what failure means in the context of forecasting. Many topics we picked here have already been discussed elsewhere in the book, but it is still useful to bring them together in one place.
21.1 There is no alternative to forecasting
When we say something “fails,” the question is, “compared to what?” You can only fail an exam because there is a reasonable possibility of succeeding at it. We do not say that unaided humans “fail at flying” simply because there is no expectation that humans can fly without equipment.
In this sense, forecasting does not fail, because we have no alternative. We need to plan for the future; to successfully plan, we need expectations. These expectations are forecasts, whether we call them that or not. Our expectations may have a lot of uncertainty. Maybe we believe our product could sell like crazy or not at all. Still, we must use some expectation framework to plan. Even a make-to-order production process relies on implicit forecasts, namely that customer demand and our supply chain will stay sufficiently stable for this simple process to continue operating satisfactorily. If a sudden significant spike in demand is imminent, we may change to make-to-stock, at least for part of our assortment. A crucial part of making such decisions is to form proper expectations, i.e., forecasting.
Thus, the question is not whether we should forecast or not. We have no choice. It is much more constructive to ask whether we can efficiently improve our forecasts.
21.2 What we can forecast
In principle, we can forecast anything, but how accurately? We can predict some demand patterns with a high degree of accuracy. Others elude us. Why are some demand patterns more challenging to forecast accurately than others?
A critical step in forecasting is understanding when we can forecast something accurately and when forecasts might not be better than blind chance. The forecastability (see Section 5.5) of any demand depends on several factors, including (Hyndman, 2021):
- How well we understand the factors that contribute to its variation
- How much data is available
- Whether the forecast can affect itself via feedback
- Whether the future is similar to the past
- How much natural or inexplicable variation there is
- How far into the future do we forecast
For instance, when forecasting the number of admissions to Accident and Emergency (A&E) Departments in the next two days, most factors on this list work in our favor. We know the key drivers of admissions demand. For example, admissions demand exhibits both hour-of-the-day and day-of-the-year seasonality. Hospitals generally have access to a long history of data on admissions. With the right skills, we can develop a good forecasting model linking admissions demand to key demand drivers. Our forecasts can be accurate. However, if a manager in the A&E department requires hourly forecasts for a longer horizon, producing such forecasts becomes more challenging.
In late 2019, the COVID-19 pandemic hit with several devastating effects on hospital service providers, thereby changing admission patterns. The future was no longer similar to the past in A&E departments. Consequently, historical data contained less information about the future. Forecast accuracy decreased as a result.
Forecasting a currency exchange rate is an arduous task. While lots of data is available on past currency rates, we do not understand the causal drivers of variation. Most importantly, forecasts of the future exchange rate influence the exchange rate itself via feedback: if we forecast the price of a currency to rise in the future, we buy it today and thus bid up the price. Forecasts become somewhat self-fulfilling prophecies. The forecast will affect people’s behaviors. In such situations, forecasters must be keenly aware of their limitations and impact.
The best we can do in forecasting is to capture the systematic structure or causal drivers contributing to the variation of the demand, find a forecasting model that accurately represents that structure, and then hope that demand characteristics do not change in the future. Understanding all factors contributing to the variation of demand is fundamental to producing an accurate demand forecast. Domain or business knowledge can play a crucial role.
21.3 We can’t achieve unlimited accuracy
Our forecasting accuracy is always limited (see also Section 5.5). If we want to forecast a fair coin toss, there is simply no way to get better accuracy than 50%, and the same holds in forecasting any time series. Thus, if we require a forecast to be better than what is possible, we have a problem. Every casino visitor faces this problem. We can’t forecast the number in American roulette with a success probability better than 1/38. Still, the casino pays out at a rate of 36:1 only, so our forecasts are not good enough to make money reliably.
If we know the momentum and spin at which the roulette ball was tossed, shouldn’t we be able to predict its result more accurately? Yes, and for high-stakes forecasts, collecting more and more information exactly this way makes sense. The challenge lies in determining whether one has reached the end of reasonable forecast accuracy improvements (Tim, 2017) – in other words, whether the resulting forecast accuracy improvement is worth collecting more data, which is usually expensive. In addition, we need to keep in mind that more complex models may even lead to worse forecasts (see Section 11.7). Also, don’t be surprised if the casino asks you to leave if you set up high-end physical measurement equipment around their roulette wheel. Card counting, an easy way to increase the accuracy of your forecasts in blackjack, is considered illegal in casinos, after all.
21.4 Confusing forecasts, targets, decisions and plans
You may sometimes hear statements like we need to hit the forecast, which suggests a misunderstanding between a target and a forecast. A forecast is not the same as a target. A forecast is an honest assessment of future demand based on all the past and future information available when generating the forecast. A target is an outcome that we strive to accomplish. We use targets to motivate and coordinate people.
Forecasting should be an integral part of any decision-making process, whether on an operational, tactical, or strategic level. It is easy to confuse a decision variable with a forecast, but rather, forecasts inform decisions. For instance, a government body will use the electricity demand forecast in the next 20 years to decide whether to build a new power plant. A healthcare provider may forecast the doses administrated for next month to be 60,000, but they may decide to keep 70,000 doses in the cold chain warehouse to avoid a missed opportunity. “70,000 doses” is not a forecast. It is a decision.
Plans are responses to forecasts, decisions, and targets. Planning involves determining the appropriate actions required to achieve targets, as informed by forecasts, following the decision-making process.
21.5 No systematic tracking of forecast quality
Decision-makers often claim that forecasts are not good enough. However, frequently nobody knows just how good they are because no one tracks their accuracy systematically. A forecast is rarely spot on, and we should not expect it to be. The first step in assessing a forecast’s quality is using a well-established error measure. We can track this measure over time to evaluate whether accuracy deteriorates or improves or whether some forecasts are systematically more accurate than others.
21.6 Inappropriate error measures
We should tailor our error measure to the decision the forecast is supposed to support. For example, if we use our forecast to drive replenishment, we would calculate quantile forecasts and assess these using a pinball loss. Of course, we should not evaluate a quantile loss using the MAPE. And as a matter of fact, it may be yet better to not assess forecast accuracy at all, but the resulting business outcome in terms of overstock and stockout rates, and try to get a handle on how problematic forecasts contribute to any problem here – or whether they even do so at all (Kolassa, 2023b). Thus, the forecast and its evaluation measures must all be tailored to the consuming process.
21.7 Data availability and quality
We can often improve forecasts by understanding the underlying drivers and leveraging better data. If your product is a commodity and price strongly drives demand, then models that do not include the price as a predictor will not be accurate. If you forecast demand for IT support call centers, releasing a new version of the product you are supporting will generate many calls. Include release dates in your forecast.
Data may be available, but its quality may not be sufficient (compare Section 5.3). And it may be expensive, illegal, or even impossible to obtain better quality data on predictors that would help us improve forecasts. Knowing our customers’ future plans might help us plan better, and such information sharing does happen. But we should expect our customers to ask what they get in return for providing us with this data. Information sharing will cost us beyond the technical effort to set up the data feed. In contrast, knowing our competitors’ plans might help us forecast better, but obtaining non-public information might well be illegal. Finally, knowing exactly when the first sunny weekend in spring (or the first snowstorm in winter) happens would help us enormously with stocking related products – but weather forecasts are not accurate far out enough to be helpful.
21.8 Too much judgmental intervention
Humans adjust most statistical forecasts before their use for subsequent decisions, e.g., in alignment meetings in an S&OP process. Sometimes such judgmental interventions add value, e.g., when the human knows of a driver that the forecasting system does not use. However, humans are notoriously good at seeing patterns where none exist and are prone to change the forecast to fit perceived patterns. They add noise and make the forecast worse in the process (see Sections 16.1 and 16.3).
Even judgmental adjustments made for good reasons can make things worse. For instance, when a big promotion is coming up, it may make sense to adjust the forecast upward. However, if the system has already accounted for the promotion, our adjustment may overshoot it. The same may happen if someone else has already adjusted the forecast for this promotion before us or if someone else does so again after us.
It is a good idea to test whether judgmental interventions improve forecasts by running a Forecast Value Added analysis (see Section 5.5). You can use this process to assess the value of any forecast improvement effort, e.g., by evaluating the error of one model without a predictor and a more complex model that includes the predictor, or by comparing the forecasts before and after they have been judgmentally adjusted.
21.9 Follow-on processes do not leverage forecasts
Decision-making processes can be disconnected from the forecast that should influence them. On the one hand, this may be because forecasts are judgmentally adjusted to the point when they are devoid of reality (see the section above), and are therefore ignored by the people who should in principle rely on them.
Sometimes, however, factors besides the forecast dominate the decisions. For instance, we may be ordering based on our demand forecast – but the minimum order amount is so high that we essentially always place very high orders when current stocks fall below some level. In other words, the safety stock is not determined by forecast uncertainty, but by minimum order agreed on between our suppliers and our purchasing department. In such a situation, forecast accuracy may not have any impact (within bounds). Investing a lot of resources into improving forecasts would make little sense. Instead, we should either try to change the logistical framework to be more adaptive so that our forecasts make a difference, or concentrate on other products where the forecasts matter more.
21.10 Feedback to the process being forecasted
Some forecasts feed back into the decisions that use them. For example, suppose we forecast the demand for intensive care beds during a pandemic. If our forecast shows that demand will outstrip supply, that may lead governments to impose strict lockdowns, which would, in turn, break the spread of the pandemic and reduce the need for hospital beds we are forecasting (Goodwin, 2023). Similarly, our forecasts may indicate that demand for a particular product is too low to use up our stock before the product becomes obsolete. In response, we might reduce prices or run promotions to stimulate demand, which will then hopefully come in higher than originally forecasted. This is of course not a failure of the forecast!
Key takeaways
There is no alternative to forecasting. The critical question is whether we do a good job at forecasting.
Whether forecasts are good or bad depends on the decisions we use a forecast for and the forecastability of the time series.
Forecasts can only be as good as the predictor data we feed them. There is a point of diminishing marginal returns in improving input data and models.
Forecast Value Added analysis can help us find value-adding and value-destroying steps in our process.
Assessing accuracy can be very challenging if the forecast feeds back into decisions that influence demand.