19 Leading forecasters and forecasting teams
Forecasting is perfomed by humans in teams, and both individual forecasters and forecasting teams need to have certain capabilities to succeed, which we examine in this chapter. We also address how forecasters typically tick, what they require to do a good job, and what you can expect from your forecaster.
19.1 The ideal forecaster
What attributes should you look for when building a world-class forecasting team? What are the essential traits or qualifications forecasters need? What skills should your forecasters bring to the table right from the start, and what skills can they reasonably develop over time?
If you are an aspiring forecaster with an eye to your personal development and career, how do you present yourself to your potential next employer? What are your strengths and your areas of improvement?
Figure 19.1: The ideal forecaster
Of course, one could have many different requirements for an ideal forecaster. A weather forecaster will have a different job profile than a supply chain forecaster or someone working at a central bank to forecast macroeconomic variables. Nevertheless, we believe that there are four high-level traits that every forecaster should have (see Figure 19.1, adapted from Kolassa, 2014): an understanding of statistics, programming and the business context, as well as proficiency in communication. Let’s look at all of these in turn.
Statistics
“Statistics” means formal statistical training and broader statistical thinking. Concerning the former, a forecaster should understand what a likelihood or a probability distribution represents. Classical time series algorithms like ARIMA (see Chapter 10) are not as valuable for actual forecasting as their ubiquity in forecasting textbooks might suggest, but forecasters should have a basic understanding of these methods.
Concerning broader “statistical thinking,” statistics is not a strict set of step-by-step instructions to analyze data. Instead, it is the art and the science of dealing with uncertainty. You think like a statistician if you understand that the data you observe stems from an ambiguous process involving unobserved and unknown dynamics – and so will any future data you are forecasting. Thus, the achievable forecast accuracy is inherently bounded: if you forecast the flip of a fair coin, even the best forecasting algorithm will not get you an accuracy above 50%.
Statisticians – and we are using this term broadly to include data scientists and business analysts that have internalized the statistical way of thinking – understand the influence of noise and uncertainty. The observations we use are noisy. This noise carries over to our fitted forecasting models. And the observations we want to forecast are themselves noisy. Collecting larger and larger datasets does not guarantee absolute accuracy, nor does leveraging ever more predictors (compare Section 11.7 on the bias-variance trade-off, which a forecaster must understand). Neither does throwing more money or processing power at the problem guarantee better forecasts.
Programming
Forecasting heavily uses computing, whether on a personal laptop or in the cloud. “Programming” does not mean software development but implies a broader familiarity with scientific computing. The tools a forecaster should be familiar with depend on the environment. Python is (currently) ubiquitous, and newer AI/ML methods typically come in Python frameworks. Some SQL knowledge is essential to be able to deal with databases. R is also often helpful as the language in which academic forecasters and statisticians implement new algorithms and methods. Besides these open-source tools, a forecaster must be familiar with a company’s commercial forecasting tools. Given some basic familiarity with computing languages, obtaining such proficiency is usually not too difficult. Even Microsoft Excel can still be surprisingly useful in the hands of an experienced user.
The term “Programming” here also includes “meta-programming” skills such as experience in DevOps, agile methodologies such as Scrum or Kanban, or the ability to write coherent requirements documents.
Business
A forecast is a tool to make a decision, such as capacity planning for a call center, replenishment plans for a supermarket, or even long-term strategic make-or-buy decisions. Forecasting does not happen in isolation. Understanding the business context will frequently be more critical than more profound knowledge of statistics and programming. For example, electricity demand exhibits intra-daily, intra-weekly, and intra-yearly seasonality (see Section 15.3) and depends on weather conditions, especially temperature. Domain-specific causal factors drive call center demand: support for business software will be more needed during working hours, but support for home products will exhibit (relatively) higher demand outside of working hours. Retail demand also exhibits multiple seasonalities and can be highly intermittent at SKU \(\times\) store \(\times\) day granularity. Forecasting the online channel poses entirely different challenges than forecasting brick-and-mortar demand. Understanding the business context is thus an essential factor in identifying reliable forecasting models and data requirements.
Forecasters do not need to know everything about their specific business domain, but they must be able to ask the right questions. A forecaster trained at an electricity exchange may not think about the lumpiness of demand in retail forecasting because it is simply outside their experience. Of course, diverse prior backgrounds may yield creative approaches to solving problems, but every forecaster must have a fundamental understanding of their business.
Communication
Finally, communication skills are indispensable for forecasters beyond statistical, technical, and domain knowledge. They will frequently communicate in very different circumstances and with highly diverse stakeholders. One day, they may need to draft and discuss a requirements document for a software developer. The forecaster may meet with an external forecasting software provider in the afternoon. The next day, they may need to explain an anomalous or disappointing forecast to a business user. The day after, they will need to explain to senior management what steps they plan on taking to make sure a disappointing forecast like this will not happen again – ideally using precisely the right amount of technical/statistical jargon to impress management with their competence, but not so much as to come across as out of touch with day-to-day business realities (compare the “business knowledge” requirement for a forecaster as discussed above).
Forecasters will need to mediate between different viewpoints. Sometimes the “statistically correct” approach would be prohibitively expensive. And conversely, a costly method may still be cost-effective. Business users are often confident that including specific predictors will absolutely improve the forecasts. The forecaster may need to explain that this is not necessarily the case (again, compare Section 11.7 on the bias-variance trade-off). Sometimes senior management is enamored with the newest and flashiest machine learning technology (possibly aided and abetted by outside vendors and consultants). The forecaster may need to bring them back to earth gently. The ability to connect emotionally to stakeholders, communicate effectively and with empathy, and understand tailoring their communication approach to the audience will ensure that the forecasting function works well and is respected within the business.
19.2 How do forecasters tick?
One can enter a forecasting function through three main career tracks, and your forecaster’s background will influence their mindset. The most common backgrounds are:
- Information Technology (IT)
- Business
- Data Science or Business Analytics
A few years back, before the advent of modern Data Science and Business Analytics, most forecasters came from a background in either IT or Business. Many, if not most, of these forecasters were self-trained. This state of affairs has recently changed, with an explosion of educational possibilities in Data Science and Business Analytics. Nowadays, if you hire a college graduate for a forecasting position, your applicants often come with such a background. And any applicant shifting from IT or Business will have likely gone through several online “Data Science boot camp” offerings. Roughly speaking, Data Science programs tend to emphasize the skills we summarized in Figure 19.1 under “Statistics” and “Programming”. Business Analytics programs tend to de-emphasize these topics in favor of more instruction in aspects that would fall under “Business” in Figure 19.1.
What difference does your forecaster’s background make in how they see the world?
- An IT person will be most familiar with the IT aspects of forecasting. They will think about data storage requirements, database setups, access latencies, runtimes, cloud vs. on-premise computations, data cleansing and preparation pipelines, etc.
- Someone who came to forecasting from the Business side of things will think in terms of what the data means. Is the behavior they observe in a time series normal or abnormal? Does it reflect new developments in the marketplace or issues with the supply chain? Can they deduce your salespeople’s efforts from observed demands, and how can they take these into account when forecasting? “Business” forecasters are usually best able to understand and apply the differences between forecasting, targets, and plans.
- Data Scientists or Business Analysts usually have a solid grounding in the IT aspects of forecasting and almost as much understanding of the statistical foundations. Naturally, their IT knowledge will not be as deep as that of an IT veteran. Still, if you compare a Data Science or Business Analytics graduate with a Computer Science graduate, your Data Scientist/Business Analyst will usually be less deep but wider in their background. For a forecasting function, this is a sweet spot. Where Data Scientists are naturally usually weak – at least at first – is the business background. The better Data Scientists recognize this and are open to learning. Modern Business Analytics programs combine Data Science with a thorough understanding of business, making these candidates predestined for business forecasting functions.
Ideally, your forecasting team should contain diverse people with many different backgrounds, so they can learn from each other and combine their strengths. Forecasting is interdisciplinary; anyone with tunnel vision will be less effective.
Thinking about the background of your forecaster provides clues as to what may be vital for them. Few people radically change their outlook, starting as a marketing major and shifting to statistics and time series analysis in mid-career. Nevertheless, for someone to move into a forecasting position (and stay there) indicates several characteristics: they will be quantitatively oriented, be comfortable with ambiguity, and be at least somewhat interested in the business background.
19.3 Making your forecaster happy
On the one hand, the previous section should have given you a few ideas on what may be valuable to your forecaster. Make them happy by making sure they get the resources they need. Examples of “a forecaster’s favorite resources” are:
Up-to-date, clean, and usable data. Depending on your specific situation, investing some effort in cleaning up data or collecting (or buying!) additional data may yield good returns regarding improved forecast accuracy. Yes, this may require cooperation from IT functions outside the forecasting team. Getting this cooperation is one instance where a forecasting team’s manager can add value.
Computing resources. Chances are that your IT department is already budgeting for enough data storage space. Still, forecasting methods, especially the more modern ones, are data- and processing-cycle-hungry. Getting your forecaster enough computing resources may be a question of getting them a sufficiently powerful laptop – or an Amazon Web Services or Microsoft Azure subscription.
Tools. Many forecasting tools, like R or Python, are free. But others aren’t. Investing in a dedicated forecasting solution may be a worthwhile investment. Or not. Analyze the decision.
Training. Forecasting, and Data Science in general, is a quickly evolving field. Without ongoing learning and training, the skillset of a forecaster will become obsolete quickly. Training may cost time and money. Staying current also requires the freedom to try out new ideas on existing data.
Interaction. There are typically few forecasters in any given company, so your forecaster(s) may be interested in interacting with data scientists and forecasters from the outside. Such communications can infuse fresh thinking or give your forecaster a little recognition (which forecasters frequently do not get in-house, where good forecasts are rarely appreciated, and bad forecasts are remembered forever). Perhaps your forecaster would be interested in participating in Kaggle competitions, and you could support this with company time and computing resources. Or maybe they could disguise some of your data and organize a data science competition at Kaggle?
On the other hand, your forecaster may also need to consider development that is outside their comfort zone. They may be motivated to learn the newest Python module relevant to forecasting. But it may be much more helpful for them to attend a communications or presentation training or to shadow a business user for a week to understand what people actually do with their forecasts.
19.4 What can you expect from your forecaster?
If you take good care of your forecaster, you can expect them to take good care of your data. They will forecast what you need and explain their forecasts to the extent possible. (Machine learning methods are hard to explain, especially to a non-technical audience.) After all, forecasts that are not understood may not be trusted, and forecasts that are not trusted may not be used, so people may resort to making up their own forecasts. An organization that cannot agree on a common forecast is at a disadvantage; this is an outcome anyone near a dedicated forecasting function should avoid.
Your forecaster should also be able to dig into bad forecasts and at least make educated guesses as to why they were bad. Maybe the future differed sharply from the past. Or maybe there was a problem with the input data. If so, your forecaster should suggest possible improvements.
Your forecaster should also follow recent developments in the forecasting world. They do not need to participate in Kaggle competitions (although encourage them to do so!). Still, they should have an informed opinion of the methods used there and whether trying them on your data would be worthwhile. The answer to the question, “should we try a Gradient Boosting Machine to forecast our data?” may be, “our input data is so bad that applying cutting-edge tools to it will not be helpful – better to work on data quality first.”
Your forecaster should provide meaningful input in any discussions with outside vendors of forecasting solutions. Of course, such a situation is always hard to navigate: is the forecaster afraid of becoming redundant or looking forward to getting new tools that will enable them to do their job more efficiently? Do they have the incentive to badmouth the external vendor or to argue for the solution with the bells and whistles that most caters to your forecaster’s inner child? Nevertheless, your forecaster should be the person in the organization best qualified to understand and evaluate a vendor’s claims, at least in forecasting performance, so listen closely to what your forecaster is telling you.
Conversely, here are two expectations you should not have of your forecaster:
- As we discuss elsewhere in this book, forecastability is always limited. Your forecaster cannot be expected to hit an arbitrary accuracy target. Telling your forecaster that “the industry standard” is a MAPE of 20%, so their bonus is contingent on them reaching 18% without regard to your data’s specificities will only frustrate them (Kolassa, 2008). The same holds if you take this year’s accuracy and require next year’s accuracy to improve on that. It may be possible – but posing this requirement blindly risks requiring your forecaster to forecast a fair coin with 60% accuracy.
- Similarly, there is frequently some wishful thinking that a new forecasting method – typically one involving Machine Learning – will yield significant advances in accuracy. Forecasters are often under pressure to wring precisely this accuracy improvement out of a new tool. Again, this may be feasible, or it may not be. You may very well have reached the limits of forecastability of your time series.
Key takeaways
- Forecasting, like all data science, requires competency in four different domains:
- Statistics
- Programming/IT
- Business
- Communication
- Forecasters who deeply understand all four dimensions are as rare as the proverbial unicorns. However, any well-rounded forecaster should be encouraged to develop in all four directions.
- Your forecaster will likely have one of three different backgrounds:
- IT
- Business
- Data Science/Business Analytics
- Understanding your forecaster’s background will help you understand through what kind of lens they perceive the world, what domains they may tend to favor in their development, and where learning outside their comfort zone may be helpful.
- You can expect your forecasters to create sound forecasts, explain them to non-technical users, and help leverage them for maximum business benefit.
- You should not expect your forecaster to hit arbitrary accuracy targets or to work wonders just because they got a new forecasting tool.