Predictive models apply statistical methods to data to make predictions. Predictions, such as how likely a customer will make a purchase, or the number of ICU beds needed as a novel lethal disease spreads in the community.
While your brain is essentially a prediction machine, automated predictive models have several advantages: they can be more accurate, less biased, more consistent, and they don’t get tired.
You probably see work in your organization that could benefit from using automated predictive models. Opportunities for improving what are today manual, time consuming or not very targeted processes.
In this article we lay out how you can help your teams accelerate those efforts. Not by giving them more tools or analytics resources to work with, but by clarifying how to think about predictive models and how to use their outputs.
At the heart of this is a mindset shift – being less wrong is more important than being right.
How your brain forecasts
If you don’t use sophisticated predictive models, how do you make assumptions about the future? Suppose you have to come up with 3-month sales forecast. How will you go about it?
You will look for reference points first. If you are in a seasonal business, you may consider last year’s sales for the months you are forecasting for and the year-over-year changes for previous months. If your business is all about new product launches and product lifecycles, you may consider sequential growth rates over the course of the lifecycle.
Whatever your reference points are, you will use those to create a baseline forecast number for the coming months. Then you will use any additional information you have to adjust that baseline. You consider supplier inventory constraints, competitors’ product launches, additional marketing activities etc. But overall, you settle on one number. Based on a couple of reference points and then adjusting it up or down.
How predictive models forecast
Naturally, if you are the person who came up with this informed estimate, the first thing you will want out of a predictive model is that number. It’s actionable. You can tell your procurement person how much to buy. It’s aligned with or can inform the marketing activities. The math is straight forward. You want the model to replace your thought process above, and ideally come up with a more accurate number.
Predictive models don’t give you a number or one answer. We will go into more detail on this below. They couch predictions with probabilities. There’s an X% chance Y will happen. You will sell between X and Y units over the next 3 months. They will give you a range of likely outcomes.
You, however, have decisions to make. You want to know what specific assumption about the future you should base it on. Tell me how many units of product A we expect to sell over the next 3 months, so that I can place an order with our supplier. Tell me if getting on a plane is too risky if I don’t want to get sick. It can be maddening not to have clear answers.
If predictive models don’t have an answer, then how do you get value out of them?
The cost of being wrong
To answer that question, let’s look at a different scenario. Let’s assume a deadly respiratory virus is spreading rapidly in your community. You are planning the number of ICU beds you need to have available across your hospitals. You turn to your public health agency to get a forecast of severe cases in your area.
They can give you a prediction for: (1) The expected number of severe cases (2) the highest and the lowest number of severe cases you should expect with 90% certainty. Which one do you choose?
If you are a somewhat compassionate person, you will choose #2. You know there is no way to make an accurate prediction of the number. “Expected” means it’s an average of likely outcomes. There’s still a decent chance the actual numbers will come in above or below.
Having too few beds means patients will likely die as they cannot be cared for. The value of a human life is obviously extremely high. Economists and attorneys even attach a number to it, of usually several million dollars. The cost of having an empty ICU bed is only about $5,000 per day. You can afford to be overestimating the need for ICU beds, but you can’t afford underestimating.
A range of likely outcomes
Predictive models give you the range of likely outcomes. If they are good models, they will almost always be right in that the actual outcome will be somewhere between their lowest and highest prediction. In our case above, the predictive model would tell you that there’s a 90% chance that by day X you will need to care for at least 80 severely sick patients but no more than 140.
Let’s say you planned for the worst case, but the low-end prediction comes true. You will lose $5,000 * 60 beds = $300,000 per day due to over-capacity, but no lives are lost.
Here’s the point: Knowing the range and probabilities of all likely outcomes is hugely valuable, if there are different cost or consequences to under- versus overestimating. The latter is the case in most real-life situation. “Just give me the mid-point forecast” is hardly ever the number of interest or the number you want to manage to.
Plan for the mid-point, 110 beds, and you have a 50% likelihood to lose money due to over-capacity (max $150,000 per day), but also an equal likelihood to lose between 1 and 30 lives. If a life is worth more than an empty bed for two weeks, you would never consider this scenario.
The value of predictive models
So, what’s the value of predictive models?
A forecast range, also known as a prediction interval, lets you assess up front the consequences of your actions. You already know you will likely be wrong. The actual outcome will be different from whatever number you chose to act on. Or, if the outcome is more discrete (e.g., is the customer actually the type of customer I’ve pegged them to be?) you will be wrong in a number of cases.
However, if you know (1) the range of likely outcomes and (2) the cost of being wrong, you can determine the cost of being off the mark for any outcome you choose to plan for. And if you know the cost of any choice, this means you can select the best option to move forward with – the one that minimizes the overall cost of being wrong.
You can apply the same idea to forecasting sales of widgets (if you overestimate demand, your cost will be the cost of liquidating excess inventory; if you underestimate demand, it’s the missed margin), targeting customers with special offers, determining resource needs or really any action that requires making predictions about the future.
In short, this is the value of predictive models: Certainty about all the outcomes that can happen. And a clarity about the best action to take, if you understand the cost of being wrong.
Changing the mindset
If you contrast this idea with the approach we typically take to forecasting without a model, the key to putting predictive models to work is a shift in mindset. A different way of thinking about how we make assumptions about the future and act on them.
How do you lay the groundwork for this shift in thinking? Here are a few ideas:
- When projecting the benefits of a project, promotion or process change, start asking for best case and worst-case scenarios. Ask people to explain their rationale for coming up with the low-end or high-end estimates. Frame these thresholds as a very small chance (5%) the outcome will be worse or will be better than projected. Ideally put those estimates and the rationale in writing.
- Discuss the trade-offs. Have people do a back of the envelope calculation of the financial impact of the actual outcome being lower or higher than what they are considering acting upon. E.g., if you buy more impressions and drive more traffic for a particular product than you have inventory for, what’s the harm? What if you don’t drive enough traffic and you are not selling through the product’s inventory? Finding out how the cost differs on each side can inform whether to prepare for a more optimistic or pessimistic scenario.
- Have your teams understand the variance of key metrics they use. By how much do conversion rates typically vary day-to-day? How much lift have different types of promotions generated in the past? How has the bounce rate for campaign landing pages varied from campaign to campaign? What’s the ratio between new and existing customers that respond to certain types of offers or campaigns?You will see that they start to talk about future expectations differently. What’s normal or expected is being within a certain range. Over or underperformance is outside of that range. Your teams now also have data to inform worst-case and best-case estimates discussed earlier.
- Ask for reality checks on a project’s, promotion’s or process change’s break-even point. This number, as a decision criterion, is only helpful if you have the context of how likely that outcome is. If you only cover the cost of your website redesign initiative if you can increase your conversion rate from 3% to 6%, but 6% is outside of the range that most comparable retailers can achieve – it may not be worthwhile pursuing it at the planned cost. Making sure you get a range of likely outcomes for whatever initiative you are looking to evaluate helps you assess the risk you are taking on.
Your business and operations teams
Business, operations and other functional teams, who are ultimately accountable for their decisions, are in the prediction business. Any decision is made with assumptions about the future.
With predictive models or without – looking at forecast ranges or prediction intervals means you are approaching decisions with a view of all likely outcomes. This idea by itself can help you avoid making decisions with implicit bias towards desirable but not very likely outcomes.
Understanding the cost of being wrong helps you choose the best action. This is critical to turning outputs of predictive models into decisions. No model or algorithm will give you specific recommendations. That is unless you give it a way to choose the best action considering a range of likely outcomes.
Your domain experts and business analysts are best positioned to assess the consequences and cost of being wrong. That’s why it’s critical for business and operations teams to understand these ideas for putting predictive models to work. Additionally, it empowers them to direct model improvements towards applications with the greatest benefit – those applications where the overall cost of being wrong can be reduced substantially.
We can help
if you need help getting your teams to embrace advanced analytics to improve and scale processes. You can learn more about our on-the-job training and coaching here.
