9 min read

What is your intuition? Do those analysis results look too good to be true? Do they just not make sense? Do the findings contradict another piece of data you learned about recently?

Whenever you have those doubts, you may want to go back to this basic question: what did you measure? How did you know the data you chose were a good representation of the real-world phenomenon you wanted to assess? This article is about finding representative measures.[1]The other angle of representativeness is whether the group of people, products, situations etc. you are measuring is expected to be very similar to the group you want to assess or make decisions for. … Continue reading

Consider this example from health care:

Who is eligible for high-risk care programs?

In many communities, health providers, such as hospitals, physicians and outpatient service providers, formally work together as a “health system.” Health systems often enroll patients with complex health needs in high-risk care programs.

These programs help patients better manage their health needs. Additionally, they have been shown to reduce the number of emergency room visits and hospitalizations, therefore saving costs.

But who decides if a patient has complex health needs? The primary care physician? What if the patient does not have one? What if that doctor doesn’t have access to the patient’s health history across providers and therefore lacks the bigger picture?

How do you measure health needs?

Health insurance companies come closest to having a complete picture. They have your entire claims history. Using this data, you can develop an algorithm that will identify the health conditions, doctor visitation patterns etc. that point to severe health problems. Once a patient shows this pattern, you can auto-enroll them in the high-risk care program.

However, to develop this algorithm you must describe “severe health problems” or “health needs”, the real-world phenomenon you want the algorithm to identify, in a language it can understand. You need a good data representation or good measure for health needs.

In this case, the algorithm’s developers chose to use health care cost. Why? It is easy to obtain since insurance companies know what they paid for treatments. Also, unsurprisingly, when you see high healthcare bills, you find that these patients indeed have poor health.

The wrong measure

What sounds like a logical and elegant solution at first, turned out to be a problem.[2]You can find a summary of the study here, including the reference to the actual study: https://review.chicagobooth.edu/economics/2019/article/how-racial-bias-infected-major-health-care-algorithm The algorithm’s recommendations led to a disproportionately lower enrollment of Black people with evidently complex health needs.

How? Many health care systems have historically spent relatively less money on the same health conditions when exhibited by Black people versus white people. As such, health care cost, the measure, turned out to be a poor global representation of actual health needs, the real-world situation they tried to assess.

When a measure is representative it means it moves in a predictable direction as the real-world behavior changes. Health needs grow, health care cost grows.

There is a pattern you can see if you graphed the data. And, importantly, the pattern holds for the entire group of people, products, events or whatever you want to draw conclusions about. On the latter point health care cost as an indicator of health needs failed.

Clarify ambiguous measures

Let’s look at some popular measures from different domains and their technical definitions:

Measure Definition
Customer Retention Rate % of customers who still shop or use your service a certain time span after they made their first purchase
Product Purchase Rate % of people who purchased a product after showing an interest in a product or service, e.g., by looking at it on your website
Graduation Rate % of students who formally completed their program out of those who were enrolled

What real-world behaviors, attitudes or circumstances would a change in these measures reflect? You don’t want to understand these measures for their own sake, but as an indicator of something you hope to affect.

Before you move on, take a minute to jot down a few options: an increase in customer retention rate may reflect what? What about an increase in Product Purchase Rate? Graduation Rate?

Here are some interpretations you may consider:

Table showing possible interpretations of different measures

Questioning and choosing the right, representative measures is a skill and practice your decision makers at all levels can develop. The examples show why you shouldn’t leave this to analysts or data scientists to decide.

It’s domain experts who understand how their world works. They can connect people’s behaviors, market dynamics, educational setting as relevant to our examples above, with how these should or may show in your data.

Here’s how your teams can develop those skills:

Find the right measures

  • Getting this right starts upstream. You need to be clear about the question you want to answer and the decision you want to take.Let’s say you are thinking about creating a referral program. What exactly are you hoping to achieve?Do you expect referrals will lower the cost of acquiring new customers? Get more customers that couldn’t have been reached otherwise? Or do you believe that customers acquired through a referral will be better customers in the long run?Which measures you will pick will differ depending on the question you are after.
  • Often, perceived urgency gets the better of us. You settle for a measure that may have flaws to arrive at data-backed answer quickly.Consider though what some additional analysis time may be worth. Write down what’s at stake if your organization gets the decision or assessment wrong.Using the referral program example, here’s what you may want to explore: Even if the program does not achieve its goals, how long will it likely go on for?As a result, how much money could you lose before you would pull the plug? How much will you forgo if you delay the decision for another couple of days, weeks or months?How confident are you in various possible outcomes right now? By how much could additional analysis shift your confidence levels?Likely you can’t answer those questions with high precision. But even estimates should allow you to do some math that makes it obvious if you should invest more analysis time. Time that you can use to determine which measures to use and to assess if they are valid.
  • As you develop your data and analysis approach, ask what other phenomena could affect the measures you are considering? Get an outside perspective. Find another domain expert and ask them this same question.What assumptions do you need to confirm to ensure the data you are working with is or is not a good representation? If you chose a revenue metric for the referral program, what other reasons for higher revenue among referred customers could you think of?
  • Think about which alternative data or metrics you could use to answer your original question. By forcing people to think about at least one alternative, you avoid going with the data that’s easiest to obtain.If your goal is to maximize future customer value – Would revenue or the number of purchases be a better indicator of future customer value?If you are after incremental customers, how would you know if the customer wouldn’t have purchased anyhow, referral program or not? You could see if the customer had visited your store or site previously or you could survey the customer’s knowledge or pre-disposition towards your product or service shortly after purchase.
  • Finally, be ready for some nuance. As you look at your analysis results, you may find an answer to your original question that only holds under certain circumstances.For example, let’s say you wanted to know if service call quality matters for customer retention.You measure the service call quality via a follow-up survey and retention by whether the customer returned to shop within x weeks. It turns out there is no relationship between the two measures. Case closed or poorly chosen measures?It may be neither. Breaking the analysis results down by customer lifecycle stages may show that there is a relationship between service call quality and retention, but only for customers who haven’t made many purchases yet.

Measure twice, cut once!

Good carpenters know the value of planning their work. Estimating the cost of getting a decision wrong and assessing your level of confidence can help you decide if more planning time is needed.

Next, ask questions: How do you know you are using the right measure? What else must be true or not true if we choose this measure?

Ultimately, you want your teams to develop data-backed insights that hold up under scrutiny and yield good decisions. Decisions you feel confident about.

We can help

If you need help upskilling your teams in how they work with data, contact us. We’d be happy to brainstorm with you how and where greater data acumen could yield immediate and tangible benefits in your organization.

Not familiar with Data Brave? Check out examples of how we’ve helped other organizations.

References

References
↑1 The other angle of representativeness is whether the group of people, products, situations etc. you are measuring is expected to be very similar to the group you want to assess or make decisions for. For example, if you use Net Promoter Scores to measure customer loyalty, you need to decide a) if answering the question of “on a scale from 0 to 10, how likely to recommend this product/service/company to a friend/colleague?” is a good measure of loyalty and b) if the people who respond to this question or survey are similar to the people you are looking to assess. A) is a question of the right measure, b) is a question of the right group you chose to collect data for. Both are critical for getting any use out of your data or analysis.
↑2 You can find a summary of the study here, including the reference to the actual study: https://review.chicagobooth.edu/economics/2019/article/how-racial-bias-infected-major-health-care-algorithm