An interview with the search giant's chief measurements strategist

May 4, 2022

14 Min Read

An interview with the search giant’s chief measurements strategist

A digital marketer walks into a bar and asks the first person he sees to marry him. If the answer is no, he will keep asking hundreds or even thousands of people and follow them around until someone eventually says yes. It sounds crazy, but that is exactly what companies do in digital marketing, according to Neil Hoyne, Google’s chief measurement strategist.

“The marketers give themselves one moment, one opportunity to drive a result, and they treat every interaction the same,” he writes in his personal capacity in the book, “Converted.” But this is short-term thinking that does not lead to lasting relationships with a company’s most valuable customers. People are not just data on a spreadsheet; they are thinking beings whose behaviors change.

At Google, Hoyne led more than 2,500 engagements with major companies to oversee data and marketing initiatives, acquiring millions of customers and quintupling conversion rates.

Mukul Pandya, who was senior fellow of Wharton’s AI for Business initiative and former editor-in-chief of management journal Knowledge@Wharton, recently interviewed Hoyne for AI Business. What follows is an edited version of their conversation.

Figure 5: 8493.jpg

Mukul Pandya: What inspired you to write “Converted”? Did you see any limitations in existing books about digital marketing and what gap were you trying to fill?

Neil Hoyne: The first limitation in existing books about digital marketing is simply that they're short-term, transaction-focused on immediate ROI. This is the single argument that carries forward in the entire book: Short-term thinking is wrong. You want to build relationships. Long-term thinking is not only a better and more successful way to build customer relationships. It's also more profitable, and the data supports it.

A lot of people say, ‘this is how we've been doing digital marketing for the past 20 years.’ I say, ‘here's a book that's telling you stop. This is a better way to do it, and a better way to look at it.’

Pandya: Many brands seem to believe that the solution to their problems lies in a four-letter word: data. But having more data does not necessarily mean that it will be used well. How can companies learn how to use data better to convert casual browsers into long-term customers that continue to provide value over their lifetime? 

Hoyne: I think that companies put a disproportionate amount of emphasis on collecting data and the systems that support that collection.

Companies seem to believe that data is the new oil and therefore the more they can capture, eventually they will be able to convert it to some type of value. And that leads to a dramatic underinvestment in the second part of the equation, which is, what do we do with all this data?

And so generally you find these under-funded, under-led data scientists who are given this broad problem, ‘Go find this value,’ without leaders translating it to say, ‘What is the business goal? What is the business objective?’

They're not seeing their data applied to anything and that's because for most leaders that are asking that question, they're not necessarily taking a step back and asking: What is it that you're hoping to get out of that data? What are those business questions you're trying to answer?

Pandya: As more brands invest in AI and analytics solutions, how is this changing the kind of insights they can capture about their customers? 

Hoyne: In my experience, a lot of companies pursuing AI are doing so with the idea that we need this tool more than we have this business question. They're worried about falling behind in the race for AI and AI competency, that they're not necessarily sure what they're supposed to pursue. … They believe all their competitors and all these disruptors are chasing after this technology. And so they just go blindly invest in AI … not thinking of what the end goal is.

For those companies that are already using it and they're often using it carefully, they're going back to what the essence of analytics and AI is supposed to be, which is understanding, almost pattern matching, ‘who are our high-value customers in this case? What are they doing? How are they behaving? What are they interacting with on our site?’ … You see, they’re starting with the questions – what are our high-value customers doing that we’re missing? ...

Figure 1: 3029.jpg

The other component of AI which is fairly fascinating is they're using it to build a scalable process knowing that these behaviors of high- or low-value customers are not consistent. (That is, people’s behaviors change.) As they get involved in the conversation (with consumers), and as they change their response to the messages of the consumers, the signals will change.

I talked about this in the book: An automotive manufacturer realized that people who were looking at financing information for their vehicles was a positive signal indicating interested intent. But after they recognized that signal, they optimized so that everything drove customers in the funnel to that financing page − and all of a sudden, that signal became weaker (due to financing ads overload), and other new signals rose up (to help them decipher customer intent).

The only way to do that in a scalable way, given all the data and signals we're collecting, is really to use artificial intelligence, machine learning. Take over that human component of analyzing the data to see what's going on here and doing this in a repeatable, scalable way.

Pandya: How can brands become better at using AI for mass personalization at scale?

Hoyne: It just depends on the objective. … If you’re focused on short-term transactions, as many companies are, I equate it to the equivalent of marketers going into a bar and immediately proposing to someone. You just have that one opportunity to win a customer over, there's not a lot of opportunity to personalize anything. You don't have a lot of data to work from because you're just looking at that interaction, and you don't have a lot of conversation by which you can learn and change your response.

The most you can do for personalization is to say, ‘maybe I'll change my message. Based on this little data that I gathered … maybe I'll tweak the website terms, I'll tweak my landing page copy or my image or my call to action. But it's still just really rough information because you're just focused on that narrow, individual moment of time.

And when we really talk about personalization, we need to broaden our lens a little bit to say what do we know about these customers? And what are their actual needs? How do we deliver them − but then (not feeling pressured) to get them to convert every time?

Here's an example. Oftentimes, when I will book hotels, I'll use one of those travel aggregators such as Booking.com, Hotels.com. If they're focused only on short-term transactions, their goal is every time I go to the site, they will try to get me to book a hotel. That's where their investments and marketing messages go. That's where their call to action goes. But when I look at my relationship with them, it actually changes − it goes from searching for a hotel room to booking a hotel room, to needing support or information about my next reservation to make managing my travel easier. But oftentimes in their mind, they'll say, ‘when you come back, our only goal is that moment, how do we get you to buy another hotel (room)?’

They’ll under-invest in that personalization to improve the customer experience because in their mind, it doesn't immediately lead to any more incremental reservations. But as they take a step back and look at the lifetime value, for instance, of customers that they service as well, that they provide ancillary benefits to, they would probably see that those customers stick around longer and probably spend a little bit more money.

Figure 2: 1929.jpg

Pandya: You have worked with thousands of companies on data-driven marketing projects involving AI and analytics. What are the most common mistakes you have seen them make? What advice do you generally offer to avoid those mistakes? 

Hoyne: I don't think enough people know what the business objectives are that they're trying to solve. They start with the technology: We need to be in AI. We need to be in machine learning. We need to have a bigger data presence. … That's the first gap − I don't think there's enough focus on the business questions. (It’s not) gross negligence by a lot of executives as much as it is lack of awareness. When they think about machine learning and AI (they’ll say) ‘it sounds very complicated. How can I possibly have a role?’ There's a certain amount of confidence that executives need to have to be part of that conversation. …

There's other mistakes too. Companies are unsure as to what level of transparency they need. Do we need to understand how these models work? How this data works? Do we have biased data? For instance, these machine learning models can only look in the cases of marketing and things you've done in the past. They can't forecast datasets and customer bases outside the people you've traditionally tried to reach. And sometimes that comes in incomplete datasets.

So does our data set need to be perfect? No. But what is good enough for us as an organization? And perhaps tying to that is probably the greatest unanswered question in most organizations − what level of proof is necessary for your business to move forward?

We talked about these transformational technologies like AI. (But) in transformative processes and strategies, there's often a significant reallocation of company resources. You’ll say, ‘we were going in this direction, but based on this new insight and this data scientist, we should go here’ − whether it's building a new product, going into a new market or moving away from a marketing channel.

When you realign resources, you have winners − people who now have more resources and their projects are seeing higher priority − and you have losers, people whose projects are not as important as they were to the organization the day before. Those people (on the losing side) can use that gray area to say these models are imprecise, the data is inaccurate. Is this really the direction we want to go? I don't think they have enough discussions internally about the burden of proof they need before making a decision.

So what a lot of companies do is they wait until the results come in under this misguided idea that the data will decide. But data is never that binary − it's never go this way, don't go that way. It's always ‘we're leaning in this direction. But it's not 100%.’ And that wiggle room sometimes is just enough to create organizational deadlock, where companies will argue about those issues. And so those are just some of the mistakes that they make.

Figure 3: 1523.jpg

Pandya: What questions should brands be asking to identify their most profitable customers? 

Hoyne: A vital step is calculating (a customer’s) lifetime value (CLV), and then going through your data, possibly with the help of ML and AI, to find out why these customers are valuable. Is it the channel they came through? The time of year we acquired them? A sales rep they spoke to? What made their journey unique and special in a way that we can replicate that success in hopes of finding higher value people?

But the conversation doesn't simply have to be what you observed. Some companies ask about the Net Promoter Score − how likely are you to recommend our firm to a friend, knowing that the result could correlate to higher lifetime value customers? I say ‘could’ because it sometimes does but not always. In other cases, you may just want to ask share of wallet. If you have two customers that are worth the same lifetime value on paper, but one you have 90% of their spend the other one you only have 10%, you may prioritize differently.

Or what about new customers coming in? Let's use the example of an airline. I'm going to be flying on United Airlines for the first time in about four years. Now, if they look at their historical records, they're going to see that Neil spent close to zero. And so (how would they value me? They can say,) ‘you deserve a seat all the way at the back of the plane.’

But it wasn't my fault that I wasn't flying. It was part of this nasty lockdown and before then I just wasn't traveling because my wife and I had a new child. But when I do travel, I travel a lot, often 100,000 miles or more a year. If United knew that, would they make different decisions about how they would treat me as a passenger? Would I get a slightly better seat at the front of the plane? Would they let me board early? Would they give me an upgrade?

If they knew (that), what will likely happen is they'll simply recognize the people who've already been spending and who’ve demonstrated that loyalty … as opposed to saying ‘let's use this ability (to offer perks) to reach out and find and acquire new customers.’

Figure 4: 7155.jpg

Pandya: If CEOs of brands were to ask you where they should start to apply the lessons of your book and make smarter use of AI and analytics to build customer lifetime value, what would you recommend they do? 

Hoyne: Step number one is just to acknowledge that you probably don't need to buy more big data systems. You have all the data. Step two is to calculate (customer) lifetime value. The techniques are already available and proven. You likely have data scientists already who can perfect the models if there are weaknesses.

The third step though, is not direct application. It's not spending more money against it. It's actually making that (CLV) metric front and center alongside all of your existing KPIs and metrics. One of the things that I'm aware of is that companies have already been successful using their existing processes and data. There's no need to unseat or tear them out and replace them with something else.

But what you do want to do is at least make (CLV) visible. So that if you're still measuring, for instance, the short-term number of leads that you brought in, segment those groups also by lifetime value. We may have brought in 10,000 customers this quarter, as we do every quarter, but 90% of those customers aren't going to come back. People in the organization naturally become curious: Why is that? What brought in the better customers? What's bringing in the worst customers? What does this mean to our future in these products? … Who should we build to? Who should we focus on?

Simply having that understanding, having that metric even if you're not incentivizing people on it, even if you don't have a specific plan of action, encourages people to discuss and to understand how (customers) change course. … Simply having the awareness that their customers are behaving differently is enough to encourage change, to encourage curiosity. It comes full circle where they start seeking out more information.

Keep up with the ever-evolving AI landscape
Unlock exclusive AI content by subscribing to our newsletter!!

You May Also Like