Our new Chief Data Scientist explores why historical data isn’t always as informative as you think, why some experiments are more useful than others, and why he sees LiftLab as exceptional.
Dr. Dirk Beyer, PhD has been a leading data science innovator at companies including DoorDash, Uber, and Neustar (now part of TransUnion). He’s recently joined LiftLab as our Chief Data Scientist. As a follow-up to our announcement of his joining the company, we sat down with Dr. Beyer to discuss the evolution and the state of marketing measurement today – and why he sees LiftLab as standing far ahead of the pack. An edited version of our conversation follows below.
LiftLab: Welcome to LiftLab. To set the stage a bit, can you describe your new role and how you came to LiftLab to begin with?
Dirk Beyer: Sure. I’m the Chief Data Scientist here at LiftLab. That primarily means I oversee our modeling and experiments and the teams that drive that development.
But really, the story starts a long time before. I first met [LiftLab co-founders] John Wallace, Mike DeVries, and Bala Kandula, back when I was the Chief Science and Innovation Officer at MarketShare, an innovative measurement company that was later acquired by Neustar. At MarketShare, we acquired John, Mike, and Bala’s previous venture, a marketing effectiveness measurement platform provider called DataSong. I led the technology validation piece of the due diligence on the deal, was very involved in integrating DataSong’s product with MarketShare’s, and was highly impressed with both the DataSong product and the three of them as leaders. They truly have a knack for hitting the right level of sophistication to solve extremely complicated business problems at scale.
We kept in touch over the years; and flash-forward to about eight months ago, when they invited me to play an advisory role in advancing LiftLab’s Agile Mix Model approach. Having seen the models and methods up close, I can honestly say that much like DataSong, LiftLab’s approach is very innovative and well thought out to fit the actual needs of marketers. So, when I was invited to join full-time, taking the leap felt like an obvious choice.
LL: So it’s a role nearly a decade in the making. Does that timeframe, can you give me a sense of how innovation develops in the space overall?
DB: In my opinion, I’d say that the early years of adoption of regression techniques for marketing measurement – the marketing science underpinnings of marketing mix modeling (MMM) saw a great deal of irrational exuberance. That exuberance drove practitioners into three camps. The first camp was comprised of the absolute believers in the omnipotence of MMM and the underlying abilities to extract information from observational time series data. The second group was made up of those who dismissed MMM outright. The third group was made up of those who saw MMM’s deep value--but also recognized its limitations and tried to deal with them by incorporating learnings from marketing science and novel data sources.
I have always been firmly in the third camp, and that firm conviction in the potential of MMM supported by broader data science practices has been a driving force in my own career. Around 2012 or so, I led the development of the industry’s first hierarchical Bayesian MMM that zooms in on impact at the product and geo level, allowing for Bayesian priors, time-varying base, latent variables and nested models. In 2020-2021, I led the team at Uber that developed Orbit, which the company announced as “an open-source package for time series inference and forecasting.” Orbit is a Markov chain Monte Carlo-based Bayesian estimation toolkit allowing for time-dependent coefficients used for MMM. All of these “fancy features” are tools to increase stability and plausibility of the models, to democratize the learnings across the organization, and to enable Marketing Scientists to incorporate their deep structural knowledge about how marketing works into the models.
Some of the terms above might sound complex to less technically knowledgeable readers, but marketing analytics folks like myself will recognize these terms quite readily. They’ll also see how many of the approaches that were leading-edge five or ten years ago have become standard across MMM platforms throughout much of the industry today. That’s how all innovation works – yesterday’s cutting edge is today’s table stakes – but having lived it, it’s fascinating to see it play out.
LL: Which brings us to LiftLab, today. What do you see LiftLab as doing that’s really game-changing?
DB: I think it’s helpful to start with understanding the status quo. Today, most marketing measurement approaches work with enormous amounts of historical data and try to build a story out from there. That sounds like a positive, but in reality, it’s an approach that has two major drawbacks.
The first, and perhaps most fundamental drawback, is that more data means more noise to sift through. It’s an approach that favors sheer data “tonnage” over a thoughtful gathering of the information.
Take historical data: many models are built on historical data going back two or three years. That seems sound, until you consider that looking that far back will only give you better estimates of the present if the world for which you built that model is constant. But the world is anything but constant. Three years ago, we were facing lockdowns and global supply shortages which, thank goodness, we’re not facing directly right now. And if the world of 2021 is so different from the world in 2024, then it doesn’t make inherent sense to use 2021 data to guide 2024 decisioning.
The same issues are at play for adding more granular geographic data, parsing out demographic information, or any other variable. More data on its own isn’t inherently valuable – it’s just more. What you want instead is a more strategic approach to gathering the right data. Don’t get me wrong, I love data as much as the next guy. But thinking that if you can’t read an effect at the country level, all you need to do is run a pooled model with 200 DMAs—you can see how that’s just asking for trouble.
A second problem of the “more is more” data approach is that it’s enormously cumbersome to stand up and operate. When you’re dealing with wrangling and managing a large number of many potentially unreliable data feeds, it becomes nearly impossible to get a program off the ground quickly. Then once you are up and running, you can’t get timely insights to guide your marketing. Even if you take away the accuracy issues I’ve laid out above, you’re still dealing with a model that just isn’t fully usable to make timely marketing decisions. That’s unfortunate, as more effective decision-making is the whole reason models like ours exist.
LL: And LiftLab is solving these issues?
DB: It is. The reason why I'm excited about LiftLab is that we’re flipping the script – and we’re doing it through fast, smart experiments integrated with a purpose-built Agile Mix Model approach.
The benefit of experiments is that they let you zero in on the factors and impacts that you really want to study. And because you can design experiments to isolate one parameter from the surrounding noise, you can study the impacts of small, subtle changes that modeling alone could never let you see. It’s analyzing marketing under a microscope.
Experiments also provide current information—not reports on what was true on average in the past, but insight on what you’ve observed in markets just now. You can bring all that information into the model making all the other information in the model stronger as well— a very powerful flywheel effect.
Of course, LiftLab is not the only marketing measurement firm that’s incorporating experiments into its models. The difference is the experimentation design—much of which comes down to speed and the “agility” in the Agile Mix Model approach. By designing experiments that are both highly precise and quick to implement, LiftLab has created a method by which it can continually bring experimental results into the models.
In other words: the Agile Mix Modeling approach is designed to work in tandem with experimentation, not as an afterthought. Other approaches I’ve encountered have sort of two tracks – a historical model, and experiments running in parallel. Marketers can use those experimental results to sense-check the models, and those experimental results get worked into the models themselves maybe once or twice a year. LiftLab has created a single platform that brings the models and the experiments together—to give marketers the in-time insights they need to confidently make the best marketing decisions. That was a solution I wanted to be involved with.
Want to hear more from Dirk? Catch him on our fourth episode of Curve Your Enthusiasm, April 23rd, 2024.
