Christine Luo is a seven-year veteran of Indeed, widely recognized as a data analytics giant. Her pedigree includes a degree in finance, work in investment banking, corporate development and product roles spanning multiple titles and departments at Indeed, product ownership at real estate technology firm OJO, and senior product leadership at Wellfound (formerly known as Angellist Talent).
Fuzy was lucky to catch Christine for a few words of product management wisdom before she starts her next exciting role. In this interview, you’ll get Christine’s expert take on data-driven product decisions, along with some great advice for making the daunting field of data a little more manageable.
Walk us through the leap from your background in finance to discovering product development and product science.
Everybody did investment banking right out of school, but I quickly learned it wasn’t going to hold my interest. However, when I joined Indeed, I used my experience in banking on the international strategy team, which was a stepping stone into where I wanted to go in the company. I switched over to a newly forming division, corporate development, where my banking background was an asset, so it was a good fit. In that role, I was talking to almost 200 startup founders every year, because we were doing investments as well as M&A. While I was talking to these founders, making decisions about whether or not we should acquire or invest, I realized that I had never built a product. Who am I to judge if these startup founders have the right roadmap and ability to build their product? I had always thought I should go do a stint in product management so I know what these folks are talking about and I can gauge their credibility and legitimacy, actually having been in their shoes. So, my original stance of going into product was to do a short stint and come back into CorpDev knowing what I'm talking about.
In CorpDev and investment banking, I saw that data is king––how much should you pay for this, how much is this valued? So I think that was a really good foundation. When I had the chance to move into product development with a company Indeed acquired, I ended up falling in love with it. Solving user problems in creative ways and seeing the results in action got pretty addicting. But there's only so many times your gut knows exactly what to do, so that’s where product science comes into play. A lot of it is looking at what people are doing and why. What buttons are users rage clicking? That’s a problem. What are the three most common click paths? They click sign up, then they click create account, and then they click cancel account. That's a problem. Data like that is really telling for a PM.
I think data helped me move up in my career quite quickly, because it's a great equalizer. In a room with senior leadership, I was able to say, “We did this, and this was the outcome,” or, “You say that, but the data says that's actually not true.” The good thing about data is it doesn't challenge the person. It challenges the assumption. It makes it less personal; and it's hard to negotiate or dismiss data when it's right there.
Using data has helped me a lot in terms of determining our highest priorities and understanding what has the highest impact. It gave me credibility in places where it might be tough for somebody who's a woman and person of color and 27 at the time.
How has your career background shaped your approach to product management?
I think the biggest benefit of my career background is that I've been at very large companies like Indeed (with 10,000 people, a product science team and a data science team) and smaller places (where there were three product managers and no data team). As much as it is an absolute luxury to have a product science team available to you, I think it gives product managers a sense of, “It's not my problem. I don't have to deal with this; I will just make sure somebody else does.” I think that's really dangerous because learning how to look at data is the skill that will give you answers and teach your gut. But if you're not the one looking at the data, then you don't get that reinforcement. It's a core skill that only gets honed if you're actually the one looking at data.The folks that do succeed in their careers are the ones who tend to be more often right than wrong. So, the question is, “How do you become more often right than wrong?” And that is a muscle you can build. It's not because some people are more intuitive than others. That can be the danger of going external or having somebody else do the data for you versus having access to it. I think with something like Fuzy, you can satisfy your curiosity yourself versus writing a Slack message and having somebody else take care of it.
Let’s say someone is not completely data fluent or not a data scientist. How do you see the right balance of using tools for support versus being in the weeds and doing the difficult calculations yourself?
It is a good point to differentiate deep data science (doing regression analysis and statistical modeling to determine what is causal, for example). I don't expect any of my product managers to know how to do that. But they should be able to answer questions like, “We ran XYZ test. What was the impact? What were the results?” Or they should be able to find out, “When people tend to check out of a cart experience, what was their last action before that? They should be able to build and understand that flow.
Tools like Fuzy make that a lot easier. But I do think you need somebody within the company who's already data-fluent to teach you, because it’s hard and data is intimidating! When you look at dashboards and there are lots of numbers and trend lines and graphs, it’s hard to know where to even start. So I do think it's important that somebody within the company is championing the use of data and is willing to be that person who's doing lunch and learns, or 1-on-1s, or shadowing sessions.
I tend to think that goes hand in hand with buying a tool like Fuzy, because somebody needs to champion the fact that you're going to pay for this tool. More often than not, the person who's advocating for a tool like this is the one who's passionate enough to teach others.
What can you do to demystify data and make it a little less intimidating to use in business decisions?
I always recommend the book How Charts Lie to all my PMs who are just starting out. It's essentially “data misconceptions 101.” There are good resources out there that will teach you how to interpret xy graphs, when to use a pie chart versus a bar graph––questions that some might fear are too dumb to ask. The next thing is connecting with someone in your company who's good at data, or who regularly makes decisions using data, and asking to sit next to them for an hour (or hopping on a Zoom and screen sharing). I tend to think hands-on learning is the best, but some people like watching YouTube videos or reading a book.
For folks trying to implement some kind of data-driven tools and process, what tips do you have for getting executive and cultural buy-in?
Data is one of those things where everybody says, “Yes, we're data-driven,” but when you look underneath the hood, what you end up seeing is that people like data if it confirms their point of view. You might also hear that the data isn’t clean or a variety of other objections. What I like about data is that it is a great equalizer in the group. The most senior person and the most junior person are staring at the same chart. Data gives you authority that you may not have yourself if this is your first year out of school. What I would say for somebody who's trying to push for more data, especially at early stage companies, is data will help you make more right decisions than wrong. When you're at the beginning stages of a business, wrong decisions can make your company fail. Wouldn't you want to give yourself the greatest chance of succeeding? You can do that through data because there's no lying in there.
Buy-in may depend on who the decision maker is, but if it's a co-founder, they started the company because they want to solve a problem. You can frame data as something that will help set you up for success in solving that problem. With a CS person, the case is: through data, we can see the biggest pain points that customers are experiencing and then go solve those. Tailoring the value proposition to each stakeholder is a good place to start.
Can you describe the reaction that happens when you are able to present data that directly connects to a positive business outcome?
There was a project at Indeed in which I was responsible for the employer side of a product (which allowed SMB employers to post jobs). We continued to hear feedback from job seekers that they did not know what the salary was or where the job was located––things that were very important for hourly jobs and skilled labor. Without knowing the pay and the location of a job, it's really hard to gauge your commute and if the job is worth applying for.
We had always heard pushback from employers that salary is negotiable and safety concerns about location sharing. But we felt we could build a job posting funnel where employers would provide that information if we told them why it’s important for job seekers and how it will benefit them as employers. And we thought of various treatments for obscuring the exact address and providing salary ranges. I think a lot of people bet against us because they'd had years of people telling them they wouldn’t get this information. But we shipped it and we were able to gather 17% more job salaries and 23% more addresses. Those components still exist today.
What factors or criteria would you consider to determine whether you should build your own product science capacity or buy a PSaaS tool?
I think people truly underestimate how much it takes to maintain their own product science team or tool. Hiring takes a long time and it's extremely expensive. Indeed is notorious for building everything themselves. Part of that was Indeed spinning up before there were tools like Fuzy, so they had to build it themselves. They ended up with these very large product teams whose only job was to maintain an internally-built product. Those people were not working on revenue-generating products or the core product of job postings or application processes. The other thing you realize quickly is, because this is an internal tool, it is never the highest priority. On the other hand, when you use a tool like Fuzy, that company’s entire job is to make sure this product is working well, solves your problem, and serves your needs. I think people underestimate how complex it is to do that yourself. So that decision point is about the stage of growth.
Then, I think too often people assume their use case is unique so they should build it themselves. But that’s probably not true for most companies. I would say this is a case of the 80-20 rule: 80% of companies are not unicorns by definition, and they should use something off the shelf. And I would also say that probably 80% of the inquiries you want to run or questions that you have could be answered by a general solution like Fuzy, whereas a much smaller proportion lies outside those capabilities. Once you get to be Uber, you might have very specific questions like, “what is the impact of a one-way versus a two-way street?” Otherwise your basic questions are about user behavior and engagement, which a tool like Fuzy can easily answer for you––and probably suggest questions that you should be asking if you're not asking them already.
Speaking of blindspots, there’s a conundrum of the unobservable in product science––how can you know what you don't know when it comes to your data? What is your experience with that?
In my personal experience, the things I encounter outside of work have made me realize all the things I don't know. A good example is when I was listening to Hidden Brain on NPR when I was driving. They were discussing decision fatigue and how it affects you when you're going through funnels of choices. The specific example they gave was building your own car or customizing your own car. The later you are in the process, the more likely you are to accept the default choice. So by the time you get to details like what car mats you want in the tail end of questions, people aren't even reading––they just click next, next, next, next.
I had never thought about that, but our job posting funnel for Indeed was about 11 pages long. And we asked some of the most important questions at the very end. I asked myself, “Why don't we switch up the pages in a job posting panel and see what happens?”
The other thing that I would say is like listening to the experience of others. What questions are they asking? What things are they evaluating? I think what's powerful about a tool like Fuzy is they are aggregating that experience across numerous companies. Let's say they have a hundred customers––you're gaining the value of what a hundred companies-worth of people are querying for the product (versus sitting in a room with 10 people asking if there’s anything you’re missing).
Check out more interviews from product leaders like Christine
If you found Christine Luo's insights on data-driven product decisions and demystifying data intriguing, you won't want to miss the interview with Adam Dahlgreen, COO and Head of Product at Allstacks where he talks about the challenges and strategies of measuring meaningful outcomes, choosing the right features to build, and harvesting outcomes data for better decision-making and how that is being applied to engineering through Allstacks.