Ethan Handel is a senior product manager at Indeed and a friend of the Fuzy family. His eight-year tenure at Indeed includes building homegrown analytics tools, data cataloging, data warehousing, and everything in between. The last four years have seen him focus on data science, running teams of data/product scientists and engineers, conducting research in machine learning and deep learning, and leading the charge in applied data science––gathering, cleaning, and modeling data to predict hiring outcomes.
His extensive expertise in connecting the dots between raw data and desired outcomes makes Ethan a perfect brain to pick at Fuzy, where we focus on achieving that same goal through Product Science as a Service. In our interview, we got a detailed perspective on the challenges, gaps, and opportunities that exist along the path from data to insights.
How did product management and data analysis first intersect in your career?
I studied economics in school, which is why I have a lot of interest in being hands on with data. I went on to earn my MBA at NYU, specializing in business analytics, so that laid the groundwork for where I’ve ended up. I've since gone between data analyst and product management roles. That intersection is the fun part of building products: asking questions, using data, understanding behavior, finding things you don't expect. That just never gets old. So, that's the thread that has led me here.
What have you found to be the biggest obstacles in moving from data to insights?
The first problem to solve is whether you even have data, and is it the right stuff? It might even sound so basic as to be silly, but that’s a challenge for businesses at any scale and is always a moving target based on changing needs to the business, your technology stack, and the right knowledge and people. But let’s assume you have the data you want, it’s even in a pretty usable format, and the quality is there. There's still so much beyond that in going from data to insight. That leap is hugely complex but it’s also where all the value is.
It's pretty commoditized to get data, to have data, to store data in the cloud, to even leverage a bunch of smart tools to get pretty close to what you want without a bunch of experts in data engineering. However, for most companies it's still fundamentally a very human endeavor to go from data to insights. From a data analyst perspective, it's pretty easy to write queries. It's much harder to deeply know what you're querying and why––and knowing when to explore a new direction is non-trivial. There's only so many hours in the day, and there are infinite ways to slice data that could be interesting (and as with any experimental endeavor, you will fail a lot). A lot of it is intuition, but a lot of it is recognizing insights we missed. When your insights scale linearly with staffing you simply can't see it all. There are only so many hours in the day.
I think it's so hard to take the human element out of this. Of course there are things like, “How, how comprehensive is my data? How good is the quality? And how do I verify that quality?” These are all increasingly important as we have more automated ways to work with information. Take the data science case: Your model is only as useful as the consistency and quality of your features with respect to the original training data and assumptions. Building models is easy but understanding the state of a model at any given time is harder. And I think one of the main reasons it's so hard is because these kind of questions pass through a lot of hands. It's a human problem to a large degree.
In most businesses, particularly as you're growing and seeing success, the efficiency of identifying, validating, and sharing insights becomes exponentially challenging. Think about how many teams at a larger tech company work on a front end––a lot. You can see and feel broken experiences as a user, which is often a function of when teams don't quite coordinate or share information effectively. It’s the same thing with data: good data, ideal tooling, and appropriate staffing are the foundation to a data-driven culture, but these are only the starting point for getting business insight as your enterprise grows. When the knowledge of an analyst, team, and organization is disjointed, when understanding of the data is disjointed or incomplete, those kinds of problems multiply. And that's where insights become super hard. That’s where a lot of value is left on the table.
It’s easy to focus on narrower areas for specific metrics. It's much harder to connect dots when you don't really know what dots to look for. I think that's where insight development become difficult, and that’s where there's a lot of opportunity. Solutions that increase the velocity of high quality insights will change how companies work with data.
How do product teams and leaders currently ‘connect the dots’? What are the challenges and opportunities in that process?
In my experience, it does happen through tools sometimes, but it can still be inefficient and in some ways rudimentary. It looks like sharing queries from one database to another or good documentation within products and analysis functions. Again, these are often manual tasks. It can also look like sharing dashboards, depending what tools you're working with. But visibility, collaboration, and quality become an issue or limitation there. For sharing insights, the reality is the number one tool here is probably Slack, and I'll be the first to say, I hate that. It feels so inefficient to have a proliferation of smaller conversations with the people you think should know when anything changes. But that's how a lot of things get done––probably most. Database queries and data cataloging can serve that function, but that’s highly dependent on the solution you have and each have different limits.
Between direct communication and various tools, it’s a piecemeal thing. Then there's also the process question. How do you document an insight that someone can find later, as well as how you reached that insight? It sounds kind of shocking, but there's not a great, consistent way to do that. And some of that information is so important! Think about how much time you saved when you realized why someone came to a conclusion. Even just seeing that someone else has seen an insight is really valuable too. You start to build a shared understanding that connects dots both in your business as well as in the data, and this is vital to business results and a data culture as a competitive advantage.
That becomes really important when everyone is focused on their mandate, in their silo. Imagine being able to make meaningful connections there automatically. It just doesn't really happen organically, especially as you scale. Scale is a big issue with the collaboration. When you all work in one room, you can look over the edge and ask what someone's doing. That fails as you get into the hundreds and thousands of people.
There’s one final thing that interests me about the ‘connecting dots’ value proposition. I've noticed over time from a business leadership perspective, there’s a certain amount of lost productivity when basic insights aren’t automated.
It's really difficult, maybe impossible, to quantify. How much does it cost when we really aren’t working effectively with data between teams? I can’t easily put a dollar figure on it, but I know it’s a lot. There would be some really interesting qualitative research looking at how people spend their time with different sets of tools. I think this doesn’t get the attention you think it might because it’s largely unobservable to understand how much quicker a task could have happened. But I think when you see it, you know it––that experience of, “This is so much easier, and we have time to do other things.”
How do existing tools fail to meet product leaders’ needs to surface and circulate insights that are critical for making business decisions?
Something I've noticed is collaboration in analytics tools is pretty basic. I don't think it hits the mark very often. It is a hard problem to solve, but if you think about how teams and business leaders want to work with data, you see the same problems over and over again––questions that have already been answered, teams with expertise that you didn't know existed. Being able to get the building blocks of insights in a place that's visible, accessible, and transparent is an enormous problem with enormous value. Automation of analytical insight and collaboration is relatively low in common solutions, but it’s surely one of the biggest opportunities for businesses to get the most value out of the data they already collect. Systems and tools that meaningfully surface insights to you and those around you will always be a good investment, for any business.
I’ve seen some tools approach it in different ways, but I haven't seen many convincing solutions. I think they fall short by putting the burden on the user. Ideally, a lot of basic types of questions would get answered before you ask them. There's a lot of common product analytics problems one can start to automate which are not rocket-science problems, but are super valuable nonetheless. We should see increasingly good tools that challenge the assumption of what the user should be doing to get to an insight. Whether it’s a consumer-facing product or an internal analytics product, the best experiences are the ones that simplify, reduce uncertainty, and make the user feel like they have superpowers on the way to completing their objective.
More Product Leader Insights for Cultivating a Data-Driven Culture
From asking the right questions, to the collaborative process, and choosing the right tools to help connect the dots, Ethan is a well of wisdom for teams hoping to turn their data into meaningful insights that improve core business outcomes. Along with other experts we’ve interviewed, he has a lot to share about the human factors that contribute to a data-driven culture. Here are a few you should check out:
How to demystify data and make data-based decisions less intimidating: An interview with product expert Christine Luo
Why outcomes data builds better teams and products: An interview with product expert Adam Dahlgren
Lessons learned in data science at Vast and Indeed: An interview with product expert Tom Wilbur