Table of Contents
I remember the first time I got asked a product data science interview question. I was interviewing for a data science role at LinkedIn a year out of college. The interviewer came in and nicely asked me how I would determine if a new feature on LinkedIn messaging would affect the user experience.
I was pretty confident in telling them that we could AB test this new feature. But they responded that they didn't have the capacity to do so. So now what?
I froze, because honestly, I had no idea.
No one is really good at product interviews unless they've seen them before. It's a specific type of interview question that is meant to test your ability to understand how to build products, and yet is especially tough to learn given the foreign nature of interpreting a case question in under an hour.
This specific type of interview is increasingly common for analytical roles. Big tech companies, such and Facebook and Google, focus mainly on product metrics and analytics, so data scientists function as an intersection between product managers, analysts, and software engineers. Data science, by definition, is the combo between product management and software engineering.
But I digress, let's dive into how to solve product interview questions.
Why do product interviews come up?
Product interview questions stem from companies testing how you would respond to the natural life-cycle of a product.
When looking at the general roadmap for any kind of product in technology, there's a team. This team usually includes a group of engineers, a product manager that sets up the roadmap and controls the major decisions, and one or two data scientists that measure product performance.
A typical timeline of the team’s decisions may look similar to this:
- The product manager asks the data specialist for product insight
- The two collaborate on building this feature based on data
- The feature is built, tested, and launched
- Further work is done to improve and analyze its successes or failures.
The data scientist role collaborates quite often with major product decision makers and builders, so these product data science questions often test your business intuition and ability to influence the product. A data scientist acts as a guardian of the data and is tasked with conveying the internal processes within the product, which the product manager relies on heavily as a key component of contributing to the product roadmap.
Many times, you’ll have to know what you’re investigating and the methods to measure success of a product, so these product data science questions have become increasingly important in the interview process.
Depending on the role, product interviews will come up at various frequencies. For instance, for machine learning roles, product interviews will almost never come up. The inverse can be said about product analyst interviews, where you'll encounter probably more than one product interview.
To better familiarize yourself with product interviews, check out this article about the Product Analyst Interview.
Nevertheless, if we're talking about big tech companies, almost every single data science role at Facebook, Apple, Amazon, Google, etc.. will ask a product interview question. And many times multiple times. This is due the very nature of the role being one that works with product owners very closely.
Frequency of Product Interview Questions
|Position||Product Interview Frequency|
|Product Analyst||Very Frequent|
|Data Product Manager||Very Frequent|
The MOST IMPORTANT thing you can do to prepare for a product data science interview
Believe it or not, the most important thing you can do to prepare for a product interview is to do what the recruiters tell you.
Use. The. Product.
Don't believe me? Think of the behavioral psychology standpoint.
Let's pretend I am instead the interviewer in the last scenario I posted where I flunked the interview. I work at LinkedIn and spend 8+ hours a day thinking about how to grow and improve messaging between users. I run AB tests, have growth benchmarks, tested different designs, ran thousands of SQL queries and analyses, and now have just been pulled into an interview.
What's my expectation of a data science candidate?
Well I mean, they don't have to blow it out of the water, but they do have to be someone I like to work with. All of my coworkers are smart and friendly people, and I tend to be able to bounce some good ideas on InMail and other LinkedIn messaging features off of them.
Is this expectation realistic that this data science candidate in front of me has thought about LinkedIn messaging as much as my coworkers have? Probably not.
But as an interviewer, I am judging you against all the other people I have interviewed and the only other people I know, my data science coworkers.
Whenever you interview as a candidate, remember that your interviewer is likely testing you on knowledge that you do not have as a non-employee of the company. That means that to catch up, you need to do as much product research as you can.
Thinking about the product as a person that works at the company is very different from just using the product on a regular basis. While I may use LinkedIn for browsing articles and connecting with other professionals, I don't use it in a way in which I am cognizant about which LinkedIn features have potential for growth in Q3. Looking at a company and understanding their product from a business perspective is imperative to understanding and connecting with employees in technical and product discussions during the interview.
Some ways to do this would be to guide your thinking with a few questions. For example with LinkedIn we can brainstorm:
- How does LinkedIn make money?
- What are some new features LinkedIn released. Why do you think they released them?
- Which features on LinkedIn drew you into the website? Which features drew you into a particular product?
- Why would a person pay money to use a certain service on LinkedIn?
For an in-depth look at the LinkedIn data science interview, check out this article!
The faster you start treating an interview like a business consultation, the better prepared you'll be for a product interview.
Different Types of Product Interview Questions
At Interview Query we have bucketed these interviews into about five distinct categories. Some get re-worded so they're trickier, some are repeated almost word by word from the examples we've found. Here are brief summaries on each type of interview question before we dive into the course structure below.
The first most common type of product interview question is on investigating metrics. These questions often center on a certain metric going up or down, and how this metric may communicate the health of a product. A common question template is “why is feature x dropping by y percent."
For example: “Why are Facebook friend requests dropping by 10 percent?”
With this question, we’d want to know information about the drop– is it important, does it affect the business, what may be the cause? An additional addendum to this question is on fractional metrics, which play into a concept on analyzing a metric that is a fraction, for example: “Why are Facebook friend requests per user dropping by 10 percent?”
The second most common product question is on measuring success. These questions ask for methods to measure the success of the feature or product. Examples could be “How would you measure the success of Facebook marketplace?” or “How would you measure the success of Yelp reviews?”
These are all features of an overlying platform or product, which ties into the product roadmap timeline that we referenced above as analyzing the success of released features. The company wants to know if the feature has improved anything and if it was worth the time and effort to build. At this point, the data scientist steps in and thinks of methodology to investigate the data, and determines its success.
The third type is feature changes. A typical question is in the format “Let’s say we want to add/change/improve a new feature to product X. What metrics would you track to make sure it’s a good idea?”
At first glance, this may seem similar to the previous question– you would track a feature change and its metrics. The small difference of a feature change is that the variables at play change as well. Instead of analyzing the entire product, you’re tasked with analyzing how this feature may have affected users.
The fourth type of question is about metric trade-offs. You can imagine this question to be along the lines of “If you are the product manager for Facebook and you see that comments are down by 10%, yet reactions are up by 15%, how would you deal with it?”
This question is extremely common especially in huge platforms such as Facebook, Google, and Amazon, in which there are definitely causal effects on other features when the sites updates or new features are released. A data scientist in this situation would need to decide which metrics matter, why this change might have occurred, and tackle the severity of the change.
The last type of question, which may not be as common as the other four, is on growth. A possible question could be “we want to grow x metric on y feature; how would we do that?”
Primarily, only companies focused on growth would utilize this problem, thus this question may appear in interviews for startups in growth mode or for companies with a huge acquisition team.
Lastly, these companies want to gain a sense of your strategic skill and business intuition in tackling these questions. A piece of advice is to think big and always take a step back before you dive in, as a common mistake is to instantly dive in and get pigeonholed into one specific idea. If you concentrate too much on a specific area, it’s difficult to envision the full cycle of the product or your ultimate product goal.
To get a better sense of similar questions asked in a product data science interview, check out this article on Product Analyst Interview Questions and Answers.
Framework for Tackling Product Questions
There's a couple different frameworks out there tackling product interview questions. At Interview Query we have combined these frameworks with an additional focus on metrics and applied data that employers are expecting from data scientist candidates.
While product managers cover a decent amount of this in their interviews, data scientists can focus less on the end user experience and more on larger behavioral changes from their users. Therefore a good general framework should be:
1. Clarifying the Question
What are the product goals? What's the background context? We almost always start at an information disadvantage. So what questions do we need to ask to bridge the gap?
2. Make Assumptions
Make some assumptions about the problem to narrow the scope. State what you'll explore in your analysis and what you won't.
3. Analyze User Flows
Examine exactly how the product works. How does a user get to a certain feature? How does a user use a certain feature? What kinds of different users are there?
4. Define Hypothesis
Start hypothesizing situations to explore that would help understand the root cause of the issue.
5. Draw Metrics to Support your Hypothesis
Use metrics as an example to further illustrate how it could prove or disprove your hypothesis.
6. Tie Your Analysis to the Product Goals
Finally tie your analysis back to the product goals. Give some sort of summary statement that can prioritize which ones matter and what the next steps are.
- Investigating Metrics
- Fractional Metrics
- Measuring Success
- Feature Change and Success
- Metric Trade-Offs
If you're interested in the product data science course, sign up for Interview Query to get the latest updates on our course release!