Karat Surveyed 400 Hiring Leaders. 71% Say Technical Interviews No Longer Work.
Why Technical Interviews Are Breaking in the AI Era
The traditional technical interview has a transparency problem. Companies are still running the same code tests and take-home projects they designed years ago, while both sides of the table know the signal those formats produce has weakened. Candidates use AI to generate solutions in seconds. Hiring managers suspect it and can rarely tell. Almost nothing about this dynamic has changed.
Karat published new data this week that quantifies how widespread the problem has become. The interview platform surveyed 400 engineering leaders across the U.S., India, and China, and the numbers are striking: 71% of leaders say AI is making technical skills harder to assess. Traditional evaluation methods, such as code tests and take-home projects, are degrading as reliable hiring signals the fastest.
For data scientists and ML engineers, this matters directly. The interview formats used to screen DS candidates were already under strain before AI tools became ubiquitous. The Karat data suggests companies know their current approach is not working but are struggling to agree on what should replace it.
What the Karat Data Actually Shows
The survey found that AI raises engineering productivity by an average of 34%, but that gain is not distributed evenly. Strong engineers who leverage AI to automate routine work and explore more ideas see outsized returns. Weaker engineers benefit less. As a result, 73% of hiring leaders now say strong engineers are worth at least three times their total compensation, a significant jump from prior years.
That widening productivity gap is changing what companies want to identify in an interview. Getting the right answer on a code problem matters less than it used to. What hiring managers say they now need to assess: judgment, adaptability, and what the Karat report calls AI fluency, which is the ability to use AI tools effectively while understanding the underlying work well enough to catch when the AI is wrong.
Why Code Tests and Take-Homes Are Losing Credibility
The report identifies code tests and take-home projects as the two formats losing signal fastest. The reason is straightforward: both evaluate output, not process. A candidate who pastes a prompt into an AI assistant and a candidate who reasons through the problem from first principles can produce identical results. Interviewers reviewing the submission cannot distinguish between them.
For DS roles, this maps to a familiar problem. SQL exercises, Python take-homes, and machine learning modeling assignments all share the same vulnerability. When the output is what gets evaluated, and AI can produce polished output on demand, the signal collapses.
62% of organizations still prohibit AI use in technical interviews, which means most companies are still using formats with this problem while simultaneously forbidding the tool that would make the problem visible.
What Companies Are Shifting Toward
The Karat data points toward live interviews as the format holding up best under AI. Live sessions let interviewers observe how a candidate works through a problem in real time: what clarifying questions they ask, how they handle ambiguity, when they reach for AI and when they don’t. That process signal is much harder to fake.
Chinese companies have moved faster in this direction, according to Karat. They are nearly twice as likely to allow AI during live interviews compared to U.S. companies. Meanwhile, a previous report on technical hiring notes that Meta has introduced an AI-enabled coding round to replace its traditional algorithmic interviews. The framing has shifted from no AI allowed to use whatever tools you would use on the job, and show us how you think while using them.
For DS candidates, this suggests interview loops at more adaptive companies are starting to look less like knowledge exams and more like structured work sessions. Questions about experimental design, stakeholder tradeoffs, model limitations, and business impact interpretation are harder to answer well with AI alone, while also being more predictive of actual job performance. Practicing with a real interviewer who can probe your reasoning is one of the most direct ways to prepare for this format. IQ’s coaching service pairs candidates with DS practitioners for live feedback.
What This Means for DS Interview Prep
A lot of standard DS interview prep is still optimized for the old format. Grinding SQL problems and memorizing statistical test conditions made sense when interviews were designed to screen for those skills in isolation. If companies are moving toward process-based assessment, that kind of preparation has diminishing returns on its own.
Candidates likely to perform well in an AI-fluent interview are those who can articulate their reasoning under observation, explain why they would or would not trust a particular model output, and demonstrate judgment when a problem is underspecified. That is a different skill than memorizing the correct output for a given input.
Practice that mirrors this, like working through ambiguous case questions, explaining tradeoffs out loud, and getting feedback in real time, is harder to do alone. IQ’s AI Interviewer lets you run live mock sessions and get feedback on your reasoning, not just your answers.
The Bottom Line
The Karat survey is one data point, and the pace of change varies by company. But 71% of hiring leaders acknowledging that technical interviews are harder to trust is not a fringe signal. It reflects a real tension that most candidates currently in a job search have already sensed, even if they could not name it.
The practical implication is that candidates who can demonstrate judgment and process (not just technical output) are better positioned as evaluation formats continue to evolve. The companies moving fastest on this are already interviewing differently. Understanding what they are actually measuring is the first step to preparing for it.