AI interview prep for data scientists and ML engineers
Data and machine learning interviews hinge on one question: how did you know the result was right? An AI screen will ask about messy data, how you validated a model, an experiment that surprised you, and what happened after a model reached production. Here is what to expect, two worked answers, the mistakes that cost data candidates the most, and free practice at your level.
Entrevista con IA para ciencia de datos y machine learning: esta guía en español
What AI screens ask data scientists and ML engineers
Validation
“How did you test a model before anyone relied on it?”
Listening for: A holdout that matches how the model is used (often split by time), a baseline to beat, and a metric chosen for the decision rather than by habit.
A result that looked too good
“Tell me about a time a model looked too good. What was wrong?”
Listening for: The leak you found (a feature from the future, duplicates across splits, the target hidden in an id), and how you caught it.
Experiments
“Describe an A/B test whose result surprised you. How did you read it?”
Listening for: Sample size, the metric fixed before the test, novelty or seasonality, and what was decided.
Data you could not trust
“What did you do when the data you needed was missing, biased or wrong?”
Listening for: How you found out, what you did instead of dropping it, and what it did to the result.
Models in production
“A model got worse after launch. How did you find out, and what did you do?”
Listening for: The monitoring you had (or wish you had had), drift versus a pipeline bug, and the fix.
Simple versus complex
“When did a simple method beat a complex one for you?”
Listening for: A fair comparison on the same data, and the cost of the complex option in time, money or upkeep.
Explaining uncertainty
“How do you explain a forecast with a wide range to someone who wants one number?”
Listening for: A real case, the words you used, and what the person did with it.
Worked example: a lending model that looked too good
Each example shows a weak and a strong answer to the same question. The “With Aethel's rules” lines apply Aethel's published scoring rules, one concrete example of how an AI interview can judge an answer.
Question: “Tell me about a time a model looked too good. What was wrong?”
Weak answer
“Once my model had really high accuracy, around 99 percent, which seemed suspicious. I realized there was data leakage, so I removed the leaky features and retrained it. After that the accuracy was more realistic and the model worked well in production.”
Why it is weak: It names leakage but not the leak, the data or the model, so the interviewer cannot tell whether you found it or read about it. "More realistic" and "worked well" are not results, and accuracy alone is a weak metric for most real problems.
With Aethel's rules: Situation 1, reasoning 1, result 1: 3 of 8 points, and with no specific case the area is at most 60 on Aethel's scale.
Strong answer
“At a consumer lending startup I built a model to predict which new loans would be 30 days late in their first six months, trained on about 120,000 past loans. My first gradient-boosted model had an AUC of 0.94, far above the 0.71 of the scorecard it was meant to replace, which made me suspicious. I ranked the features by importance, and the top one was the number of collection calls on the account, which only exists after a payment is missed. Two more features were also updated after approval. I rebuilt the training set so every feature came from a snapshot taken on the approval date, and switched from a random split to training on one year of loans and testing on the next six months, which is how the model would be used. The honest AUC was 0.78. I showed both numbers to the credit team with the reason for the drop, and after a three-month shadow run on new applications, the top tenth of scores had about 2.4 times the late rate of the average applicant, close to what the holdout predicted.”
Why it is strong: The suspicion has a number behind it, the leak is named precisely (a feature that exists only after the outcome), and the fix changes both the features and the split to match real use. The answer reports the lower, honest number, explains it to the people who relied on it, and checks it against three months of new applications.
With Aethel's rules: Situation 2, reasoning 2, result 3: 7 of 8 points, which is 91 on Aethel's fixed scale. A real score can differ.
What comes next: Expect: how do you stop the next feature from leaking? A check in the pipeline that every feature is dated before the prediction date is a strong answer.
Worked example: an A/B test that fooled the clicks
Question: “Describe an A/B test whose result surprised you. How did you read it?”
Weak answer
“We ran an A/B test on a new recommendation widget and it increased engagement, which was surprising because we did not expect much. We rolled it out to everyone. A/B testing is really important for making data-driven decisions.”
Why it is weak: No product, no sample size, no metric definition, and no reason the result surprised anyone. An interviewer will ask how long it ran and whether engagement was the right metric, and the answer has nothing to say.
With Aethel's rules: Situation 1, reasoning 0, result 1: 2 of 8 points, and with no specific case the area is at most 60 on Aethel's scale.
Strong answer
“At a recipe app with about 900,000 weekly users, we tested a new home screen that put personalized recipes first. The metric we fixed before the test was recipes cooked per user per week, from the "I made this" button, with clicks as a secondary metric. After two weeks and about 180,000 users in each group, clicks rose 14 percent, but recipes cooked were flat, with an interval of plus or minus 2 percent. The product team wanted to ship on the clicks. I split the results by how long people had used the app, and the click lift came almost entirely from users in their first week, which looked like novelty, while the personalized recipes took longer to cook on average. I recommended two more weeks instead of shipping, and in weeks three and four the click lift fell to 3 percent with cooking still flat. We did not ship it, and the team changed the ranking to weigh cooking time, which became the next test.”
Why it is strong: The metric was fixed before the test, the surprise is a gap between two metrics, and the candidate explains it with a split that tests one hypothesis. The recommendation goes against what the team wanted, with a reason, and the extra two weeks confirm it.
With Aethel's rules: Situation 2, reasoning 2, result 2: 6 of 8 points, which is 87 on Aethel's fixed scale. A real score can differ.
What comes next: Expect: why not ship anyway, since clicks rose and nothing got worse? Have a reason ready, such as the cost of a slower home screen or a metric you did not want to trade.
Common mistakes in data and ML answers
- Reporting accuracy and stopping. On imbalanced data, accuracy hides everything. Say which metric fit the decision, such as recall at a fixed precision, and why.
- Random splits for problems in time. If the model predicts the future, test it on later data than it was trained on. Interviewers listen for this.
- Dropping messy data without asking why. Missing or strange values usually mean something. Say what they meant before saying what you did with them.
- Naming algorithms instead of choices. "I used XGBoost" is not a reason. A fair comparison with a simple baseline on the same split is.
- No decision at the end. An analysis nobody acted on is half a story. Say what was decided, or why nothing was.
- Hiding uncertainty. Intervals, ranges and the limits of your data belong in a strong answer. They are not a weakness to skip.
Numbers worth having ready
An interviewer places a case by its numbers. Find these for each of your cases before a data and ML interview; rounded is fine:
- Rows, users or events in your training data, and the time range they cover
- The baseline you compared against, and its score on the same holdout
- The metric you chose for the decision, with its value before and after
- For an experiment: users per group, how long it ran, and the interval around the effect
- Latency, cost or refresh schedule for a model in production
- The decision the work changed, with its size: money, time, stock or people
Practice data and ML questions free
Pick a level, type an answer to each question, and read the quick check after each one. Free, no account, nothing saved, and not a score.
Junior. Prepare an analysis, a query or a course project, and how you checked it.
Practice junior data and ML questionsMid-level. Prepare a model or experiment you owned from data to decision.
Practice mid-level data and ML questionsSenior. Prepare metric strategy, ML platforms and work that changed how a business decides.
Practice senior data and ML questions
Questions from data scientists and ML engineers
Will an AI interview ask me statistics or SQL questions?
Some screening tools include a written or coding test, so ask. Aethel's interview asks about your own work instead: how you checked data, validated a model and explained a result. Statistics come up in your reasoning, not as a quiz.
Does this apply to data analysts and analytics engineers?
Yes. Replace the model with the analysis or the dashboard: how you knew the numbers were right, who used them, and what changed. Validation questions apply to a SQL query as much as to a model.
Can I talk about a competition or course project?
Yes, especially at junior level. Say what you did beyond the starter notebook: a validation scheme, a feature you tested and dropped, or an error you found. Be clear that it was a competition or a course.
How do I talk about models without sharing private data?
Describe the data by its shape and size (about 120,000 loans, 30 features, two years) and round the results. The method and the checks are what the interviewer needs, not numbers your company keeps private.