Data & Machine Learning
Experimentation & A/B Testing
8 practice questions. Free questions open a full answer guide; the rest unlock with Pro.
- You ran a standard A/B test on a two-sided marketplace feature and the treatment looked like a clear win, but a colleague worries the result is biased because treated and control users interact through the same supply. How do you reason about that risk and design an experiment you'd actually trust?
- A stakeholder is watching the dashboard for an experiment in flight and wants to call it as soon as the result crosses statistical significance. Walk me through why stopping the moment you see p < 0.05 is a problem, and how you'd let them monitor safely.Go Pro
- Your team runs A/B tests but most of them come back inconclusive after two weeks, and the pressure is to just ship on the directional read. As the senior engineer, how do you make the experiment program actually capable of detecting the effects you care about?Go Pro
- An A/B test shows your variant winning with a statistically significant lift. Before you recommend shipping it, what would you check to make sure the result is trustworthy?Go Pro
- Before launching an A/B test, how do you decide how long to run it or how many users you need?Go Pro
- Why is it a problem to check an A/B test repeatedly and stop as soon as the result looks significant, and what should you do instead?Go Pro
- When you run an A/B test, how do you decide which single metric determines whether the experiment won?Go Pro
- How do you design an A/B test to evaluate whether a new ML model outperforms the current one in production?Go Pro
Want questions matched to your role? Paste a job title, job description, or CV for a personalized set, or go Pro to unlock the full bank.