The reported questions fall into four groups: product sense and metrics, SQL and data handling, A/B testing and statistics, and behavioral and leadership. Candidates also mention cross-validation, boosting and model evaluation as technical topics. That mix suggests splitting preparation evenly between analytical reasoning you can say out loud (metric design, diagnosing a drop, experiment design) and hands-on skills you can show (window-function SQL, a clean cross-validated model).
Metric diagnosis and experimentation sit closest to the day-to-day work as candidates describe it: investigating why a business metric moved, and designing tests for new features. Practise both with insurance-shaped examples, for instance a quote-to-purchase conversion rate, a renewal retention rate, or the accuracy of a risk score. These are practice scenarios, not claims about how 1st Central measures anything. For a 10% drop in a conversion rate, rehearse one fixed order of checks (is the number real, how was it defined, which segment, which change) so the answer stays structured under pressure.
Communication carries weight in the reports. Candidates describe a role that works with product managers and engineers, and they are advised to explain findings to non-technical audiences and not to assume an interviewer knows the details of past projects. For every project you plan to mention, prepare the business problem, your own contribution, the method and why you chose it over alternatives, the result, and what you would change.
The stages and their order are what candidates report, not a published process, and the accounts differ: one FAQ-style account describes two formal conversations, while the stage list names four. Ask the recruiter which conversations to expect and what each covers, then use the plan below as a base and shift days toward the stage that comes first.
Initial Screening
reportedCandidates report an initial screening as the first stage, used to judge whether you fit the role. Little else is described about its format, so treat it as the place to give a crisp account of your background: your SQL and Python or R experience, the experiments or models you have owned, and why this role interests you. Candidates are also advised not to assume the listener knows the technical details of past work, so keep the first version of every project story free of jargon and add depth only when asked.
What to demonstrate
- Fit with the Data Scientist role, which is the stated purpose of this stage
- How your background lines up with the requirements candidates describe: SQL, Python or R, and applying data science to business problems
How to prepare
- Write a short plain-language summary for two projects: the problem, what you did, and the business result. Say each one aloud to a friend outside data and check they can repeat it.
- Prepare a concrete reason for wanting this role and an insurance business. Read about 1st Central's insurance products on its public pages and note two specific observations about where customer retention or risk assessment could use data.
- Ask the recruiter how many conversations to expect, who you will meet, and which areas each covers, since reports differ on the structure.
Technical Assessment
reportedCandidates report a technical assessment that goes deeper into expertise, with A/B testing and statistics described as core pillars. Be ready to run an experiment from hypothesis to decision: choose the primary metric and guardrails, pick the randomisation unit, size the test, and interpret the result, explaining p-values and confidence intervals in plain language. The role also lists SQL as essential with Python or R, and candidates report SQL window functions as a fundamental. They do not say which stage tests SQL, so keep query fluency warm in case it appears here.
What to demonstrate
- Experiment design and statistics, which candidates describe as core areas of the technical assessment
- Statistical ideas explained plainly: significance, p-values and confidence intervals
- Awareness of experimentation pitfalls such as selection bias, novelty effects and Simpson's paradox
How to prepare
- Do the sample-size question end to end with numbers. For a 10% baseline conversion and a one percentage point lift (to 11%) at alpha 0.05 and power 0.8, n per group is about 7.85 x [p1(1-p1) + p2(1-p2)] / delta^2, roughly 14,750. Then explain how each input moves it.
- Run an A/A simulation in Python: 1,000 simulated tests at alpha 0.05 should flag about 5% as significant. Add repeated peeking and watch the false-positive rate climb, which gives you a concrete answer on experimentation pitfalls.
- Practise the product-metric questions out loud: a success metric for a new insurance feature, a diagnosis plan for a 10% conversion drop, and the short-term versus long-term trade-off.
- Keep SQL warm in case it appears in this stage: write window-function queries with SUM() OVER for running totals, LAG and LEAD for gaps between events, and RANK for ordering, then check each on rows with ties and NULLs.
Behavioral Assessment
reportedCandidates report a behavioral assessment aimed at cultural fit within the team. The behavioral questions candidates report cover influencing a stakeholder through communication and collaboration, a significant project challenge, a data science project you led and what you would change, and what to do when your recommendation conflicts with senior leaders' intuition. Candidates are advised to use the STAR method (Situation, Task, Action, Result). Build a small set of stories that each cover several of these prompts instead of memorising one answer per question.
What to demonstrate
- Communication and collaboration with cross-functional partners, which candidates report as a theme
- Resilience in a project that went wrong or got difficult
- Ownership of a project, including results and what you would do differently
How to prepare
- Pick three projects and, for each, write the stakeholder, the decision at stake, your specific action and a measurable result in one line. These cover the influence, challenge and led-a-project prompts between them.
- Prepare one story where your data-driven recommendation met resistance from someone senior: what evidence you brought, how you adapted your message, and what was decided.
- Write the 'what I would do differently' sentence for each story. Make it a concrete change in method or process, not a general lesson.
- Rehearse each story at both a short and a longer length, so you can adjust to the time you are given.
Final Technical Deep-Dive
reportedCandidates report that the process ends with a final technical deep-dive on advanced skills. Which topics fall here is not reported, so prepare the whole technical list: experiment design, SQL, and the modelling topics candidates mention (cross-validation, boosting, model evaluation and validation). Candidates are advised to be ready to defend a chosen model or test against alternatives, and to walk through a past project in depth, so expect to justify each choice and not only describe it.
What to demonstrate
- Depth of understanding on the advanced technical topics reported for the role
- Ability to defend a modelling or testing choice against alternatives
- Clear explanation of a past project's problem, technical approach and business outcome
How to prepare
- For boosting, be ready to say how it differs from bagging or a random forest, which settings control overfitting (learning rate, tree depth, number of trees, early stopping), and when a regularised logistic regression is the better call.
- For cross-validation, know the variants and when each is right: stratified k-fold for imbalanced targets, grouped folds when rows share a customer, and forward-chaining splits for time-ordered data, so no future information leaks into training.
- Pick the evaluation metric for a model you have built and justify it against alternatives, such as AUC versus precision-recall versus log loss or calibration, in terms of the decision the score drives.
- Prepare a deep walkthrough of your strongest project: the data issues you cleaned, the baseline you beat, the validation scheme, and the business effect.
PracHub editorial advice for the preparation topics above.
Answering a 10% conversion drop by listing causes instead of running a sequence
Open by checking whether the drop is real: logging changes, pipeline delays, and a changed definition. Then pin down whether 10% is relative or in percentage points, compare it with normal seasonal variation, and split by funnel step, channel, device and customer segment. Only then match the pattern to releases, pricing changes or external events, and say what result would change your conclusion.
Defining a p-value or confidence interval incorrectly, or giving a sample size with no inputs
A p-value is the probability of data at least this extreme if the null were true; it is not the probability the effect is real. State the inputs for sample size every time: baseline rate, minimum detectable effect, alpha, power, variance and the randomisation unit. Add that peeking and many comparisons inflate false positives.
Writing a window-function query without stating the partition, ordering and frame
Say the partition and ordering before you type. A running total with SUM() OVER (PARTITION BY user ORDER BY day) uses a RANGE frame by default, so tied dates share a total; write ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW when you need one row at a time. Remember that LAG and LEAD return NULL on the first or last row, and RANK leaves gaps after ties.
Naming boosting or cross-validation without defending the choice or the split
Tie the model to the problem: why boosting beats a simple baseline here, which hyperparameters control overfit, and how you checked. Match the split to the data: random k-fold leaks information when rows share a customer or arrive over time, so use grouped or forward-chaining folds, and keep preprocessing inside each fold.
Telling project stories that end at delivery or that blur your contribution
For every project, state the decision it changed, the number that moved, and what you personally did versus the team. Explain the context for a listener who does not know the project, and keep a concrete 'what I would change' ready.
Choose a category, try a prompt, then open its approach, worked solution or follow-up when you need it.
Simulate false alarms in a merchant chargeback monitoring rule
Baseline matured first-chargeback rate is 12 per 10,000 settled transactions. A monitoring rule alerts when a merchant's observed monthly rate exceeds twice baseline. For monthly settled transaction counts of 500, 2,000, 10,000 and 50,000, simulate the false-alarm probability per merchant-month under the baseline, and the power to detect a merchant whose true rate is 30 per 10,000. Then, for a portfolio of 4,000 merchants split 60, 25, 10 and 5 percent across those four counts, give the expected number of false alarms per month.
Approach
- Recognise the rule is a threshold on an integer count, not on a continuous rate. At n = 500, twice baseline is 24 per 10,000, so the first observable value above it is 2 chargebacks, or 40 per 10,000. Derive the trigger count for every n before simulating anything.
- Draw binomial counts with numpy at p = 0.0012 and take the share at or above the trigger for the false-alarm rate, then repeat at p = 0.0030 for power. Use at least 200,000 draws per cell so a probability near 0.001 has a usable standard error.
- Cross-check every simulated cell against the Poisson approximation with lambda = n*p, which is tight here because p is tiny. A mismatch almost always means the trigger count is off by one.
- Weight the per-merchant false-alarm probabilities by the portfolio mix, and report the share of expected alerts contributed by each size band rather than only the total.
- Close on the operating consequence: a fixed multiplicative threshold is not a constant false-alarm rate across merchant sizes, so either the threshold scales with n or small merchants need a minimum volume before the rule applies.
Follow-up
- How would you set a threshold that holds the false-alarm rate roughly constant across merchant size?
- The rule reads the transaction month, but disputes arrive for up to 120 days afterwards. What does that do to the alert and how would you fix it?
- What does a month of these false alarms cost, and how would you decide whether it is worth paying?
Estimate a delinquency roll-rate matrix and project twelve months
fct_loan_performance_monthly gives loan_id, as_of_month_end, months_on_book, delinquency_bucket, charge_off_flag, prepaid_in_full_flag and restructured_flag. Build a month-to-month transition matrix over the five delinquency buckets plus absorbing charged_off and prepaid states. Loans that stop appearing must be routed to an absorbing state rather than dropped. Project the current book forward 12 months by repeated matrix multiplication and report the projected share reaching charge-off. Handle restructured_flag explicitly, and name one place the Markov assumption fails on this data.
Approach
- Build consecutive month pairs per loan by shifting as_of_month_end within loan_id, then verify the shifted value is exactly one month later. A gap is not a transition, it is an exit you have not resolved yet.
- Resolve exits before counting anything. A loan whose last row carries charge_off_flag moves to charged_off, one carrying prepaid_in_full_flag moves to prepaid, and one that disappears with neither is a data question to raise rather than silently discard, because discarding it is survivorship that inflates every cure rate.
- Count pairs into a 7 by 7 matrix and row-normalise. Assert every row sums to one and the two absorbing rows are the identity; a row that does not sum to one means exits were dropped.
- Decide and state the restructure rule. Restructuring resets days_past_due, so a dpd_60_89 to current move on a restructured loan is not a cure. Either give restructured loans their own state or carry the pre-restructure bucket, but do not let that move land in the cure cell.
- Project by taking the current bucket distribution as a row vector and multiplying by the matrix twelve times. Report the charged_off entry, and report it again from an all-current starting vector so the reader can see how much of the projection comes from loans that are already delinquent today.
- State the homogeneity failure plainly: transition rates depend strongly on months_on_book, so one pooled matrix applied to a book with a young mix understates early-life delinquency. If the mix is moving, estimate separate matrices by seasoning band.
Worked solution 45 min
- Sort by loan_id and as_of_month_end, shift to form (from_state, to_state) pairs, and flag pairs whose month gap is not exactly one.
- For each loan's final row, assign the absorbing destination from charge_off_flag or prepaid_in_full_flag, and list loans that vanish with neither as an exception count to report.
- Apply the restructure rule, then build the 7 by 7 count matrix with a cross-tabulation over ordered state categories and row-normalise it.
- Assert row sums equal one and absorbing rows are the identity, then take the current month's bucket distribution as a row vector.
- Multiply twelve times, report the charged_off component, and repeat from an all-current vector for comparison.
Follow-up
- How would you validate the projection against what actually happened, and over what window?
- The cure rate out of dpd_30_59 rose five points last quarter. What are the candidate explanations and how would you separate them?
- When would you prefer a vintage curve to a roll-rate projection, and why?
Write integrity checks for the authorization and settlement lifecycle
You are given fct_payment_authorization as a pandas DataFrame with auth_id, requested_at, amount_minor, transaction_currency, auth_result, decline_reason_code, is_reversal, parent_auth_id, captured_at, captured_amount_minor, settled_at, settlement_amount_minor, settlement_currency and settlement_fx_rate. Write a function returning one row per integrity check with the check name, failing row count, failing share and up to five example auth_id values. Cover at least six checks, one of which reconciles captured_amount_minor against settlement_amount_minor through settlement_fx_rate. Partial capture, zero-amount verification and a decline with no capture are all legitimate and must not be flagged.
Approach
- Separate contract violations from observations before writing any code: an approved row carrying a decline_reason_code is structurally impossible, while a capture two days after requested_at is merely slow and belongs in a different severity tier.
- Express each check as a boolean mask over the whole frame and collect the masks in a dict, so the summary table is one comprehension over mask.sum() rather than a row loop.
- For the reconciliation, leave minor units before comparing: expected = captured_amount_minor / 10exponent[transaction_currency] * settlement_fx_rate * 10exponent[settlement_currency]. Build the exponent table covering zero-decimal and three-decimal currencies instead of assuming two everywhere.
- Guard the legitimate cases explicitly so each mask fires only on the genuine contradiction: captured_amount_minor below amount_minor is partial capture, amount_minor of zero on an approved row is account verification, a null captured_at on a declined row is correct.
- Sort the output by failing share times a stated severity weight, because a check firing on 0.01 percent of rows can still be the one that breaks a ledger reconciliation.
Follow-up
- Which of these would you run as a blocking pipeline assertion and which as a monitored metric, and why?
- The FX check fails on 3 percent of rows, all in one settlement currency. How do you decide between a data bug and a rounding convention?
- How would you detect that a currency's minor-unit exponent is wrong in your reference table, using only the transaction data?
Describe a situation where you had to join multiple complex tables to …
Describe a situation where you had to join multiple complex tables to derive a specific business insight.
Approach
- Open with the business question and the grain of the output (for example one row per policy per month), then list the tables and their keys, such as quotes, policies, customers and claims.
- Join outward from the grain table, and collapse one-to-many tables (claims per policy) to one row per key in a CTE before joining, so sums and counts do not inflate from fan-out.
- Use a LEFT JOIN where absence matters (policies with no claims) and COALESCE the counts to 0; keep right-table filters in the ON clause so the LEFT JOIN does not turn into an inner join.
- Handle history tables explicitly: join on the key plus a date condition (event_date BETWEEN valid_from AND valid_to) so a mid-term policy change maps to the right version.
- Show how you validated it: uniqueness checks (GROUP BY key HAVING COUNT(*) > 1), row counts after each join, a reconciliation to a known total, then the insight and the decision it changed.
Follow-up
- How did you confirm the joins did not duplicate rows?
- A dimension table keeps several historical rows per key. How do you pick the right one?
- How would the query change if you also needed customers who never bought a policy?
Can you explain how you use SQL window functions to perform time-serie…
Can you explain how you use SQL window functions to perform time-series analysis or running totals?
Approach
- State the shape: func() OVER (PARTITION BY key ORDER BY time frame). A running total is SUM(amount) OVER (PARTITION BY customer_id ORDER BY txn_date ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW).
- Know the default frame: with ORDER BY and no frame clause it is RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, which includes peer rows with the same ORDER BY value, so tied dates share one running total; write ROWS when you want row-by-row accumulation.
- Rolling metrics: AVG(x) OVER (ORDER BY day ROWS BETWEEN 6 PRECEDING AND CURRENT ROW) is a 7-day average only if there is one row per day, so fill gaps with a date spine first (generate_series in Postgres), or use a RANGE frame with an interval offset where the engine supports it.
- Period-over-period: LAG(x) and LEAD(x) give the previous or next value for day-over-day change or the gap between events; the first row returns NULL unless you pass a default as the third argument.
- Window functions run after WHERE, GROUP BY and HAVING, so aggregate first (SUM(SUM(x)) OVER (ORDER BY day) for a cumulative daily total) and filter on a window result in an outer query or CTE (or QUALIFY where supported). Cost is mainly the sort per partition, O(n log n).
Follow-up
- Why does a running total repeat the same value on tied dates, and how do you fix it?
- How do you compute a 7-day rolling average when some days have no rows?
- How would you keep only the latest record per customer, and which ranking function would you use if timestamps tie?
Accident-quarter loss ratio on earned rather than written premium
From fct_policy_period_monthly, compute the accident-quarter loss ratio by product_line: incurred losses, being paid_loss_minor plus case_reserve_minor plus ibnr_reserve_minor, over earned_premium_minor for the same accident quarter. State explicitly whether loss_adjustment_expense_minor is included and apply that choice consistently. Also output the same ratio computed on written_premium_minor so the two can be compared. The table holds current values with no valuation-date snapshot. Say in one line which comparison this schema cannot support and what you would need to support it.
Approach
- Derive the accident quarter from as_of_month with date_trunc, and note that the table already attributes losses to the month of the loss event while earning premium pro rata into the same month, which is what makes the two sides comparable at all.
- Aggregate earned_premium_minor, written_premium_minor and the three loss components to product_line and accident quarter in one pass, keeping loss adjustment expense as its own column so the inclusion choice is a final-select decision rather than something buried in a CTE.
- Compute both ratios side by side and a third column for their difference, because the size and sign of that difference is a direct read on whether the book grew or shrank in the quarter.
- State the limitation plainly: every row carries today's reserve estimate, so each accident quarter is observed at a different development age and a cross-quarter comparison mixes development with underwriting. A fixed development age needs a valuation-date dimension, that is one row per accident period per valuation, which this table does not have.
- Guard against the mirror-image error on the numerator by confirming ibnr_reserve_minor is non-zero on recent quarters; if it is null or zero there, the recent periods are understated twice over and the series is not usable.
Worked solution 40 min
- CTE quarterly: group fct_policy_period_monthly by product_line and date_trunc('quarter', as_of_month), summing earned_premium_minor, written_premium_minor, paid_loss_minor, case_reserve_minor, ibnr_reserve_minor and loss_adjustment_expense_minor.
- Final SELECT: build incurred_minor as the three loss components plus the LAE column, with the LAE inclusion written as a named expression so the choice is visible on the page.
- Emit loss_ratio_earned and loss_ratio_written, both cast to numeric, plus their difference and the written-to-earned premium ratio.
- Order by product_line and accident quarter, and append the one-line note about the missing valuation dimension to the query as a comment.
Follow-up
- Written premium exceeds earned premium by 18 percent this quarter and by 3 percent two years ago. What happened to the book, and what does it do to each ratio?
- How would you build a development triangle from a valuation-dated version of this table, and what would you use the chain-ladder factors for?
- Statutory presentation conventionally takes the expense ratio on written premium while the loss ratio uses earned. How do you avoid a combined ratio that quietly mixes the two bases?
How do you balance short-term metric optimization with long-term produ…
How do you balance short-term metric optimization with long-term product health?
Approach
- Make the tension concrete for insurance: a discount or simpler quote flow can lift quote-to-purchase conversion this month while attracting price-sensitive or higher-risk customers who lapse at renewal or claim more. Name the short-term metric and the long-term outcome it could damage.
- Pair one primary metric with long-horizon guardrails such as renewal rate, loss ratio (claims cost divided by earned premium), cancellations and complaints, and agree the guardrail thresholds before the change ships.
- Bridge the time gap with validated leading indicators: on historical cohorts, check which early signals (early cancellations, payment failures, first-month contacts) actually predict 12-month retention or claims, and use only those as surrogates.
- Keep a long-running holdout or track launch cohorts over time, and compare them only once they have matured, since recent cohorts have had less time to lapse or claim.
- Put both horizons in one unit when possible, for example expected customer value (margin over expected tenure minus expected claims), so the trade-off becomes a single comparison rather than an argument.
Follow-up
- How would you choose the threshold at which a guardrail blocks a launch?
- If the long-term outcome takes a year to observe, what would you decide now and how would you revisit it?
- How would you tell a novelty effect apart from a lasting improvement?
How would you design a metric to measure the success of a new insuranc…
How would you design a metric to measure the success of a new insurance product feature?
Approach
- Clarify the feature, who is eligible for it and the decision the metric must inform (ship, iterate or roll back) before naming anything.
- Build a small hierarchy: adoption (share of eligible customers who use the feature), a success metric tied to the business outcome (for example quote-to-purchase conversion or renewal rate), and guardrails such as loss ratio, cancellations, complaints and time to complete.
- Define each metric exactly: numerator, denominator, eligible population, unit (customer, quote or policy) and window (for example purchase within 30 days of quote), so two analysts would compute the same number.
- Test the metric before trusting it: it should move if the feature works, be hard to game, and have a known baseline and variance so you can size an experiment around it.
- Measure impact against a counterfactual (A/B test or staggered rollout), never adopters versus non-adopters, because customers who opt in differ from those who do not (selection bias).
Follow-up
- Adoption is high but the success metric does not move. What do you conclude?
- How would you measure impact if you could not randomise who gets the feature?
- Which guardrail would you watch most closely, and why?
If you noticed a sudden 10% drop in a key conversion metric, how would…
If you noticed a sudden 10% drop in a key conversion metric, how would you go about diagnosing the root cause?
Approach
- Confirm the drop is real first: tracking or release changes, a metric definition change, a partial day of data, pipeline delays; cross-check against an independent source such as completed policy sales in a transactional table.
- Compare against normal variation: day-of-week and seasonal patterns, the same period last year, and known calendar events, to judge whether 10% falls outside the usual range.
- Decompose the ratio: did the denominator change (a surge of low-intent quote traffic) or the numerator (fewer purchases)? Walk the funnel step by step to find where users stop.
- Segment by channel, device, browser or app version, region, new versus returning customers and product; separate mix shift (more traffic from a low-converting segment) from a within-segment rate drop, since the aggregate can mislead (Simpson's paradox).
- Check internal causes (deploys, pricing changes, running experiments, outages) and external ones (marketing changes, competitor pricing), then size each hypothesis and close with a recommended action and owner.
Follow-up
- The drop is spread evenly across every segment. Where do you look next?
- How would you quantify how much of the drop each segment explains?
- How would you separate a mix shift from a genuine change in conversion rate?
What are some common pitfalls you encounter when designing and monitor…
What are some common pitfalls you encounter when designing and monitoring product metrics?
Approach
- Goodhart's law: once a metric becomes a target it gets gamed (for example conversion bought with discounts), so pair each target with a guardrail.
- Ratio pitfalls: a rate can rise because its denominator shrank; be explicit about ratio of averages versus average of ratios, and make the analysis unit match the unit the metric is built on.
- Mix shift and Simpson's paradox: an aggregate moves because the composition changed while every segment stayed flat, so monitor key segments alongside the total.
- Immature data: claims, cancellations and renewals arrive with a lag, so recent cohorts look artificially good; compare cohorts at equal age and report only matured periods.
- Monitoring hygiene: alert thresholds that ignore seasonality cause false alarms, silent definition or instrumentation changes break time series, and watching dozens of metrics guarantees some move by chance.
Follow-up
- How would you set an alert threshold for a metric with strong weekly seasonality?
- Give an example where a metric improved while the business got worse.
- How would you handle a metric definition change in a historical dashboard?
How do you determine the required sample size for an experiment before…
How do you determine the required sample size for an experiment before you begin?
Approach
- List the inputs: baseline rate or variance of the primary metric, the minimum detectable effect that would change the decision, significance level (commonly 0.05 two-sided), power (commonly 0.8) and the allocation ratio.
- For two proportions: n per group ≈ (z_{1-α/2} + z_{1-β})² × [p1(1-p1) + p2(1-p2)] / (p1 - p2)². With α = 0.05 and power 0.8 the z term is (1.96 + 0.84)² ≈ 7.85; for means, n ≈ 2σ²(z_{1-α/2} + z_{1-β})² / δ², which gives the rule of thumb 16σ²/δ².
- Work an example: baseline 10%, MDE of 1 point (to 11%) gives 7.85 × 0.1879 / 0.0001 ≈ 14,750 per group. Since n scales with 1/δ², halving the MDE roughly quadruples the sample.
- Turn n into duration using eligible traffic per day, round up to whole weeks to cover weekly cycles, and note that slow outcomes like renewal may force a validated proxy metric.
- Adjust for the design: clustered randomisation inflates n by the design effect 1 + (m - 1)·ICC, unequal splits and multiple variants need more, and variance reduction such as CUPED cuts the required n by roughly a factor of (1 - ρ²).
Follow-up
- Traffic supports only half the required sample. What are your options?
- Why not stop the test as soon as the result is significant?
- How would you choose the minimum detectable effect with a stakeholder?
What are the most common experimentation pitfalls that lead to false p…
What are the most common experimentation pitfalls that lead to false positives?
Approach
- Peeking: checking repeatedly and stopping at the first p < 0.05 pushes the false-positive rate well above 5%. Fix the horizon in advance or use a sequential design with alpha spending (for example O'Brien-Fleming boundaries).
- Multiple comparisons: with 20 independent tests at α = 0.05, the chance of at least one false positive is 1 - 0.95^20 ≈ 64%. Commit to one primary metric before the test, and apply Bonferroni or Benjamini-Hochberg across secondary metrics and segments.
- Unit mismatch: randomising by customer but analysing by session or quote treats correlated rows as independent, which makes confidence intervals too narrow. Use the delta method or a cluster bootstrap at the randomisation unit.
- Broken randomisation: test for sample ratio mismatch with a chi-square test against the intended split, avoid filtering on anything measured after assignment (selection bias), and run A/A tests to check that the pipeline yields about 5% false positives.
- Time effects: novelty effects that fade, and Simpson's paradox when the allocation ratio changes during a ramp-up, so compare within periods of constant allocation and check whether the effect holds over time.
Follow-up
- How would you detect a sample ratio mismatch, and what would you do if you found one?
- Stakeholders insist on watching results daily. How do you allow that without inflating false positives?
- A segment shows a significant lift that the overall result does not. How do you treat it?
Success metrics for loosening a fraud decline threshold
A risk team proposes lowering the risk_score cutoff that produces auth_result = 'declined_risk_rule'. Settled volume per active customer is the north star; net fraud loss in basis points of settled volume is the guardrail. The two move in opposite directions by construction. Specify the readout: primary metric, guardrail, the maturity window each is read at, and the decision rule agreed before launch. Show the expected-cost arithmetic that sets the cutoff using an average ticket of 200 units, a 1.5 percent contribution margin, 35 percent recovery on fraud losses, and 12 units of downstream value lost per false decline.
Approach
- Refuse the two-metric framing and convert both sides into one currency. Approving a fraudulent transaction costs the amount net of recovery; declining a good one costs the forgone margin plus the downstream value of the customer's reaction. Decline when p times C_FN exceeds (1 minus p) times C_FP, so the break-even probability is p* = C_FP / (C_FP + C_FN).
- Put the numbers in. C_FN = 200 times (1 minus 0.35) = 130, C_FP = 200 times 0.015 plus 12 = 15, so p* = 15 / 145 = 10.3 percent. Then show the threshold is amount-dependent: at a 2,000 ticket C_FN = 1,300 and C_FP = 42, giving p* = 3.1 percent, so a single global cutoff is already the wrong shape before any tuning starts.
- State the precondition that makes this arithmetic legal: risk_score has to be calibrated, so that a score of 0.10 corresponds to an observed 10 percent fraud rate. A score that only ranks makes p* meaningless. Check the reliability curve before quoting any cutoff to anyone.
- Set the maturity windows separately. Volume is readable within days, fraud loss is not, so the guardrail is read only on transaction months with at least 120 days of dispute maturity and the decision stays open until then, or a leading indicator is agreed in advance with its bias written down.
- Agree the stopping rule before launch in the right units: revert if matured net fraud loss per unit of incremental settled volume exceeds the figure implied by p*. Fraud loss is supposed to rise when the cutoff loosens, so a rule that triggers on any rise is a rule that was never going to allow the change.
- Report the swap set rather than portfolio totals: the transactions the new cutoff approves that the old one declined, and their realised loss rate. Portfolio aggregates dilute the change into invisibility.
Worked solution 30 min
- Compute p* at ticket sizes of 50, 200 and 2,000 with the given margin, recovery and false-decline cost, and tabulate them.
- Bucket historical declined_risk_rule authorizations by risk_score decile and, for each bucket, write down what outcome data exists and what does not.
- Write the readout spec: primary metric, guardrail, the 120-day maturity rule, the swap-set table and the numeric stopping rule.
- Write the calibration precondition in two sentences and say how you would test it.
Follow-up
- Fraud loss in basis points falls after launch. Name two ways that happens without any improvement in decisioning.
- How do you keep observing outcomes in the region the rule still declines?
- What changes if the 12 units of downstream value is a guess with no evidence behind it?
Portfolio delinquency improving while the loan book doubles
The blended 90-plus days-past-due rate across fct_loan_performance_monthly fell from 3.1 to 2.2 percent over two quarters while monthly funded volume roughly doubled. Credit leadership wants to know whether underwriting improved. Columns: loan_id, as_of_month_end, origination_month, months_on_book, original_principal_minor, principal_balance_minor, days_past_due, delinquency_bucket, restructured_flag, charge_off_flag, charge_off_date. Produce the view that answers the question honestly, and state in one sentence what the blended rate can and cannot tell you.
Approach
- Name the mechanical floor first. A first instalment falls due roughly a month after funding, so a loan cannot reach dpd_90_plus until around its fourth month on book. Every recent origination therefore enters the denominator with a numerator that is structurally zero.
- Build a vintage table: rows origination_month, columns months_on_book, cell equal to the share of that cohort whose worst days_past_due reached 90 or more, or whose charge_off_flag became true, at or before that age.
- Use each loan's worst state to date rather than its current bucket, and take the pre-restructure worst state, because restructuring resets days_past_due and would otherwise read as a cure.
- Compare cohorts only at equal months_on_book, and render cells beyond a cohort's current maturity as absent rather than zero, so the table cannot be misread left to right.
- Decompose the blended move into an age-mix component and a within-age component, so the write-up states how much of the 0.9 point improvement is arithmetic rather than asserting it.
Follow-up
- What does the diagonal of a vintage table represent, and when is reading it the right thing to do?
- How would a change in charge-off timing policy show up in this table, and how would you separate it from credit quality?
- Which single chart goes in front of the credit committee, and what do you say when someone asks for the blended series anyway?
The plan follows the reported question groups: metrics first, then SQL, experimentation, modelling, stories, a day of PracHub practice questions on loss ratios, matured cohorts and decision-threshold metrics, and a full rehearsal across all four stages. Each day produces something you can check.
Prepare, practise & reflect
One practical outcome each day. Spend longer where you need it.
0 / 7 done01Metric design and drop diagnosis
- Write a metric tree for a new insurance product feature: one primary metric (for example quote-to-purchase conversion), two guardrails, and the diagnostic rates that sit beneath it. Say who acts on each one.
- Write a fixed diagnosis sequence for a sudden 10% drop in a conversion metric: confirm the data, confirm the definition, compare with normal variation, cut by segment and funnel step, then check releases and external events.
- Write a short answer on balancing short-term optimisation with long-term health, naming a holdout group, a retention-style long-term measure, and the guardrail that would stop you from over-optimising.
- List the common pitfalls in metric design and monitoring (a metric that can be gamed, a ratio whose denominator changes, a Simpson's paradox across segments) with one example each.
Deliverable: A one-page metric tree, a diagnosis checklist, and written answers on trade-offs and pitfalls for the four product-metric questions.
Practice prompt ↗Practice prompt ↗Practice prompt ↗Practice prompt ↗Worked solution ↗02SQL joins and window functions
- Create a small database (SQLite or Postgres) with customers, quotes, policies and payments tables you generate yourself, seeded with a customer who has no quote, a tie on a timestamp and a NULL key.
- Write a running total with SUM() OVER, a gap to the previous event with LAG, a next-event lookup with LEAD, and a ranking with RANK versus ROW_NUMBER on tied rows. Check every result by hand.
- Join three or more tables to answer one business question. Print row counts after each join to catch a one-to-many fan-out, and fix any inflated sum by aggregating the many-side first.
- Write a two-minute spoken explanation of how you would use window functions for time-series analysis, then record yourself and re-listen.
Deliverable: One .sql file with verified window queries, a multi-table join with row counts at each step, and the spoken explanation.
Practice prompt ↗Practice prompt ↗03Experiment design and statistical significance
- Compute sample size by hand for a conversion test (baseline, minimum detectable effect, alpha 0.05, power 0.8), then confirm it with a Python power calculation. Vary the minimum detectable effect and note how n changes.
- Simulate A/A tests in Python: run 1,000 and count significant results at alpha 0.05. Repeat with a rule that stops at the first significant look, and record how much the false-positive rate rises.
- List the main experimentation pitfalls behind false positives: peeking, multiple comparisons, sample ratio mismatch, selection bias, novelty effects, interference between units. Write a one-line detection or fix for each.
- Write plain-language explanations of a p-value and a 95% confidence interval that a product manager could repeat correctly.
Deliverable: A notebook with the sample-size calculation and both simulations, plus a one-page sheet of pitfalls and plain-language definitions.
Practice prompt ↗Practice prompt ↗04Modelling: cross-validation, boosting and evaluation
- Fit a logistic regression baseline and a gradient-boosted model on a public tabular dataset in Python, scoring both with stratified k-fold cross-validation.
- Introduce a leak on purpose (a feature built from future information, or duplicate customers across folds), watch the score inflate, then repair it with grouped or time-ordered folds.
- Tune learning rate, tree depth and number of trees with early stopping, and write down which setting reduced overfitting most.
- Write a short answer on handling missing and noisy values before modelling: outlier checks, missingness indicators, imputation fitted inside each fold, and a data-quality check in the pipeline.
Deliverable: A notebook comparing baseline and boosted model under honest validation (with the leak demonstration), a note on which setting cut overfitting most, a half-page defence of the final model, and a written answer on missing and noisy data.
Worked solution ↗05Project stories and stakeholder influence
- Choose three projects and write each as a STAR outline with a named stakeholder, the decision at stake, your own actions and a measured result.
- Write the full answer to the influence-a-stakeholder prompt, including the objection you faced and the compromise or pilot that moved the decision.
- Write the challenge story and the 'what I would do differently' line for the project you led, and prepare a second story where senior intuition conflicted with your data.
- Deliver each story aloud and cut anything that does not add to the decision, your contribution or the result.
Deliverable: Three STAR outlines with measured results, the written stakeholder-influence answer, the challenge story with its 'what I would do differently' line, the senior-leadership-conflict story, and a recording of the influence story.
Practice prompt ↗06PracHub practice on loss ratios, cohorts and threshold metrics
- Work the PracHub practice question on accident-quarter loss ratio (an insurance metric) using earned premium, and note why written premium would give a misleading ratio.
- Work the PracHub practice question on portfolio delinquency improving while the loan book doubles (a lending context). Write the cohort-based check that tests whether the improvement is real.
- Work the PracHub practice question on success metrics for loosening a fraud decline threshold (a fraud context), then compare its structure with your day-1 metric tree.
- Note one habit from each exercise (matured cohorts, correct denominator, guardrails) and add it to your diagnosis checklist.
Deliverable: Three worked answers plus an updated metric checklist that names the new habits.
Practice prompt ↗Practice prompt ↗Practice prompt ↗07Full rehearsal across the four stages
- Run four short mock sessions with gaps between them, mirroring the four stages: a screening introduction, a technical session (a window-function query and a sample-size answer), a behavioral session, and a deep-dive on one project and the model behind it.
- In the technical mock, diagnose a 10% conversion drop aloud using your fixed sequence and explain false-positive pitfalls without notes.
- Re-do the weakest answer from each mock, then write the questions you will ask your recruiter and the team about their current challenges and how the role helps.
Deliverable: Mock feedback notes, a re-recorded weakest answer from each session, and a list of questions for the recruiter and the team.
Practice prompt ↗Practice prompt ↗Practice prompt ↗Practice prompt ↗Worked solution ↗Expand any day for tasks and deliverables. Your progress is saved on this device.
Candidates report that the behavioral stage covers communication, collaboration and resilience, in a role that works with product managers, engineers and non-technical stakeholders. Prepare a few stories that show you influencing a decision with data, working through a setback, and owning a project end to end. Each story should end with a result and an honest lesson.
Tell us about a time you used your communication and collaboration ski…
Tell us about a time you used your communication and collaboration skills to influence a stakeholder.
Approach
- Pick a story where you had no formal authority over the decision, such as a product manager, a pricing or underwriting lead, or an engineering lead who believed something your data disagreed with. Open with the situation in two sentences: who the stakeholder was, what they wanted, and what was at stake.
- In the action, show the communication choices you made on purpose: you asked what they were worried about before presenting, restated your finding in their metric (retention, conversion, loss ratio, cost) instead of yours, led with one number and one chart, and raised the likely objection yourself before they did.
- Show collaboration as well as persuasion. Describe how you adjusted your own plan after listening, for example by agreeing a smaller pilot, an A/B test, or a segment-level cut to lower the risk of their saying yes.
- Finish with the result and your exact share of it: what decision changed, which number moved, and how you checked afterwards. Separate what you did from what the team did. Close with one concrete thing you would do differently.
- The mistake that sinks this answer is a story that ends at delivery (I sent the analysis and they agreed) or that casts the stakeholder as a villain. Make sure it names the disagreement, your mechanism for resolving it, and the outcome.
Follow-up
- What was the stakeholder's strongest objection, and how did you respond to it?
- What would you have done if they still said no after you presented?
- How did you know afterwards that the decision had been the right one?
Retract a published number after finding a currency bug
Two weeks ago you published an interchange and fraud analysis that summed amount_minor across fct_payment_authorization without converting currencies. Minor units are not two decimals everywhere: some currencies carry none and some carry three, so the sum has no interpretation. A pricing decision is already in flight on the back of it. You now have corrected figures. Produce the retraction: what you send, to whom, in what order, and what you change in the process so this class of error is caught next time rather than trusted next time.
Approach
- Size the error before announcing it, because saying the number is wrong without a magnitude and a direction forces every reader to assume the worst case.
- Check whether the conclusion actually flips: if the ranking that drove the pricing decision is unchanged, that belongs in the first sentence beside the correction rather than buried at the end.
- Tell the person acting on it first and directly, then the wider distribution, using the same text, so nobody learns about it secondhand.
- Write the correction as four parts: the old number, the cause in one clause, the effect on the pending decision, and the new number. Leave out self-flagellation, which makes the reader do emotional work instead of acting.
- Fix the class rather than the instance: a rule that a sum over amount_minor either groups by transaction_currency or passes through both conversion steps, exponent scaling and then a dated rate into one named reporting currency, plus a standing reconciliation of the settled subset to the settlement ledger inside each settlement_currency.
Follow-up
- The corrected figures do not change the decision. Do you still send the correction, and what does that choice signal?
- What automated check would have caught this, where would it live, and what would it cost in false alarms?
Defend a vintage finding that contradicts the portfolio dashboard
The lending dashboard shows blended 90-plus days-past-due falling for four consecutive quarters while originations grew 60 percent. Using fct_loan_performance_monthly, you build a vintage view keyed on origination_month by months_on_book and find the three most recent vintages are worse than their predecessors at the same age. The business lead presents that dashboard weekly and pushes back hard, suggesting you picked favourable cohorts. You get one meeting and the vintage table. Present the finding so it survives the cherry-picking objection and ends in a decision.
Approach
- Reconcile before you contradict: show that aggregating your vintage table along the calendar diagonal reproduces the published blended series, so the disagreement is about age mix rather than about data quality.
- Make the mechanism arithmetic rather than rhetorical: a loan cannot reach 90 days past due before it is 90 days old, so rapid origination growth shifts weight onto young months-on-book where the rate is structurally near zero.
- Show every vintage rather than a selected pair, all indexed at months_on_book equal to 12, with cohort sizes printed beside each curve so nobody can claim the divergence rests on a thin cohort.
- Handle restructuring explicitly, because restructured_flag resets days_past_due: count each loan on its worst pre-restructure state, or recent vintages will look better than they are.
- Close on the decision rather than the chart: state what the divergence implies for the cutoff or the channel mix, and state in advance what evidence would make you withdraw the claim.
Follow-up
- Two cohorts differ at month 12. How do you separate a seasoning effect from a genuine credit-quality effect?
- Someone argues the recent vintages are simply a broker-channel mix shift. How do you test that, and what would confirm it?
- 01
Tell us about a time you used your communication and collaboration skills to influence a stakeholder.
- 02
Describe a significant challenge you faced in a project and the specific steps you took to overcome it.
- 03
Tell us about a data science project you led; what were the results and what would you do differently?
- 04
How do you handle situations where your data-driven recommendations conflict with the intuition of senior leadership?
- 05
Tell me about a time you explained a model or test result to a non-technical audience and had to correct how they were reading it.
Is this an official 1st Central interview guide?
No. It is PracHub's own preparation material for the Data Scientist role, based on what candidates report. The stages and questions are not a published 1st Central process and can change, so confirm the current format with your recruiter.
PracHub interview research ↗How many rounds are there and how long does the process take?
Candidates report four stages (Initial Screening, Technical Assessment, Behavioral Assessment, Final Technical Deep-Dive) over roughly 3-5 weeks. One FAQ-style account describes a series of two formal conversations instead, so the structure may vary by team. Ask your recruiter for the exact steps, and tell them early if you have a deadline.
PracHub interview research ↗What should I prioritise for the technical rounds?
Candidates report A/B testing and statistics as core, along with SQL window functions (RANK, LEAD, LAG, SUM() OVER), cross-validation, boosting and model evaluation. Add product metrics and diagnosing a metric drop. The plan gives a day to each of these, with checkable output such as a SQL file and an A/A simulation.
PracHub interview research ↗Which languages and tools should I be comfortable with?
The role lists SQL as essential and Python or R for data manipulation, statistical analysis and machine learning. Prepare to write queries with joins and window functions, and to run a test, a cross-validated model and a power calculation in whichever of Python or R you choose.
PracHub Data Scientist practice ↗Do I need insurance experience?
Candidates report that prior experience applying data science to business problems is valued, for graduates and experienced hires alike, and that understanding the insurance context helps. You can prepare for that without prior insurance work: practise metrics for quote conversion, renewal retention and risk-score accuracy, and read 1st Central's public product pages.
PracHub interview research ↗How should I handle questions about my past projects?
Candidates are advised to explain context clearly and not assume the interviewer knows the details of past work. For each project, cover the problem, your specific contribution, the method and why you chose it over alternatives, the result, and what you would change. Be ready to defend your model or test choice, and to say how you reached a conclusion, not only what it was.
PracHub interview research ↗Sources & methodology 3 sources ↗
Official role evidence, timestamped platform data and clearly labeled preparation advice.
- 01PracHub interview research ↗
PracHub editorial research into this company and role, maintained with this guide. Candidate-reported, not an employer publication.
platform · Accessed 2026-09-22 - 02PracHub Data Scientist practice ↗
Cross-company practice questions for this role.
platform · Accessed 2026-09-22 - 03PracHub interview preparation framework ↗
The framework the preparation plan follows.
platform · Accessed 2026-09-22
Explore related engineering environments and compare the preparation focus.
3M Data Scientist Interview Questions & Guide 2026
Communication practice for 3M data scientist candidates: turn an analysis into a recommendation, name the uncertainty honestly, and answer the follow-up a skeptical stakeholder will raise.
Virtu Financial Data Scientist Interview Questions & Guide 2026
Take-home style practice for data scientists preparing for Virtu Financial: a messy dataset, a fixed time budget, and a written analysis, reviewed for whether it states its assumptions and whether a reader could reproduce your numbers.
USAA Data Scientist Interview Questions & Guide 2026
Data-quality drills for USAA data scientist prep: survivorship and selection bias, leakage through time-ordered features, and extreme values where provenance decides the call. A logging fault or unit slip gets corrected or dropped; a genuin
ValueLabs Data Scientist Interview Questions & Guide 2026
Causal-inference exercises for ValueLabs data scientist prep: difference-in-differences, matching, instrumental variables, and knowing when observational data simply cannot answer the question asked.