What makes the analysis hard here is the shape of the data rather than its volume. The account population is small by consumer standards, often a few thousand paying accounts, and revenue across it is extremely skewed, so the top fraction of a percent holds a large share of the total and any mean over accounts is effectively a statement about a handful of customers. Outcomes within an account are strongly correlated because colleagues share workspaces and influence each other, which both inflates apparent sample size and lets treatment leak between arms when randomisation is done at the user level. Contracts are annual, so churn is a rare, censored, calendar-driven event that most monthly metrics are structurally incapable of measuring. And the telemetry itself is largely machine-generated: a customer changing one line in a CI configuration can multiply request volume overnight without a single human deciding anything about the product.
The skills that carry the most weight are, in order, SQL across mismatched grains with an explicitly defended denominator, cohort and survival thinking applied to contracts rather than to individuals, and experiment design that survives clustering and heavy tails. Concretely that means as-of joins against a versioned subscription table, randomisation at the account level with the design effect accounted for in the power calculation, pre-registered winsorisation or a capped metric when the outcome is revenue, and variance reduction using pre-period usage as a covariate. Unit economics matters more than in most domains because infrastructure cost is a real and allocable per-account quantity, so being able to say which accounts are unprofitable and why is a routine expectation. The last skill is unglamorous and decisive: when the product number and the finance number disagree, being able to say which one is right, by what definition, and what the reconciling difference is.
Data science work in business software, infrastructure and developer tooling sits between a telemetry stream and a contract system, and most of the job is making those two agree. The unit of analysis is the account, not the person: an organisation signs, an organisation renews, and a hundred engineers inside it are correlated observations rather than a hundred independent ones. The recurring questions are narrow and concrete, covering which accounts will expand at renewal and which will contract, what a unit of usage costs to serve for a given account at its negotiated discount, whether a packaging change moved consumption or merely moved it between SKUs, and which accounts a capacity-limited customer-success team should contact this week. The deliverable is usually an account-level list with a threshold attached, handed to a team with finite capacity, so a ranking without an operating point is an unfinished piece of work. The audience is rarely another data scientist, so a definition also has to survive being repeated by someone who did not write it.
Account-grain SQL against versioned contracts
editorialQuery usage, membership and contract tables together at the account grain without fanning out rows, and resolve every contract question as of a date rather than as of now. This is the substrate for nearly every other analysis in the domain, because the subscription table is versioned and the usage tables are not.
What to demonstrate
- Writing an as-of join from a daily usage fact to fct_subscription_period, selecting the version whose term_start_date and term_end_date bracket the usage_date, rather than joining to is_current and silently backdating today's plan over last year's usage
- Joining a one-row-per-account dimension to a many-row-per-account fact without multiplying revenue, including the case where dim_account is type 2 and contributes several versions in the window
- Stating the denominator and its exclusions out loud before writing the SELECT, specifically is_internal accounts, non-production environments, service accounts and synthetic traffic classes
- Recognising when a question needs a rollup from fct_api_request and when it must come from fct_usage_daily, and explaining why the two will not reconcile exactly
How to prepare
- Write one query returning monthly net revenue, allocated COGS and gross margin per account, joined as-of to the contract version live during each month, and confirm the total ties to the sum of net_amount_cents
- Practise the type 2 dimension pattern until it is automatic: filter to the version live at the event timestamp, never to is_current, and be able to say what breaks when you get it backwards
- Build a query that separates interactive, CI, batch and synthetic traffic into columns for the same account and week, and look at how different the engagement story is under each filter
- Rehearse explaining, in two sentences, why request counts from fct_api_request and billable quantities from fct_usage_daily diverge, including retries, failed requests and restatement
Retention, expansion and contract cohort arithmetic
editorialBuild net revenue retention, gross logo retention and expansion rate from a versioned contract table, and defend each denominator. This is the numerical language the domain uses to talk about itself, and small definitional errors here change the headline number by tens of points.
What to demonstrate
- Constructing net revenue retention as a ratio of sums over a cohort frozen twelve months earlier, and explaining why the mean of per-account ratios gives a different and much noisier answer given that contraction is floored at zero while expansion is not
- Restricting logo churn to the renewal-eligible base and articulating why a monthly rate on the whole base is off by roughly the reciprocal of the annual renewal fraction
- Handling censoring properly when contracts have not yet reached their renewal date, including when a survival estimator is required instead of a simple rate
- Separating booked_at from term_start_date and knowing which one belongs in a sales-effectiveness question and which in a revenue question
How to prepare
- Compute net revenue retention two ways on the same data, ratio of sums and mean of ratios, and be ready to explain the gap from the shape of the expansion distribution
- Derive from first principles how much a naive monthly churn rate understates the truth when all contracts are annual, and be able to do that arithmetic aloud
- Work through the ways net revenue retention can rise while the business shrinks, and name the guardrail that catches each one
- Build a renewal-eligible cohort for a given month from fct_subscription_period, including the 45-day grace for late paperwork, and sanity-check that roughly a twelfth of the base appears
Experimentation with clustered, skewed outcomes
editorialDesign and read out experiments where the randomisation unit is an account, the sample is a few thousand clusters at most, and the outcome of interest is revenue or consumption with an extreme right tail. Most standard experimentation instincts are calibrated on large-sample consumer data and mislead here.
What to demonstrate
- Choosing the account as the randomisation unit and computing power with the design effect 1 + (m - 1) * rho, rather than quoting a user-level sample size
- Recognising when the experiment is simply not powerable on revenue and proposing a proximate outcome, a longer horizon, or a switchback design for a platform-level change that cannot be randomised across accounts at all
- Applying variance reduction with a pre-period covariate, for example CUPED using the account's prior consumption, and stating the approximate variance reduction as one minus the squared correlation
- Pre-registering the winsorisation or capping rule and being explicit that it changes the estimand rather than pretending it is a neutral cleaning step
How to prepare
- Run a power calculation for an account-randomised test on consumption, with and without the design effect, and note how much longer the clustered version has to run
- Practise the diagnostic for a heavy tail: plot the running sample variance against sample size and say what it means when it does not stabilise
- Prepare one clear explanation of interference inside a shared workspace and how it biases a user-level estimate toward zero
- Have a switchback or stepped-wedge design ready for an infrastructure change that is global by nature, including how you would handle carryover between periods
Usage telemetry hygiene and unit economics
editorialTurn raw machine-generated traffic into a defensible measure of human value, and attach cost to it. The gap between what the platform serves and what a customer values is wider in this domain than almost anywhere else, and allocating infrastructure cost to an account is a routine expectation rather than a finance task.
What to demonstrate
- Systematically excluding internal accounts, non-production environments, synthetic monitors, load tests, service-account traffic and retried requests, and being able to quantify how much each exclusion removes
- Computing gross margin at the account level from net_amount_cents and cogs_cents, and identifying structurally margin-negative accounts rather than reporting a blended rate that conceals them
- Establishing the metering settling time empirically from first_written_at against restated_at, and excluding an appropriate trailing window from every reported figure
- Explaining how a rising error rate can inflate metered volume through client retries, and designing a metric that does not reward that
How to prepare
- Take one week of request-level data and produce the same engagement number under four progressively stricter traffic filters, then be ready to argue for one of them
- Build the per-account margin distribution and answer, with a number, what share of revenue sits in margin-negative accounts
- Measure how much a given usage_date's total shifts between first write and settlement, and set a lag window from that measurement rather than from convention
- Write down the specific mechanism by which a platform outage can increase billable units, and the guardrail metric that would expose it
Renewal-risk and expansion scoring with an operating point
editorialBuild an account-level model that ranks renewal risk or expansion opportunity, with leakage-free features and a threshold tied to the capacity of the team that will act on it. The output is a worklist, so ranking quality at the top matters far more than calibration across the whole distribution.
What to demonstrate
- Constructing features strictly from data available before the prediction point, and spotting the leakage that comes from features updated retroactively, such as a churn reason code, a downgrade amendment or a support ticket opened after the renewal conversation began
- Choosing an evaluation metric that matches how the list is used, meaning precision or recall at the k the team can actually work, not a global AUC over thousands of accounts most of which will never be contacted
- Handling class imbalance and small positive counts honestly, since annual contracts produce few churn events per period, and knowing when the honest answer is that there is not enough signal to model
- Turning the score into a decision by attaching a threshold to the expected value of an intervention and the capacity constraint, and stating what happens to accounts below the line
How to prepare
- Build a leakage audit habit: for every feature, name the timestamp that guarantees it existed before the prediction date, and drop anything you cannot date
- Practise reporting precision at k for several plausible team capacities instead of a single headline number
- Prepare an argument for why a simple, inspectable model that a customer-success team will actually trust can outperform a stronger model they ignore
- Work out how you would evaluate whether the intervention itself worked, given that assignment of coverage is not random, and name the design you would ask for
7 candidate reports. Individual accounts describe a particular role and hiring cycle.
Cloudflare Software Engineer interview with Battleship coding and Austin office tour
My process took long enough that it started to feel like the goalposts were moving. It began with a hiring manager call, followed by a live coding round where I built a Battleship-style game. Then came a larger loop with an "orange cloud" behavioral interview, a debugging discussion, a system design conversation, and a PM-focused interview. I also had another hiring manager conversation that incl…
Read full experienceCloudflare Frontend Engineer interview with a repeated system design round
I went through an extensive sequence that started with conversations with the hiring manager and engineering manager about my background, then moved into several live coding and architecture or system design exercises. The later technical stages included an "orange cloud" behavioral interview, PM and cross collaboration rounds, and a final C-level executive conversation about why I wanted the job…
Read full experienceCloudflare Account Executive interview with a mock discovery call
From the recruiter call onward, the process moved quickly, though it was also confusing at times. I met multiple interviewers, including hiring managers and people from different functions. Some of them seemed impatient, as if they were looking for a specific "sales-ready" trait instead of trying to get to know me conversationally. The recruiter was nice, but the introductions and context weren’t…
Read full experienceCloudflare Solutions Engineer: missed interviews and scheduling problems
My process started normally enough with an interview with the hiring manager, followed by a live coding round that was scheduled for later. The people involved on the technical side seemed cool and friendly. I wasn't as nervous as I expected because the interviewers were engaged and explained what they worked on. The recruiter and coordination side made the experience much harder. There were sche…
Read full experienceCloudflare Account Executive interview with manager call and sales role-play
I started with a manager call, followed by a second-stage sales role-play. Product knowledge mattered a lot, and I needed to know Cloudflare's products well to do well. The role-play didn't feel like an easy warm-up. It was a genuine evaluation of how I thought and sold. The process was tough for me. I didn't move on after the sales role-play, so it ended earlier than I'd expected. My impression…
Read full experiencePracHub editorial advice for the preparation topics above.
Treating raw request or usage volume as engagement
Most traffic in this domain is emitted by machines. Continuous-integration pipelines, scheduled batch jobs, synthetic monitors, backfills and client retries can all grow by an order of magnitude from one configuration change made by one engineer, and none of it represents a new decision to use the product. The inversion is what makes it dangerous: when the platform degrades, clients retry, so error-driven retry volume rises at the exact moment the customer is most likely to leave, and an engagement dashboard built on raw counts shows growth immediately before a churn. Filter on traffic_class and on successful status before anything else, and keep failed-request volume as its own separate series.
Computing monthly churn against the entire customer base when contracts are annual
An annual contract has no opportunity to churn except at its renewal date, so an account that is eleven months from renewal is in the denominator while being incapable of appearing in the numerator. The resulting rate is smaller than the real one by roughly the ratio of the base to the renewal-eligible base, and it oscillates with the seasonality of when deals were originally signed rather than with anything about the customers. The corresponding trap on the other side is counting a churn on the date the record was updated rather than on term_end_date, which shifts losses into whichever month the operations team did its paperwork.
Generalising beyond the population the sample actually supports
State the frame the sample was drawn from and where it diverges from the population you want to talk about: time window, platform, geography, opt-in. If a group is excluded from the frame, either weight to known margins or narrow the claim rather than quietly extending it.
Answering a product-sense question with a list of features
Answer with a decision and the measurement that would settle it: the hypothesis, the primary metric, the guardrails, and the result that would make you not ship. A feature brainstorm cannot be wrong, which is exactly why it earns no points.
Choose a category, try a prompt, then open its approach, worked solution or follow-up when you need it.
Quantify billable volume created by retries after server errors
From fct_api_request (request_id, account_id, endpoint, idempotency_key, is_retry, http_status, request_at, billable_units), measure how much billable volume in a 28-day window is retry traffic that followed a 5xx. Group requests into attempt chains by (account_id, endpoint, idempotency_key) ordered by request_at; a request is error-driven if any earlier attempt in its chain returned 5xx. Requests with a null idempotency_key cannot be chained, so report them as their own class rather than assuming each is unique. Return billable_units split into first-attempt, error-driven retry, other retry and unchainable, per account.
Approach
- Split the population before measuring anything. A null idempotency_key is not a chain of one, it is an unknown; report its share of billable_units first, because if it is 40% of volume then the headline estimate is a lower bound and the deliverable has to say so.
- Within chainable rows, sort by (account_id, endpoint, idempotency_key, request_at) and derive 'any earlier attempt failed' with arithmetic rather than a per-group lambda: with is5 = (http_status >= 500), the per-chain cumsum minus the row's own value is positive exactly when an earlier attempt in that chain returned 5xx. A groupby-apply gives the same answer and is unusable at five million rows.
- Do not take is_retry as the definition. It is set by the client whenever an idempotency_key is resent, which covers retries after client-side timeouts and after 4xx as well; compute the flag yourself and then cross-tabulate it against is_retry, because the disagreement is a finding in its own right.
- Aggregate billable_units by (account_id, class) and assert the classes sum to each account's total. The spine sets billable_units to zero on 5xx responses, so the failed attempt contributes nothing and the whole inflation sits in the successful retry that follows it.
- Report the per-account share and look at its distribution, not the fleet total. One account in a retry storm dominates any blended figure, which is the same failure that makes a fleet-wide error rate useless.
Worked solution 40 min
- unchainable = df.idempotency_key.isna(); report df.loc[unchainable].groupby('account_id').billable_units.sum() before proceeding.
- keys = ['account_id','endpoint','idempotency_key']; c = df[~unchainable].sort_values(keys + ['request_at'], kind='mergesort'); c['is5'] = (c.http_status >= 500).astype(int)
- c['attempt_no'] = c.groupby(keys, sort=False).cumcount(); c['prior_5xx'] = (c.groupby(keys, sort=False).is5.cumsum() - c.is5) > 0
- c['cls'] = np.where(c.attempt_no == 0, 'first_attempt', np.where(c.prior_5xx, 'error_driven_retry', 'other_retry')); out = pd.concat([c, df[unchainable].assign(cls='unchainable')]).groupby(['account_id','cls']).billable_units.sum().unstack(fill_value=0)
Follow-up
- An account's error-driven share is 22%. Is that the platform's fault or the client's, and what do you look at next?
- How would you define a consumption-based north-star metric that an outage cannot inflate?
- Chains straddle the 28-day boundary. How large is that bias and in which direction?
Audit daily usage rows for grain and arithmetic violations
You are handed fct_usage_daily as a pandas DataFrame with account_id, workspace_id, sku_code, usage_date, billable_quantity, included_quantity_applied, overage_quantity, list_amount_cents, discount_amount_cents, net_amount_cents, cogs_cents, is_restated, first_written_at and restated_at. The declared grain is one row per (account_id, workspace_id, sku_code, usage_date). Write audit(df) returning a DataFrame with one row per failing check: check name, failing row count, and one example key. Cover at minimum grain duplication, negative quantities or amounts, the identity net = list - discount, billable = included + overage, and rows where is_restated is true but restated_at is null.
Approach
- Check the grain before anything else with df.duplicated(subset=key, keep=False), and count rows rather than groups so a key appearing twice contributes 2 — if the grain is broken every arithmetic count below it is uninterpretable.
- Express each invariant as a boolean Series over the whole frame. The cent columns are integers and compare exactly, so use !=; the numeric(18,6) quantity columns need np.isclose with atol=1e-6 because included + overage is a decimal sum.
- Handle null as its own failure mode. Comparisons against NaN return False, so a check written as rows_that_pass = (a == b - c) silently files every null-amount row wherever the negation happens to land; build each check as violations = ~condition | column.isna().
- Collect the checks as a list of (name, mask) pairs and assemble the output in one pass, so adding a check is one line and every check reports in the same shape.
- Order the output with structural failures (grain, null keys) above arithmetic failures, and report zero-count checks too — a check that silently disappears when it passes is indistinguishable from a check that was never run.
Follow-up
- Which of these should block a dashboard refresh and which should only warn?
- Rows with is_restated = true legitimately change value after first write. How do you make yesterday's audit result reproducible?
- How would you extend this to catch a partition that is missing entirely rather than wrong?
Bootstrap a confidence interval for net revenue retention
You have one row per account with arr_start_cents (ARR twelve months ago) and arr_end_cents (ARR today, zero if churned), covering the fixed cohort of accounts that had ARR twelve months ago. Net revenue retention is sum(arr_end_cents) / sum(arr_start_cents). Write a nonparametric bootstrap from scratch, without scipy.stats.bootstrap: resample accounts with replacement, recompute the ratio of sums on each resample, and return the point estimate with a 95% percentile interval from 10,000 resamples. Also report the interval you would get from the mean of per-account ratios, and explain the difference.
Approach
- Resample the account, because the account is the unit the estimand is defined over. One bootstrap draw is a vector of account indices and both numerator and denominator are recomputed from that same draw; resampling the two sides independently destroys the within-account correlation that makes a ratio estimator stable.
- Vectorise the draws: idx = rng.integers(0, n, size=(B, n)), then end[idx].sum(axis=1) / start[idx].sum(axis=1). A 10,000 by n index matrix is usually far cheaper than a Python loop; if the matrix is too large for memory, chunk over B rather than reverting to a loop.
- Take the interval from np.quantile(ratios, [0.025, 0.975]). The percentile interval differs from estimate +/- 1.96 * bootstrap SE whenever the resample distribution is skewed, which it will be here, and the skew is the thing you want represented.
- Compute the mean-of-ratios version on the same resamples, and state the exact relationship rather than guessing which of the two is larger. With r_i = arr_end_i / arr_start_i, the ratio of sums is the arr_start-weighted mean of exactly those r_i, so sum(end)/sum(start) - mean(r) = Cov(arr_start, r) / mean(arr_start) using the population covariance. The gap is positive when larger accounts retain and expand better than smaller ones, and negative when they do not; a cohort whose small accounts churn at a higher rate has positive covariance, which puts the mean of per-account ratios BELOW the ratio of sums. Requires arr_start_i > 0 for every account, which the fixed-cohort definition guarantees; r_i is floored at 0 and unbounded above, so a handful of 4x expansions among small accounts can flip the sign. Compute the covariance and report it instead of asserting a direction.
- Report the interval width beside the concentration of the cohort. If the largest account is 12% of starting ARR, a narrow interval is evidence that the resampling unit is wrong rather than evidence that the estimate is precise.
Follow-up
- The cohort has 800 accounts and the largest is 12% of starting ARR. How much do you trust a percentile interval here?
- How would you extend this to an interval on the year-over-year change in NRR?
- Two accounts merged mid-window and one contract was co-termed into the other. How do you keep the cohort fixed?
Consecutive qualifying weeks before renewal, as a ranked worklist
fct_api_request carries account_id, workspace_id, environment, request_at (timestamptz), http_status and traffic_class. fct_subscription_period carries account_id, term_end_date, auto_renew and is_current. A qualifying week for an account is an ISO week with at least 50 successful production requests in traffic_class ('interactive','batch'). Over the last 52 whole ISO weeks, and for accounts whose current term ends within 90 days, return the current qualifying-week streak length, the week it began, and the longest earlier streak. An account whose streak has broken must appear with a current length of zero.
Approach
- Bucket weeks as date_trunc('week', request_at AT TIME ZONE 'UTC'). request_at is a timestamptz, so an unpinned date_trunc silently uses the session time zone, weeks start at a local midnight, and Monday-morning traffic lands in the previous week for part of the fleet. Pinning UTC also makes the seven-day arithmetic below exact across daylight-saving transitions.
- Apply the exclusions before counting: environment = 'production', http_status < 400, traffic_class IN ('interactive','batch'). Then apply the volume floor and drop the current partial week, which can never meet a floor calibrated on whole weeks.
- Build islands with the row-number anchor: ROW_NUMBER() OVER (PARTITION BY account_id ORDER BY week_start) as rn, then week_start - rn * interval '7 days' is constant inside a run of consecutive weeks. Group by that anchor to get each streak's start, end and length.
- The current streak is the island whose end equals the last whole week; if none does, the account's current streak is zero and that is the interesting case. The longest earlier streak is the maximum length among the remaining islands.
- Join the renewal filter from the current subscription row and LEFT JOIN the streak summary so an account with no qualifying week at all still appears, rather than vanishing from the risk list precisely because it went quiet.
- Finish with an operating point. The list is worked by a team with finite capacity, so order it and cut it at that capacity, and say what happens to the accounts below the line.
Worked solution 40 min
- Build weekly_qualified: filter the fact on environment, status and traffic_class, group by account_id and date_trunc('week', request_at AT TIME ZONE 'UTC'), keep groups with count(*) >= 50, and exclude the in-progress week and anything older than 52 whole weeks.
- Add rn = ROW_NUMBER() OVER (PARTITION BY account_id ORDER BY week_start) and anchor = week_start - rn * interval '7 days'.
- Group by (account_id, anchor) to get streak_start = min(week_start), streak_end = max(week_start), streak_len = count(*).
- Per account, take current_len as the streak_len where streak_end = the last whole week else 0, current_start from the same island, and longest_prior as max(streak_len) over the other islands.
- Join to fct_subscription_period on is_current with term_end_date <= current_date + 90, LEFT JOIN the streak summary, and order by current_len ascending then term_end_date ascending.
Follow-up
- The floor of 50 requests was picked for this exercise. How would you calibrate it from data, and what would force a recalibration?
- A regional holiday week drops several accounts below the floor at once. How do you keep that out of the risk list?
- How would you evaluate whether contacting these accounts actually changed renewal, given that coverage is assigned deliberately?
Collapse retried requests into logical operations per account
fct_api_request carries request_id, account_id, api_key_id, idempotency_key (null when the caller supplied none), is_retry, request_at and http_status. For one ISO week, count logical operations rather than HTTP calls: requests sharing (account_id, api_key_id, idempotency_key) collapse to the earliest one, while every request with a null idempotency_key is its own operation. Return per account the raw request count, the logical operation count and the ratio between them, and state which direction a rising 5xx rate pushes that ratio.
Approach
- Split the stream before deduplicating. GROUP BY and PARTITION BY treat NULLs as equal to one another, while the = operator does not, so a single grouping pass folds every null-key request in an account into one 'operation' and can cut the count by orders of magnitude.
- Deduplicate the keyed rows with ROW_NUMBER() OVER (PARTITION BY account_id, api_key_id, idempotency_key ORDER BY request_at, request_id) and keep rn = 1. Including request_id in the ORDER BY makes the result deterministic when two rows share a millisecond, which they will.
- UNION ALL the null-key rows back in unchanged; they need no dedup and must not pass through the partition.
- Aggregate per account: count() over the raw week, count() over the union, and the ratio of the two. Report the ratio, not the difference, so accounts of very different sizes are comparable.
- Cross-check against is_retry, which flags a resend of the same idempotency key: raw minus logical should sit close to the count of is_retry rows, and a large gap means clients are resending without a key and the dedup is under-counting duplicates.
- Name the inversion explicitly: 5xx responses provoke client retries, so amplification rises exactly when reliability falls, and any engagement metric built on raw request counts will show growth during an outage.
Follow-up
- Which of these two counts belongs in the billable-units metric, and which in an engagement metric?
- An account's amplification ratio jumps from 1.05 to 3.0 in a day. Name three causes and the query that separates them.
Build an activation metric tree for a self-serve onboarding rebuild
The self-serve signup flow is being rebuilt. You have dim_account (account_id, created_at, is_internal, first_workspace_at, is_current) and fct_api_request (account_id, workspace_id, environment, api_key_id, http_status, traffic_class, request_at). Build a three-layer metric tree from signup through to production dependency, name the primary metric for this specific change, and state the fixed measurement window and the denominator for each layer. Two of the three layers share a denominator and the third does not, so say which invariant still holds across the tree once the denominators differ. Say which layer the onboarding team can actually move and which it cannot, and why raw signup counts are not on the tree.
Approach
- Fix the cohort once: accounts created in cohort period P with is_internal = false, collapsed to one row per account_id since dim_account is type 2. Say out loud that employee, demo and load-test accounts are removed, otherwise the tree partly measures internal testing.
- Layer one, qualified signup, on the cohort denominator: accounts recording at least one successful request (http_status < 400) with traffic_class in ('interactive','batch') within 14 days of created_at. This replaces raw signup counts, which measure registration-form friction and duplicate sign-ups from one organisation rather than demand.
- Layer two, activation, also on the cohort denominator: the share of the cohort whose first successful request with a non-null api_key_id arrives within a fixed 168 hours of created_at. Fixed window, not trailing, so that cohort weeks are directly comparable and no cohort is censored.
- Layer three, production depth, on a conditional denominator: the share of activated accounts holding a workspace with environment = 'production' whose trailing 7-day count of successful non-synthetic requests exceeds a floor calibrated from the data, for example the volume above which 90-day retention stops rising. A floor picked by intuition makes this a metric about the floor. Conditioning on activation is the right operational choice, because the question it answers is whether accounts that got started go on to depend on the product, but it means this rate is not on the same scale as the two above it: a conditional rate of 0.50 sits perfectly happily underneath a 0.30 activation rate. To place it back on the funnel, multiply it by the activation rate and report that product as the cohort share.
- Align the traffic_class filter across all three layers, otherwise the underlying account sets do not nest: an activation definition that admits traffic_class = 'ci' will count accounts that layer one excluded, and the tree stops being a funnel. With mixed denominators the invariant that survives is on account sets and therefore on counts, not on rates. Then name the primary metric for this change, the seven-day activation rate, because the onboarding flow controls the path to first successful call and nothing below it. Production depth is the lagging check that the activation gain was real rather than one sample request.
Follow-up
- Activation rises 6 points and the production-depth rate among activated accounts is flat 60 days later. Say what happened to the cohort share of production-depth accounts, and why a flat conditional rate is the better of the two readings available here.
- Median time-to-first-successful-call answers the same question with more information. Why is it a worse weekly dashboard metric, and what estimator would make it honest?
Design the retention metric suite for a quarterly board pack
You own the retention numbers in the quarterly board pack. Build them from fct_subscription_period (account_id, arr_cents, term_start_date, term_end_date, amendment_type, booked_at, is_current, superseded_by_id) rather than from invoices. Deliver net revenue retention over a trailing twelve months, the guardrails that stop it being satisfied by shrinking the business, and the reporting lag you will enforce. Specify the cohort rule, state whether you report a ratio of sums or a mean of per-account ratios and why, and name two ways the headline number rises while the business gets worse.
Approach
- Freeze the cohort at month M-12: the set of accounts with ARR above zero on that date. No account acquired after M-12 enters either side of the ratio. That constraint is the entire point, because it makes the number a statement about the installed base rather than about how well sales did last quarter.
- Read arr_cents from the fct_subscription_period version that was live on each of the two dates, selected by term_start_date and term_end_date bracketing the date, never by is_current, which silently rates last year's revenue at this year's plan. Invoiced amounts move with billing_frequency and prepayment and will not reconcile to a contract-derived figure, so pick one source and say which.
- Report the ratio of sums: total arr_cents at M over the frozen cohort divided by total at M-12. The mean of per-account ratios is a different estimand and much noisier on a few thousand accounts, because contraction is bounded below at zero while expansion is unbounded, so a handful of large expansions dominate the mean.
- Attach the two guardrails that close the two gaming routes. Gross logo retention on the renewal-eligible base catches raising the ratio by declining to serve the churn-prone segment, because it is computed only over accounts that actually had an opportunity to leave. New-logo ARR reported beside it exposes a contracting top of funnel that the ratio is structurally incapable of seeing.
- Enforce the lag and the restatement policy. A 45-day grace on term_end_date for late paperwork means the most recent 45 days are never reportable, and a month is openly restated when the grace closes rather than quietly corrected between board packs.
Follow-up
- Compute net revenue retention both ways on the same cohort. What does a large gap between the ratio of sums and the mean of per-account ratios tell you about the shape of the expansion distribution?
- A ramp deal is signed in March and starts in July. Which month does it belong to on the board's sales-effectiveness page, and which on this one?
Size an account-randomised test of a consumption feature
You have 2,800 non-internal paying accounts. The outcome is the monthly sum of billable_quantity for sku_code = 'compute_hours' from fct_usage_daily, with a mean of 640 and a standard deviation of 2,100 across accounts. Product wants to detect a 5% lift from a scheduling feature, randomised 50/50 at the account level, at 80% power and two-sided alpha 0.05. Give the minimum detectable effect the available sample actually supports, say whether the 5% target is reachable, and state what you would change in the design or the metric before agreeing to run it.
Approach
- Write the two-sample formula and plug it before discussing anything else: MDE = (z_0.975 + z_0.80) * sd * sqrt(1/n1 + 1/n2) = 2.80 * 2100 * sqrt(2/1400) = 222 compute-hours, which is 34.7% of the 640 mean. The driver is the coefficient of variation of 3.3, not the account count, so quoting a sample size without it is not a power calculation.
- Invert for the stated target. A 5% lift is 32 compute-hours, needing N = 4 * (2.80 * 2100 / 32)^2 = about 135,000 accounts, roughly 48 times the eligible population. Say the test is not powerable on this metric, and do not offer to run it longer: the account population is fixed, and extra weeks add repeated observations on the same clusters rather than new clusters.
- Attack the variance instead of the sample. Pre-register winsorisation at the 99th percentile of the pre-period distribution, or move the estimand to a bounded one such as the share of accounts exceeding a usage floor. State explicitly that this changes the estimand rather than cleaning the data, because a cap chosen after seeing the result encodes the answer.
- Add CUPED on the account's pre-period compute_hours. At a pre/post correlation of 0.80 the residual variance multiplier is 1 - 0.80^2 = 0.36, the standard error falls to 0.60 of its unadjusted value, and the MDE drops from 222 to 133 compute-hours (20.8% relative). That is still four times the requested 5%.
- Fix the randomisation unit and the balance in the same breath: randomise account_id because colleagues share workspaces, and stratify assignment on pre-period usage decile so that a handful of very large accounts cannot land disproportionately in one arm and dominate the difference in means.
- Close with an operating decision rather than a number. Run only if the team will act on a roughly 20% MDE after CUPED and winsorisation; otherwise re-specify the primary outcome as a bounded indicator and treat compute-hours as a secondary reported with intervals only.
Worked solution 25 min
- Compute the coefficient of variation, 2100 / 640 = 3.28, and check the top-decile share of total compute_hours before treating the mean as a summary of anything.
- Compute MDE = 2.80 * 2100 * sqrt(1/1400 + 1/1400) = 2.80 * 2100 * 0.0378 = 222 compute-hours, 34.7% relative.
- Invert to the required N for a 32-hour effect: 4 * (5883 / 32)^2 = about 135,000 accounts, 48x the population.
- Recompute under CUPED at rho = 0.80: variance multiplier 0.36, SE multiplier 0.60, MDE 133 compute-hours, 20.8% relative.
- Recompute on the binary variant at base rate 0.46 and write the recommendation as one decision with one operating point.
Follow-up
- The feature only applies to the 310 accounts on committed_consumption contracts. What is the MDE on that subset, and does it change the recommendation?
- Suppose the primary outcome becomes a binary indicator, account exceeded 100 compute-hours in the month, at a base rate of 0.46. What is the MDE now, and why did it improve so much?
- What would make you stop this test early, and how do you write that rule so it is not peeking?
Blended gross margin fell while every segment improved
Blended gross margin across paying accounts fell from 71 percent to 66 percent over two quarters, yet margin improved inside every plan_tier and every deployment_model. You have fct_usage_daily (account_id, sku_code, usage_date, net_amount_cents, cogs_cents), fct_subscription_period (account_id, plan_tier, term_start_date, term_end_date) and dim_account (account_id, deployment_model, is_internal, effective_from, effective_to). Produce an exact decomposition of the five-point move into mix, rate and interaction terms, and name the segment shift that carries it.
Approach
- Write blended margin as a revenue-weighted average of segment margins, m = sum over i of w_i * m_i, where w_i is the segment's share of net_amount_cents. Margin is a ratio of sums, so only revenue weights reproduce the blended figure.
- Apply the exact three-term decomposition: delta_m = sum(delta_w_i * m_i0) + sum(w_i0 * delta_m_i) + sum(delta_w_i * delta_m_i), read as mix, rate and interaction. It is exact by construction, so the three terms must sum to the observed change with no residual.
- Build segments from as-of joins: take the fct_subscription_period version whose term brackets each usage_date and the dim_account version whose effective_from and effective_to bracket it, rather than joining on is_current and backdating today's attributes over last year's usage.
- Run the decomposition on more than one segmentation, at minimum plan_tier, deployment_model and sku_code, since the mix that actually moved may not be the one anybody already suspected.
- Once the driving segment is identified, determine whether the shift came from acquisition landing new accounts in a lower-margin segment or from existing accounts migrating, because those have different owners and different remedies.
Follow-up
- The interaction term is sometimes large. What does it mean operationally, and when would you prefer a log-mean decomposition that distributes it across the other two terms?
- Revenue-weighted average margin is one construction. What changes if leadership wants margin weighted by account count, and which question does each version answer?
Four days spend equal time on query work, statistics, modelling and product judgement at deliberately shallow depth, which produces a scored map of where you actually stand. The last three days spend everything on the two areas the role weights most, and close by re-running day one to measure movement.
Prepare, practise & reflect
One practical outcome each day. Spend longer where you need it.
0 / 7 done01Breadth pass: query fluency
- Solve six prompts spanning aggregation, joins, window functions and date arithmetic in 60 minutes total, stopping at 10 minutes each whether or not it works, and mark every prompt as solved, solved slowly, or stuck.
- For each unsolved prompt write the single blocking sentence (I lost the grain, I did not know the frame clause, I could not express the date boundary) instead of reading the solution.
- Translate one pandas transformation you know well into SQL and one SQL query into pandas, checking that both return the same row count and the same totals.
Deliverable: A scored six-row table, one line per prompt, saved for the day-seven re-run.
Practice prompt ↗Practice prompt ↗Worked solution ↗02Breadth pass: statistics and inference
- Answer ten short questions in writing with nothing open: what a p-value is conditional on, what a 95 percent interval covers across repeated samples, when a paired test is the right one, what the bootstrap estimates, why multiple comparisons inflate false positives, how controlling the family-wise error rate differs from controlling the false discovery rate, what power depends on, what a missed real effect costs a product, the three situations where the central limit theorem does not rescue you (small n, very heavy tails, dependent observations), and what a standard error is the standard deviation of.
- Grade yourself against a reference and count only the answers that were exactly right, not the ones that were nearly right.
- Rewrite the two weakest answers the following morning from memory in full sentences.
Deliverable: Ten graded answers with an honest count of exact hits.
Practice prompt ↗Practice prompt ↗03Breadth pass: modelling
- Take one tabular dataset end to end in 90 minutes: a leakage-safe split, a baseline that is not a model (majority class or historical mean), one regularized linear model, one gradient-boosted tree, and a single evaluation metric chosen before you look at any result.
- Write why that metric fits the cost structure: precision at a fixed recall for alerting, calibration for anything feeding a price or a threshold, ranking metrics for retrieval, and note that area under the ROC curve is insensitive to class balance in a way that can flatter a rare-positive problem.
- Name the leak you were most likely to introduce (an encoding fit on all rows before splitting, or a feature computed after the label's timestamp) and write the check that would have caught it.
Deliverable: A notebook whose first cell states the metric and the baseline, plus two lines on what beat what and by how much.
Practice prompt ↗Practice prompt ↗Worked solution ↗04Breadth pass: product judgement
- Answer three case prompts aloud at 15 minutes each, timing how long passes before you state a success metric.
- For one case write the first segmentation you would run and the row counts you expect per segment, so that a tiny segment cannot quietly drive the conclusion.
- Take a metric definition you did not write, from a public dashboard, a textbook, or documentation you already have open, and list every place two analysts implementing it would diverge: which rows the denominator admits, whether the unit is an account or a person, what the time window is anchored to, and what happens to data that arrives late. Then write the one question that would close the largest of those gaps.
Deliverable: Three recorded case answers plus an ambiguity list for a metric someone else defined, ending in the single question you would ask about it.
Practice prompt ↗Practice prompt ↗05Depth, first area
- Rank the four areas by how many bullet points in the role description each one covers, pick the top one, and spend the entire day inside it.
- Work the six hardest problems you can find in that area and for each write the generalizable move you should have reached for first, rather than the answer.
- Re-solve the two you failed the same evening with notes closed.
Deliverable: Six generalizable moves written as instructions to yourself, not as solutions.
Practice prompt ↗Practice prompt ↗Worked solution ↗06Depth, second area, and the seam between them
- Repeat the depth protocol on the second-ranked area with the same six-problem structure.
- Construct one problem that requires both areas at once, for example a metric redefinition whose effect you must validate with a test whose readout you then have to query.
- Solve your own combined problem end to end and note where the handoff between the two areas cost you time.
Deliverable: One combined problem, solved end to end, with the handoff failure written down.
Practice prompt ↗07Integration and re-measurement
- Re-run the six prompts from day one under the same clock and compare both correctness and time.
- Run a 60-minute mixed mock that moves between areas without warning, since switching cost is what breadth passes do not train.
- Write the two areas you would still fail on, and the sentence you will use in the interview when you hit one of them.
Deliverable: A before-and-after score table plus a written plan for the two remaining gaps.
Practice prompt ↗Worked solution ↗Expand any day for tasks and deliverables. Your progress is saved on this device.
Have two ready. In one, the data was on your side and you had to move someone who outranked you. In the other, the pushback was correct and you changed position. The second is the harder story and it lands better, because it shows you separate being right from being attached to an answer. Name the person's actual objection.
Announce a metric fix that cuts the headline number
Weekly active organisations, the count on the company dashboard, has never excluded rows where dim_account.is_internal is true, and it counts traffic with traffic_class in synthetic_monitor and load_test. Correcting both reduces that count by 11 percent and removes most of the growth reported over two quarters. The figure appears in a board deck and in two teams' quarterly goals, one written on the count and one on the weekly active organisation ratio, whose denominator is accounts whose account_status was in ('trial','free','active_paid') through the week. Decide the order in which you tell people, what the dashboard shows during the transition, and what you propose happens to goals already set against the old definition.
Approach
- The interviewer is probing whether you can land a correction as an operational change with a plan attached, rather than as an announcement other people then have to clean up after.
- Quantify each exclusion separately before telling anyone: internal accounts, synthetic monitors, load tests. Three known quantities are a discussion; one alarming total is an argument.
- Be precise about which side of the metric each exclusion touches, because one team's goal is on a count and the other's is on a ratio. The traffic-class filters remove requests, so they shrink the numerator only. Dropping internal accounts removes them from the ratio's denominator as well, since internal accounts carry ordinary account_status values and therefore sit in that denominator. Internal accounts are active in almost every week while the real base is not, so the numerator loses a larger share than the denominator and the ratio falls by less than the count does. Compute both and say which one the 11 percent is before anybody assumes.
- Check whether the trend changes, not only the level. A constant 11 percent shift is a rebasing and nothing more. A shift that widens over time means the reported growth was partly internal or synthetic, which makes the existing goals unachievable as written and changes what you are asking teams to do.
- Sequence the disclosure: the metric owner and the two teams whose goals move first and privately, then the board channel with a written bridge, then the dashboard. The dashboard is last because a number that changes without explanation is read as instability rather than as a fix.
- Run both series for one reporting period with the bridge visible, restate history rather than letting the series break at a date, and set the date the old series is removed.
- Propose the goal treatment yourself: rebase each target by the shift measured on the metric that target is written against, rather than leaving each team to negotiate individually, which is where corrections of this kind usually die.
Follow-up
- One team's quarterly goal is now unreachable. Rebase the target or let it miss, and what does each choice teach the organisation?
- How would this have been caught when the metric was first defined?
- What else on that dashboard shares this failure mode, and how would you find out this week?
Disagree with a product manager about an adoption claim
A product manager is about to present that a new SDK release drove a 40 percent rise in requests among adopting accounts, computed from fct_api_request counts grouped by sdk_version. You find the rise is concentrated in traffic_class equal to ci, that rows with is_retry true grew alongside it, and that restricting to interactive non-retry traffic leaves a 3 percent lift. The launch review is in two days. Decide how you raise this, with whom and in what order, and what you propose the claim becomes.
Approach
- The interviewer is probing whether you can correct a colleague without ambushing them, and whether your own counter-analysis carries the caveats theirs lacked. Go to the product manager privately before the review. A correction delivered in the room is a status move and loses the argument you are actually trying to win.
- Bring a decomposition rather than a verdict: the same accounts and window, requests split by traffic_class with retries held out as their own column, so their 40 percent and your 3 percent reconcile line by line and neither has to be taken on trust.
- Reproduce their figure exactly first. If you cannot land on 40 percent with their method, you do not yet know what you are disagreeing with.
- Ask whether the continuous-integration lift is itself valuable. An account wiring the SDK into its pipeline has increased integration depth, which is the dominant switching cost in this domain, so the honest claim may be that integration depth rose while interactive usage moved 3 percent. Improving the claim beats deleting it.
- Name the mechanism that makes the raw count dangerous: clients retry when the platform degrades, so retry volume climbs exactly when the customer is most at risk. Pull the 5xx rate for the same accounts and window before anyone concludes anything, and note that billable_units is zero on 5xx rows, so request counts and billable quantities diverging is itself the signal.
- Close with a standing definition for launch metrics so the next release does not repeat the exercise.
Follow-up
- The product manager argues that continuous-integration traffic is real usage and declines to split it out. Is that position defensible, and under what metric definition?
- Suppose the 5xx rate for those same accounts also rose 40 percent. What is the claim now?
- The review happens and the raw number is presented regardless. What do you do next, and what do you not do?
State the measured impact of your own work honestly
You are asked for the business impact of a renewal-risk worklist you shipped nine months ago. Customer success used it, and renewals in the covered segment came in 6 points above the prior year. Coverage was assigned by the team itself: they worked the top of your list and also the accounts they were already worried about. Produce the impact claim you are willing to defend, the number you refuse to claim, and the design you would have asked for at the start.
Approach
- The interviewer is probing whether you can separate what you shipped from what you caused, and whether you would have built the measurement in rather than reconstructing it afterwards. Both halves are being scored.
- Name the confound precisely. Coverage assignment is doubly selected: the largest accounts get an owner because they are valuable, and distressed accounts get one because they are at risk. The naive covered versus uncovered comparison mixes a strong positive selection with a strong negative one and can come out with either sign depending on which rule dominated. Matching on account size does not fix it, because the risk signal that triggered coverage is the same signal that predicts the outcome.
- Split the claims by what each needs to be true. Ranking quality is defensible from precision at k on out-of-time renewals. Adoption is defensible from timestamps showing what share of listed accounts were contacted. The outcome claim is not defensible without a design, and saying so is the point of the exercise.
- Look for identification before giving up on it. A capacity cut-off, a territory boundary, or a period in which the list existed but was unstaffed can assign coverage for reasons unrelated to account health, and any of those supports a bounded estimate.
- State the design you would ask for now and its price: a randomly withheld slice of the list, held for two renewal quarters, with the expected cost in renewals stated openly. That cost is what it takes to be able to answer this question at all.
- Give a bounded number rather than none. Six points with an explicit statement of how much of it you can attribute is more useful than either claiming the whole figure or declining to quantify anything.
Follow-up
- Your manager wants the 6 points in a promotion packet. What wording do you accept, and what do you strike?
- What would have had to be true for the naive covered versus uncovered comparison to be valid?
- If the holdout costs the team real renewals, how do you justify asking for it, and to whom?
- 01
Weekly active organisations, the count on the company dashboard, has never excluded rows where dim_account.is_internal is true, and it counts traffic with traffic_class in synthetic_monitor and load_test. Correcting both reduces that count by 11 percent and removes most of the growth reported over two quarters. The figure appears in a board deck and in two teams' quarterly goals, one written on the count and one on the weekly active organisation ratio, whose denominator is accounts whose account_status was in ('trial','free','active_paid') through the week. Decide the order in which you tell people, what the dashboard shows during the transition, and what you propose happens to goals already set against the old definition.
- 02
A product manager is about to present that a new SDK release drove a 40 percent rise in requests among adopting accounts, computed from fct_api_request counts grouped by sdk_version. You find the rise is concentrated in traffic_class equal to ci, that rows with is_retry true grew alongside it, and that restricting to interactive non-retry traffic leaves a 3 percent lift. The launch review is in two days. Decide how you raise this, with whom and in what order, and what you propose the claim becomes.
- 03
You are asked for the business impact of a renewal-risk worklist you shipped nine months ago. Customer success used it, and renewals in the covered segment came in 6 points above the prior year. Coverage was assigned by the team itself: they worked the top of your list and also the accounts they were already worried about. Produce the impact claim you are willing to defend, the number you refuse to claim, and the design you would have asked for at the start.
Are these confirmed Cloudflare interview questions?
No. Every prompt here is an original PracHub practice exercise written for the Data Scientist role and for B2B software and infrastructure data problems. This guide does not claim to reproduce Cloudflare's interview questions, rounds or hiring timeline. Confirm the actual format, team and scope with your recruiter.
PracHub Data Scientist practice ↗What should I ask the interviewer?
Ask things whose answers would change whether you accept the job. How decisions get made when the data is ambiguous. What happened the last time an experiment contradicted a senior person's intuition. What share of the team's work is ad hoc versus owned. How the previous person in this role was evaluated. Skip anything the careers page answers. You are also being read on what you thought was worth asking about.
PracHub Data Scientist practice ↗How much SQL is enough for a data scientist interview?
Enough to write a correct multi-table query under time pressure with no reference open. That means joins including anti-joins, aggregation with HAVING, window functions such as ROW_NUMBER, LAG and running sums, date bucketing, and conditional aggregation with CASE inside SUM. Recursive CTEs and query tuning rarely decide a loop. The usual failure is not exotic syntax but silently changing the row count with a fan-out join.
PracHub Data Scientist practice ↗Is a portfolio project worth building before I interview?
Only if you can defend every choice in it for half an hour: why that data, what you tried and abandoned, how wide the error bars were, what you would do differently now. One project you can discuss at that depth beats five polished notebooks you have not thought about since. For most candidates, practice reps on SQL, cases and metric design convert to offers faster than building something new.
PracHub Data Scientist practice ↗Sources & methodology 2 sources ↗
Official role evidence, timestamped platform data and clearly labeled preparation advice.
- 01PracHub Data Scientist practice ↗
Cross-company practice questions for this role. Not an employer question record.
platform · Accessed 2026-09-22 - 02PracHub interview preparation framework ↗
The shared preparation framework these checkpoints and the seven-day plan follow.
platform · Accessed 2026-09-22
Explore related engineering environments and compare the preparation focus.
Zscaler Data Scientist Interview Questions & Guide 2026
Data-quality drills for Zscaler data scientist prep: survivorship and selection bias, leakage through time-ordered features, and extreme values where provenance decides the call. A logging fault or unit slip gets corrected or dropped; a gen
ZScalar Data Scientist Interview Questions & Guide 2026
Communication practice for ZScalar data scientist candidates: turn an analysis into a recommendation, name the uncertainty honestly, and answer the follow-up a skeptical stakeholder will raise.
Zeta Data Scientist Interview Questions & Guide 2026
Data-quality drills for Zeta data scientist prep: survivorship and selection bias, leakage through time-ordered features, and extreme values where provenance decides the call. A logging fault or unit slip gets corrected or dropped; a genuin
Zebra Technologies Data Scientist Interview Questions & Guide 2026
Causal-inference exercises for Zebra Technologies data scientist prep: difference-in-differences, matching, instrumental variables, and knowing when observational data simply cannot answer the question asked.