分享一下我准备AB testing 相关面试板块的checklist。 有用的话记得加米哦.--\nTarget Level：E6, Staff+, DS product analytics. check 1point3acres for more.\n\n面试Senior时，很多特别深入的方法是不会考的。如果恰好一个问题可以有non AB testing的方式解决， 可以提出来给自己加分。面试senior 看的是drive问题的能力，和完整解决问题，考量trade-off。一板一眼的准备实验“八股文”，就够了。\n\n而面试Staff+ 就会在实验方面特别特别细致和深入地问问题。对话方式就从“你怎么解决network effect的问题” 变成了 面试者需要自己提出 要注意的nuances，解决方法A,B,C,各自优缺点，根据问题本身我选择A 因为blablabla。理解问题，宏观架构framework，每一个component需要注意什么，trade-off and recommendation. 工作上的经验积累，和踩过的那些坑，都成了面试的很好作料。. Χ\n\n以下就是我自己梳理的实验问题都准备点什么，之后还会继续添加：\n\nConcepts. 1point 3 acres\nSignificance level = alpha: P(detect | H0)\nPower = 1-beta = P(detect | H1)\nSample Size\nSignificance level, power, MDE, variance\nMinimal Detectable Effect (pre-experiment). Small MDE needs larger sample\nEffect side = delta / std (post-experiment)\nPost-experiment power\nSignificant level, effect size, sample size, variance\n Why: experiment result not ss: (1) not enough power; (2) enough power, no impact\n.google и\nStats:\nRatio metrics:\nbinomial distribution, n ~\nDelta method. Waral dи,\nCount metrics: nomal distribution, n ~. ----\nt-test\n\nProportion:. Χ\nChi-square test\nSSRM: ~ Chi-square: (O - E)^2 / E^2. .и\n\n\nSimpson Paradox\n Slice by important segment to check for subgroup behavior\nMultiple Testing Problem\n Bonforroni → family wise error rate\n Holms → family wise error rate, less conservative\n False Discovery Rate → proportional False Positives\nNovelty effect\n Hold longer and trace cohort overtime\nResidual effect - previous iteration impact next iteration, e.g. cool-off. ----\n re-hash\nSelection bias\n Check Sample ratio mismatch: Chi-square test. From 1point 3acres bbs\nNetwork effect (2 sided marketplace)\n cluster-based randomization (rand on cluster)\n Ego-cluster randomization (rand on member)\n\nnon-AB testing methodologies\nDiff in diff. 1point 3 acres\nIV\nPropoensity Score Matching\nSynthetic Control\n\nReinforecement Learning.google и\nMulti-arm-bandit\nThompson Sampling. ----\n\nSequential Testing\nVariance Reduction methods.google и\n\nBayesian v.s. Traditional AB testing