PII Detector Precision, Token-Sampled Document Length and Fake-Name Collisions

A data pipeline anonymizes customer datasets by replacing every real name with a fake one. This round asked three short probability questions about that pipeline. Give exact answers, and explain how you reached them. ### Clarifying Questions - In Part 1, does "just as good as the first" mean that the second detector also catches every PII token and also flags 1% of the other tokens? - In Part 3, are the $n$ real names all different, given that each real name always maps to the same fake? - In Part 3, if three real names end up with the same fake name, is that one collision or one for each pair? ### Part 1 — How often is a flag real PII? The first step is finding the names: a detector goes through a document token by token and flags whatever looks like personally identifiable information (PII). A document has 100 PII tokens and 100,000 tokens that are not PII. The detector catches all 100 PII tokens, and it also flags 1% of the other tokens. 1. Pick one of the detector's flags at random. How likely is it to be real PII? 2. A second detector, just as good as the first, is added, and a token is replaced only when both detectors flag it. Assume the two make their mistakes independently. Pick a token flagged by both at random. How likely is it to be real PII? ```hint Count what you pick from Turn the rates into numbers of flagged tokens of each kind, and look at which tokens make up the pool you are picking from. ``` ```hint Read the assumptions For the second question, ask what "just as good" and "independently" each tell you about which tokens both detectors flag. ``` #### What This Part Should Cover - The composition of the pool of flagged tokens in each scenario - Exact probabilities for both questions, and why they differ so much - The assumptions behind the second answer, and what requiring two flags costs ### Part 2 — Is the sampled document typical? Documents in a delivery vary enormously in length. To check how much PII was missed, a reviewer picks a token at random from the whole delivery and reviews the document it came from. Is that document longer than, the same length as, or shorter than the average document in the delivery? ```hint How documents get picked Ask how the chance that a particular document is chosen depends on its length. ``` #### What This Part Should Cover - How a document's chance of being chosen relates to its length - The expected length of the chosen document compared with the average, and when the two are equal - What this means for conclusions drawn from the review ### Part 3 — Expected fake-name collisions Assume finding the names is solved: every name that needs replacing has been located, and each real name always gets the same fake, so one company never turns into three. There is a list of $V$ fake names. For each of the $n$ real names in a dataset, the pipeline picks a fake from the list uniformly at random, without checking whether that fake has already been used. A collision is any pair of real names that end up sharing the same fake name. On average, how many collisions should you expect in a dataset of $n$ names? ```hint What to count Collisions are defined on pairs of real names. Ask whether you need to know how different pairs depend on each other to find the average. ``` #### What This Part Should Cover - A precise definition of what is being counted - The expected number of collisions as a formula in $n$ and $V$, with its derivation - The scale at which collisions become likely, and how the pipeline could avoid them ### What a Strong Answer Covers - Translates each scenario into counts or indicator variables before computing - Exact answers with short derivations and sanity checks - States the assumptions each answer relies on (perfect detection of PII, independent errors, uniform random choice) and what changes if they fail - Connects each result to a decision in the anonymization pipeline ### Follow-up Questions - The two detectors tend to make the same mistakes. What range of answers is possible for Part 1, Question 2, between fully overlapping and never-overlapping errors? - Suppose each detector caught only a fraction $r$ of the PII tokens, independently. What share of PII does the "both must flag" rule catch, and is that rule a good choice for anonymization? - How many fake names are needed to keep the expected number of collisions below one, and what is the probability of at least one collision? - How would you estimate the average amount of missed PII per document from reviews chosen through random tokens?