Naive Bayes Interview Questions: Smoothing, Log Probabilities, and Text Features

A Naive Bayes interview often starts with Bayes' rule and quickly becomes an implementation test: what happens to a class score when a word never appeared, why do long documents underflow, and which text representation matches the likelihood model? A useful answer does more than recite conditional independence. It computes a tiny example, names the denominator in smoothing, and keeps the vocabulary and validation split fixed. This guide gives you a four-document fixture you can calculate by hand, a comparison of Multinomial and Bernoulli models, and failure checks for a real classifier.

For broader context on training-versus-test boundaries, see PracHub's feature leakage interview exercise. Evidence boundary: Model equations and API behavior below follow the official scikit-learn Naive Bayes guide. The documents, scores and interview drills are original editorial examples. They are not presented as reported questions from an employer, and no model benchmark was run for this post.

State the scoring rule and its assumption

Question: What is Naive Bayes actually computing? For class c and feature vector x, classification compares a prior P(c) multiplied by class-conditional feature likelihoods. The simplifying assumption is conditional independence of features given the class. For count-valued text features in Multinomial Naive Bayes, that yields a score proportional to P(c) × product_w P(w|c) ^ count(w,x). The scikit-learn guide gives the corresponding parameter estimate and explains why multinomial and Bernoulli variants represent different event models.

The assumption is plainly imperfect for language: “not good” and “good” are related, and correlated tokens can be counted as if they were independent evidence. Yet an imperfect generative assumption can still produce a useful classification baseline. Do not infer that all predicted probabilities are well calibrated merely because the classifier calculates a posterior-like score; scikit-learn cautions about Naive Bayes probability estimates. Decide whether the interview needs a class ranking, a calibrated risk score, or a thresholded action.

State the feature event before the formula. If x_w is a word count, repeated occurrences matter. If it is a binary presence flag, absence may matter too. If it is a continuous measurement, a Gaussian feature model may be a candidate. “Naive Bayes” without a variant is underspecified; a strong candidate asks for the representation and then chooses the likelihood.

Compute a four-document Multinomial example

Original exercise: Two spam documents contain win win and win prize. Two ham documents contain team meet and team plan. The training vocabulary is {win, prize, team, meet, plan}, so V=5. Count documents for class priors and tokens for Multinomial likelihoods. With equal class counts, both priors are 2/4=1/2.

The official MultinomialNB equation is (N_wc + alpha)/(N_c + alpha × V). At alpha=1, each class denominator is 4+5=9. For the message win prize, the unnormalized spam score is (1/2) × (4/9) × (2/9) = 4/81. The ham score is (1/2) × (1/9) × (1/9) = 1/162. The spam-to-ham ratio is eight, so spam wins. You can reach the same decision by comparing log scores; there is no need to compute a normalized posterior merely to select the larger class.

Now remove smoothing. The ham likelihood of win is zero, so any ham score for a document containing win becomes zero regardless of other evidence. That is the zero-frequency problem. Smoothing supplies a nonzero estimate to vocabulary terms unseen in a class; it does not prove the unseen word is probable in that class. The value of alpha is a modeling choice to tune on validation data, not a universal constant selected because “Laplace means one.”

Make the unseen-word rule explicit

Question: What if the test message contains bonus, a word absent from every training document? bonus is not in the fixture's training vocabulary. In a fixed CountVectorizer vocabulary, it contributes no feature and thus no factor to the fitted MultinomialNB score. That is different from win being in the vocabulary but absent in the ham class; the latter has a smoothed ham likelihood. Mixing these two cases is a common error in hand calculations.

If you instead expand the vocabulary using the test message, you have changed the feature space and the smoothing denominator after training. For a proper evaluation, fit the vocabulary on training data only and transform validation or test data with the same fitted vectorizer. The scikit-learn common-pitfalls guide recommends fitting preprocessing only on the training partition; a Pipeline makes the split boundary easier to respect.

In production, a high share of out-of-vocabulary tokens can indicate new jargon, languages, product names, or adversarial spelling. The appropriate response depends on the use case: revise tokenization, use character n-grams, retrain on reviewed examples, or add an explicit unknown-token feature under a designed vocabulary policy. Do not silently claim that add-one smoothing solves words never represented in the feature matrix.

Why sum logarithms rather than multiply probabilities?

Question: Why do implementations work in log space? A document can have hundreds of nonzero token factors, all below one. Repeated floating-point multiplication can underflow toward zero. Logarithms turn the product into a sum: log P(c) + sum_w count(w,x) × log P(w|c). Because log is monotonic, the larger score still wins. This is both a numerical-stability measure and a convenient way to inspect the contribution of a term.

For the fixture, the spam log score is log(1/2)+log(4/9)+log(2/9) and the ham log score is log(1/2)+2log(1/9). Their difference is log(8), which encodes the same eight-to-one unnormalized ratio. The log score is not itself a probability; exponentiating one class's log score without normalization will not generally give P(c|x). If calibrated probabilities matter, evaluate them on a held-out set and use an appropriate calibration procedure; do not confuse a mathematically normalized model output with reliable real-world risk.

An implementation answer should also handle an empty transformed document. When all tokens are outside the vocabulary or filtered out, the feature contribution is zero, so the model falls back to comparing class priors (subject to the implementation's exact conventions). That behavior should be tested deliberately because it may produce a confident-looking label for an input the vocabulary cannot describe.

Multinomial, Bernoulli, and TF-IDF are different choices

Question: When would Bernoulli Naive Bayes differ? BernoulliNB models binary occurrence: whether a term appears at all. A repeated win does not count twice after binarization. Bernoulli scoring also includes likelihood factors for features that are absent. On our fixture, win appears in both spam documents, so with add-one smoothing P(win present|spam)=(2+1)/(2+2)=3/4; it appears in neither ham document, so P(win present|ham)=1/4. This denominator counts documents and two possible outcomes, not total tokens and vocabulary size. The BernoulliNB documentation makes this event-model distinction explicit.

For short, binary indicators such as “contains suspicious link,” Bernoulli may be a natural baseline. For bag-of-words counts, Multinomial is a natural baseline. Neither conclusion says one always wins. Compare them on the same training and validation partitions and metric. If you use TF-IDF, note that the Multinomial event model is literally based on counts, while scikit-learn documents that fractional TF-IDF features can work in practice. Present that as an empirical modeling choice rather than pretending TF-IDF is a count-generating process.

The input pipeline matters as much as the estimator. A tokenizer that removes not may reverse sentiment evidence. A vocabulary capped by frequency can drop rare but high-signal words. A case-folding or character n-gram policy can change which terms are recognized at inference. For interview answers, give one example of a preprocessing decision and one failure it could cause.

Evaluate the classifier that the product will use

Question: Is accuracy enough for spam? If 98% of messages are ham, predicting ham for everything yields 98% accuracy and catches no spam. Decide whether missed spam or false-positive blocking is costlier. Measure precision, recall, PR-AUC where appropriate, confusion-matrix counts, and performance by language, sender cohort, or other relevant slice. Choose a decision threshold on validation data and keep the final test set for an unbiased check. The scikit-learn classification-metrics guide defines these measures; the cost choice is a product judgment, not a property of Naive Bayes.

The split unit must match the deployment risk. Duplicated messages or near-identical campaigns shared between train and test inflate apparent generalization. If the task is future spam, use a temporal holdout and avoid learning vocabulary or tuning alpha from that future interval. If the task is a new sender, hold out senders. Report both an overall metric and the failure slice that motivated the model.

When the model degrades, trace the pipeline in order: raw document, tokenization, recognized feature IDs, class prior, smoothed per-class log likelihoods, final score, threshold and action. This isolates “unknown vocabulary” from “prior shifted,” and both from “label definition changed.” Merely increasing alpha without finding the failure stage can hide the problem.

An interview-ready implementation outline

Given a request to implement MultinomialNB from scratch, first freeze the vocabulary from training documents. Count documents per class for priors, and token frequencies per class for likelihoods. Allocate a vector of length V per class, add alpha to every token count, and divide by N_c+alpha V. Store log priors and log likelihoods. At prediction, count only known tokens in each document, take a dot product with the class log-likelihood vector, add the class log prior, and choose the maximum. Decide tie behavior and empty-document behavior explicitly.

Test the four-document fixture above, then add three cases: a word seen only in one class, a test-only unknown word, and a repeated known word. Their outputs should distinguish smoothing, vocabulary freezing, and Multinomial counts. If classes are imbalanced, show how priors change even when token likelihoods do not. The fixture is deliberately small enough that you can inspect every intermediate quantity rather than trusting one overall accuracy value.

Do not claim this outline handles every use case. Sparse matrices matter for realistic vocabularies; dense class-by-vocabulary arrays may still be manageable, but dense document matrices usually are not. Calibration, concept drift and adversarial text require separate evaluation. A concise interview answer can state those limitations without derailing the basic algorithm.

Practice five related PracHub questions

The first two public questions directly exercise Naive Bayes implementation. The others extend the answer into text classification, system design and evaluation. Their existence does not imply any employer uses the exact exercise in this guide.

Start with Implement Naive Bayes classifier from scratch. If you can produce the 4/9 likelihood, explain why bonus contributes no feature, and compare the two log scores, you can defend the essential implementation instead of relying on a memorized formula.

Sources and further reading

• scikit-learn: Naive Bayes user guide
• scikit-learn: MultinomialNB API reference
• scikit-learn: BernoulliNB API reference
• scikit-learn: Common pitfalls and data leakage
• scikit-learn: Classification metrics