Test classifier difference with McNemar's test

Q: Test classifier difference with McNemar's test

This question evaluates understanding of paired-proportion hypothesis testing and comparative classifier evaluation using McNemar's test, exact binomial inference, paired confidence intervals, and multiple-comparison considerations in the Statistics & Math domain for a Data Scientist role.

Q: How do I approach Statistics & Math interview questions?

Statistics & Math questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master statistics & math interviews.

Question

Paired Comparison of Two Classifiers via McNemar's Test

You evaluated two classifiers on the same 10,000 labeled examples and summarized the paired outcomes in a 2×2 table:

Both correct: n11 = 8,740
Both wrong: n00 = 740
A correct / B wrong: n10 = 300
A wrong / B correct: n01 = 220

Let b = n10 (A correct, B wrong) and c = n01 (A wrong, B correct).

Answer the following:

Using McNemar's test with continuity correction, test H0: the error rates are equal for A and B. Compute and show the intermediate numbers (b, c, |b − c|, b + c), the test statistic, and the p-value.
Compute the exact binomial two-sided p-value for the same H0 by conditioning on b + c. Explain when you would prefer the exact test over the asymptotic McNemar test.
Provide a 95% confidence interval for the paired accuracy difference (A − B). State which method you use and why.
Discuss the assumptions behind McNemar's test, when it is inappropriate, and how you would adjust for multiple testing if comparing A against 10 other models.

Test classifier difference with McNemar's test

Paired Comparison of Two Classifiers via McNemar's Test

Solution

Comments (0)

Test classifier difference with McNemar's test

Overview

Paired Comparison of Two Classifiers via McNemar's Test

Solution

Comments (0)