Calculate Probability of Honest and Relevant Chatbot Answers

Q: Calculate Probability of Honest and Relevant Chatbot Answers

This question evaluates understanding of basic probability (independence and joint event probabilities), expectation calculations for sample counts, and the design and interpretation of hypothesis tests for comparing models.

Q: How do I approach Statistics & Math interview questions?

Statistics & Math questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master statistics & math interviews.

Question

Chatbot Evaluation: Honesty and Relevance

Scenario

You are evaluating a customer-service chatbot. Define two events for any given answer:

H: the answer is honest, with P(H) = 0.7
R: the answer is relevant, with P(R) = 0.8 Assume H and R are independent.

Questions

What is the probability that an answer is both honest and relevant, P(H ∩ R)?
Given logs of 1,000 answers, how many would you expect to be neither honest nor relevant?
Describe how to run a hypothesis test to compare two LLMs’ relevance rates at significance level α = 0.05.

Calculate Probability of Honest and Relevant Chatbot Answers

Chatbot Evaluation: Honesty and Relevance

Scenario

Questions

Solution

Comments (0)