Design a Binary Text Classifier Using Only a Log-Probability Scoring Helper

Context

You are building a binary text classifier without fine-tuning. You have access only to a helper function:

• score_batch(prompt, inputs) → for each input string x in inputs, returns per-token probabilities (or log-probabilities) of x under the provided system prompt.

Assume:

• The same input x can be scored under different system prompts.
• The helper returns either probabilities or log-probabilities (you must handle both cases robustly).
• You can process inputs in batches.
• If the API exposes temperature or sampling controls, you may optionally adjust them.

Tasks

1. Construct system prompts for class A and class B.
2. For each input, query the helper with both prompts and produce a continuous score s in [0, 1] estimating P(class = A | x), not just a hard label.
3. If the helper returns log-probabilities, show how to compute a numerically stable score using log-sum-exp and log-prob normalization.
4. Define batching and how token-level probabilities are aggregated into a single sequence-level score.
5. Describe threshold selection and evaluation metrics (ROC-AUC, precision/recall, F1), including calibration.
6. Propose performance improvements: prompt engineering, repeated sampling, temperature control, prompt ensembling, score calibration, and handling class imbalance.
7. Discuss failure modes and how to validate/iterate offline without fine-tuning. Provide pseudocode and the inference-time complexity.