Design a Post-Training Pipeline to Improve a 7B Model's Coding for an Enterprise

Design the post-training process that improves the coding capabilities of a 7-billion-parameter language model for one enterprise customer. Cover the full pipeline: data collection, supervised fine-tuning (SFT), reinforcement learning from human feedback (RLHF), evaluation, the inference engine that serves the tuned model, and the feedback loop that brings production usage back into training. Expect every stage to be probed in depth. The interviewer pays particular attention to how the datasets are built and how the reward model is chosen, and asks directly why an RLHF stage is needed at all. ### Constraints and Clarifications - Assume you start from a pretrained 7B decoder checkpoint that you are allowed to fine-tune, and that you own the training and serving stack. - Assume the customer can share its code and developer interactions under an agreement you must respect. What exactly may be used for training is a clarifying question, not a given. ### Clarifying Questions - Which coding tasks matter most to the customer: inline completion, chat-based code generation, code review, test writing, or multi-file changes? - Which languages, frameworks and internal libraries dominate the customer's codebase? - May the customer's proprietary code and usage logs be used for training, and must the data and the tuned weights stay inside the customer's environment? - Is the starting checkpoint a raw base model or one that is already instruction-tuned? - What latency and hardware budget does serving have, and how many developers will use it? - How will the customer judge success: suggestion acceptance, tasks completed, fewer defects, or something else? ### Part 1 — Data collection and dataset construction How do you collect and construct the training data for each later stage? ```hint Checkable versus judged In the coding domain some qualities of an answer can be checked by a machine and others need a person. Decide which data each kind of signal should produce. ``` #### What This Part Should Cover - Data sources, and the licensing, privacy and secret-handling constraints on each - Quality filtering, deduplication and separation from evaluation data - The format each stage needs: demonstrations, comparisons, or tasks with a checker ### Part 2 — Supervised fine-tuning, and why RLHF is still needed Describe the SFT stage. Then answer the interviewer's probe: "If SFT can also learn alignment when we add user preference data to it, why do we still need an RLHF step?" ```hint Imitation versus scoring Compare what a model learns from imitating a response someone chose with what it learns from being scored on responses it generated itself. ``` #### What This Part Should Cover - The SFT recipe: data mixture, full fine-tuning versus adapters, and protecting general capability - What the SFT objective can and cannot extract from preference data - On-policy versus off-policy learning, and when SFT alone is a reasonable choice ### Part 3 — RLHF and the choice of reward model Design the reinforcement-learning stage. How do you choose or build the reward model, and how do you know it is good enough to optimize against? ```hint A signal most domains lack Code offers a kind of reward that most other domains do not have. Think about where that signal is reliable and where it can be gamed. ``` #### What This Part Should Cover - Candidate reward sources and how they are combined - How a reward model is validated before and during optimization - The RL algorithm, regularization toward the SFT model, and reward hacking ### Part 4 — Evaluation, inference engine and feedback loop How do you evaluate the model before release, how do you serve it, and how do you close the loop from production usage back into training? ```hint Before and after launch Separate the offline gates you can run before shipping from the signals you can only get once real developers use the model. ``` #### What This Part Should Cover - Offline evaluation: customer-specific tasks, regression on general ability, and contamination control - Serving a 7B model within the latency budget, including memory and throughput - Which production signals become training data, and how often the model is retrained and released ### What a Strong Answer Covers - One coherent pipeline in which each stage's output feeds the next - Concrete choices with their trade-offs, not just stage names - Risks named and mitigated: forgetting, contamination, reward hacking, privacy - Metrics that tie the pipeline to the customer's definition of success ### Follow-up Questions - How would direct preference optimization (DPO) compare with an online RL method here, and what would you give up by choosing it? - The customer's codebase changes every day. How do you keep the model current without retraining from scratch each time? - How would you detect that the policy has learned to game a test-based reward? - After launch the customer asks for a smaller, faster model. How would you distill the 7B model, and what would you measure?