Design a Pipeline That Distills Reusable Agent Skills for Enterprise Customers

In an expertise round built around the team's own work on AI agents, you are asked: **design a pipeline that automatically extracts, or distills, agent skills for enterprise users.** For this question, assume a skill is a reusable, named package of instructions, optionally with scripts, tool-call recipes or reference files, that an agent loads when a task calls for it, so it does not have to rediscover the procedure every time. The pipeline takes material an enterprise customer already has and turns it into skills that the customer's agents can use. Walk through the design end to end: inputs, the stages that produce a skill, how a skill is validated before anyone relies on it, how it is delivered to agents, and how it is kept correct over time. ```hint Where the signal lives Ask which records show not only what an agent or a person did, but whether it worked. A procedure distilled from runs that failed teaches the wrong procedure. ``` ```hint A skill must earn its place Think about how you would show that an agent does better with a candidate skill than without it, before any customer relies on it. ``` ### Clarifying Questions - What raw material is available: agent execution traces and chat transcripts, written procedures and runbooks, code and queries, or a mix? - Who consumes the skills: only the customer's own agents? In what format must a skill be packaged? - Must skills stay private to each customer, or may a generic skill learned from one customer be offered to others? - Should new skills go live automatically, or does an administrator at the customer approve each one? - How is success measured: task success rate, fewer agent steps or tokens, or ratings from the people who use the agent? - Roughly how much raw material does a customer produce, and how quickly should a new recurring task turn into a skill? ### What a Strong Answer Covers - A precise definition of a skill and of the pipeline's inputs, consumers and success metric - The stages that turn raw material into a deployed skill, and what each stage outputs - How candidate skills are evaluated offline before release and monitored after release - Enterprise concerns: tenant isolation, sensitive data in the source material, the permissions a skill may exercise, administrator review and audit - Operating the pipeline over time: duplicate and conflicting skills, versioning, drift and retirement, and the cost of the model calls involved ### Follow-up Questions - Two customers' pipelines produce nearly identical skills. Would you share one skill between them, and how would you do it safely? - A skill distilled last quarter starts failing because the customer changed a table schema or a tool. How is that detected, and what happens next? - How do you stop a trace or document that contains credentials, personal data or injected instructions from ending up inside a skill? - A customer has hundreds of skills. How does the agent decide which ones to load for a task without filling its context window?