Handling High-Cardinality Categorical Features and Cold-Start Items in Embedding Models

You work on a model that scores items for users, such as a recommendation or feed-ranking model. Its inputs include categorical features and learned embedding features for items. In a machine learning engineer technical screen, the interviewer asked two follow-up questions about these features: how to handle categorical features that have a very large number of dimensions, and how embedding features should handle cold-start items, that is, items with little or no interaction history. ### Constraints and Clarifications - The model, its training objective and the data volumes are not specified. Assume a model that scores (user, item) pairs and is trained on logged interactions such as clicks, and state any other assumptions you make. - Answer for a neural model with embedding tables, and say where your answer changes for a linear or tree-based model. ### Clarifying Questions - Does "many dimensions" mean that one categorical feature has a very large number of distinct values, or that there are many categorical features? - Roughly how many distinct values do the largest features have, and how quickly do new values appear? - Is there a memory or latency budget for serving the embedding tables? - How soon after a new item is created must the model score it sensibly? ### Part 1 — Categorical features with many distinct values Some categorical features, such as IDs, have a very large number of distinct values, and new values keep appearing. How would you represent these features in the model? Compare the options and say when you would choose each. ```hint Count the parameters Work out how the input width and the parameter count grow with the number of distinct values, and how much training signal a value seen only a handful of times can provide. ``` ```hint The value you have never seen Decide what your representation does with a value that first appears after training, and make sure training and serving agree on it. ``` #### What This Part Should Cover - The cost of one-hot encoding at this scale, and alternatives that bound the size and memory - Treatment of rare values and values unseen in training - Trade-offs between the options: collisions, leakage, memory and the model type - How the embedding size is chosen and how large tables are served ### Part 2 — Embedding features and cold-start items Item embeddings are learned from interactions. How do you give a new item, which has few or no interactions, a useful representation so the model can still score it well? ```hint What a new item already has A new item has no interaction history, but it is not a blank. List what is known about it at the moment it is created. ``` ```hint Train for what you will serve Ask whether the model ever sees, during training, the situation a cold item creates at serving time. ``` #### What This Part Should Cover - Backing off from the learned ID embedding to information available when the item is created - Training so the model behaves well when the ID embedding is missing or uninformative - Collecting interactions quickly and moving an item from cold to warm - Measuring cold-start quality separately from overall quality ### What a Strong Answer Covers - Choices tied to the model type and to the memory and latency budget - Concrete numbers for table sizes or bucket counts, not only names of techniques - One consistent policy for unseen values from training to serving - Awareness of training-serving skew and of feedback loops that keep new items from ever being shown - Evaluation sliced by value frequency and by item age ### Follow-up Questions - How would you pick the embedding dimension for each categorical feature? - New values of a feature appear every minute. How do you keep their embeddings useful without retraining the whole model? - How would you show, offline and online, that your cold-start approach helps new items without hurting everyone else's experience? - What changes if the cold-start entity is a new user rather than a new item?