Word2Vec Training, Homonymy in Static Embeddings, and the Idea Behind GloVe

Answer four questions on word embeddings: how Word2Vec works and is trained, whether a word gets the same Word2Vec vector in two different sentences, how the homonymy problem of static embeddings could be solved before attention and BERT, and what GloVe is. ### Clarifying Questions - Should the answer cover both Word2Vec variants, skip-gram and CBOW? - In the "castle" sentences, is the word meant as a homonym, a lock in the first sentence and a building in the second? - For the homonymy question, is success judged by the word representations themselves, or by accuracy on a downstream task? ### Part 1 — How Word2Vec works How does Word2Vec work, how is it trained, what is fed to it as input, and what is its architecture? ```hint Training pairs from raw text Think about which prediction task turns unlabeled text into training examples, and which weights you keep once training is over. ``` #### What This Part Should Cover - The two variants, skip-gram and CBOW, and how training examples are built from raw text - The architecture: its layers and weight matrices, and which of them become the word vectors - The training objective, and how its cost is kept manageable for a large vocabulary ### Part 2 — The same word in two sentences In Word2Vec, will the word "castle" have the same vector in the sentences "I opened the castle" and "I went into the castle"? The first sentence reads naturally only if the word means a lock, so treat it as one written word with two senses: something you open and a building you enter. ```hint Where the vector comes from Ask at what point, if any, the surrounding sentence enters the computation when you look up a trained word vector. ``` #### What This Part Should Cover - Whether the vectors differ, and why - What a single vector for a two-sense word ends up representing - What this means for downstream models ### Part 3 — Homonymy before attention and BERT How could the homonymy problem of static embeddings be addressed before attention-based contextual models and BERT? ```hint Where to intervene For one written word to get different vectors in different sentences, some step of the pipeline has to change. Ask which step that could be: the vocabulary, the tokens the model trains on, or the lookup at use time. ``` #### What This Part Should Cover - Several distinct approaches, each explained through the step of the pipeline it changes - How each approach chooses the right representation for a given occurrence at use time - The limits and costs of each approach ### Part 4 — GloVe What is GloVe, and what is its essence? ```hint One window at a time Ask what information Word2Vec absorbs only implicitly, one window at a time, and whether a model could be fitted to that information directly. ``` #### What This Part Should Cover - What GloVe is fitted to, and its objective and weighting function - The intuition that motivates the objective, and why it yields linear relations such as analogies - How GloVe compares with Word2Vec ### What a Strong Answer Covers - The skip-gram and CBOW objectives, and how their training cost is kept manageable, stated correctly - The answer to the castle pair, tied to how a trained static embedding is used at lookup time - Several pre-transformer remedies for homonymy, with their limits - GloVe's objective and its relationship to Word2Vec - Practical details: window size, handling of very frequent words, vector dimension, which weights are kept ### Follow-up Questions - Why is negative sampling needed, and how are the negative words chosen? - Why does vector arithmetic such as king minus man plus woman land near queen? - How would you represent words never seen during training? - Word2Vec with negative sampling has been shown to implicitly factorize a matrix. Which one?