Embeddings

Embeddings represent words, images, users, and other entities as dense vectors. This lesson explains how to learn them with Word2vec, contextual language models, auto-encoders, supervised visual models, and networks trained on entity interactions.

What an embedding represents

An embedding encodes an entity, such as a word, document, image, person, or ad, as a vector in a lower-dimensional space. The learning task shapes which relationships that vector captures. Related entities can then be compared through their positions in the space.

Neural networks can learn these dense representations as part of a prediction task. The architectures below illustrate how embeddings support ML systems and how a learned representation can transfer to another task.

Transfer learning refers to transferring information from one ML task to another. Embeddings easily enable us to do that for common entities among different tasks. For example, Twitter can build an embedding for their users based on their organic feed interactions and then use the embeddings for ads serving. Organic interactions are generally much greater in volume compared to ads interactions. This allows Twitter to learn user interests by organic feed interaction, capture it as embedding, and use it to serve more relevant ads.

For another example, train Word2vec embeddings on Wiki text and use those representations as inputs to a spam-filtering model. The embedding supplies features; the downstream model still has its own prediction task.

In this lesson, we will go through some general ways of training neural networks to learn embeddings, using real-world example scenarios of their usage.

Text embeddings

We will go over two popular text term embeddings generation models and examples of their utilization in different ML systems.

Word2vec

Word2vec produces word embeddings by using shallow neural networks (having a single hidden layer) and self-supervised learning from a large corpus of text data. Word2vec is self-supervised as it trains a model by predicting words from other words that appear in the sentence(context). So, it can utilize tons of text data available in books, Wikipedia, blogs, etc. to learn term representation.

Representing words with a dense vector is critical for the majority of Natural language processing (NLP) tasks. Word2vec uses a simple but powerful idea to use neighboring words to predict the current word and in the process, generates word embeddings. Two networks to generate these embeddings are:

1. CBOW: Continuous bag of words (CBOW) tries to predict the current word from its surrounding words by optimizing for following loss function:

$Loss=$ $-log$ (p($w_t$|$w_{t-n}$,…,$w_{t-1}$, $w_{t+1}$, $w_{t+n}$)) where $n$ is the size of our window to look for the corresponding word. It uses the entire contextual information as one observation while training the network. Utilizing the overall context information to predict one term helps generate embeddings with the smaller training dataset. The architecture would look like the following:

2. Skipgram: In this architecture, we try to predict surrounding words from the current word. The loss function will now look like:

$Loss=$ $-log$ (p($w_{t-n}$,…,$w_{t-1}$, $w_{t+1}$, $w_{t+n}$|$w_t$)) where $n$ is now the size of surrounding words that we are trying to predict using the current word. In training this network, each context pair will result in a different training example that the model tries to predict. This architecture is most helpful when we have a large training set. The model architecture now looks like:

Example

Any machine learning task that wants to utilize text terms can benefit from this dense embedding vector, which captures word semantic meanings.

Suppose you want to predict a user’s interest in a document from their reading history. Average the Word2vec embeddings of the title terms from documents they engaged with to create a user representation. Represent the candidate document by the average of its own title-term embeddings. Their dot product can then serve as an input to the prediction model. Alternatively, pass both vectors to a neural network and let that model learn how to use the pair.

Context-based embeddings

Once trained, Word2vec embeddings have a fixed vector for every term. So, a Word2vec embedding doesn’t consider the context in which the word appears to generate its embedding. However, words in a different context can have very different meanings. For example, consider these two sentences:

1. I’d like to eat an apple.

2. Apple makes great products.

Word2vec will give us the same embedding for the term “apple” although it points to completely different objects in the above two sentences.

A contextual model receives neighboring terms when it generates the representation. That lets the representation change between the two meanings of “apple” above. A trained Word2vec model instead looks up the same stored vector for a term without receiving the current sentence.

Two popular architectures used to generate word context-based embedding are:

1. Embeddings from Language Models (ELMo)

2. Bidirectional Encoder Representations from Transformers (BERT)

The idea behind ELMO is to use the bi-directional LSTM model to capture the words that appear before and after the current word.

BERT uses an attention mechanism and is able to see all the words in the context, utilizing only the ones (i.e., pay more attention) which help with the prediction.

The entity-linking case study applies contextual representations to entity recognition and linking.

Visual embedding

The following two approaches learn image representations from different training tasks: reconstruction and supervised prediction.

Auto-encoders

Auto-encoders use neural networks consisting of both an encoder and a decoder. They first learn to compress the raw image pixel data to a small dimension via an encoder model and then try to de-compress it via a decoder to re-generate the same input image. The last layer of encoder determines the dimension of the embedding, which should be sufficiently large to capture enough information about the image so that the decoder can decode it.

Train the encoder and decoder together with backpropagation to reduce the difference between the original image and its reconstruction. After training, the encoder produces the image embedding; the decoder’s role was to provide the reconstruction task that trained it.

Once we have trained the model, we only use the encoder (first N network layers) to generate embeddings for images.

Auto-encoders are also an example of self-supervised learning, like Word2vec, as we can use an image data set without any label to train the model and generate image embeddings.

Visual supervised learning tasks

Visual supervised learning tasks such as image classification or object detection, are generally set up as convolution layers, pooling layers, and fully connected network layers, followed by final classification(softmax) layers. Let’s consider the example of the ImageNet VGG16 model that is shown in the figure below. The input passes through a set of convolution, pooling, and fully connected layers to the last softmax layer for the final classification task. The penultimate layer before softmax captures all image information in a vector such that it can be used to classify the image correctly. So, we can use the penultimate layer value of a pre-trained model as our image embedding.

An example of image embedding usage could be to find images similar to a given image.

Another example is an image search problem where we want to find the best images for given text terms, e.g. query “cat images”. In this case, image embedding along with query term embedding can help refine search relevance models.

Learning embeddings for a particular learning task

Most of our discussion so far has been about training a general entity embedding that can be used for any learning task. However, we can also embed an entity as part of our learning task. The advantage of this embedding is a specialized one for the given prediction task. One important assumption here is that we have enough training data to be able to learn such representation during model training. Another consideration is that training time for learning the embedding as part of the task will be much higher compared to utilizing a pre-trained embedding.

Let’s consider an example where we are trying to predict whether a user will watch a particular movie based on their historical interactions. Here, utilizing movies that the user has previously watched as well as their prior search terms can be very beneficial to the learning task. We can do this by embedding sparse vector of movies and terms in the network itself, as shown in the image below. The recommendation case study develops this use of embeddings.

Network/Relationship-based embedding

Most of the systems have multiple entities, and these entities interact with each other. For example, Pinterest has users that interact with Pins, YouTube has users that interact with videos, Twitter has users that interact with tweets, and Google search has both queries and users that interact with web results.

We can think of these interactions as relationships in a graph or resulting in interaction pairs. For the above example, these pairs would look like:

1. (User, Pin) for Pinterest

2. (User, Video) for YouTube

3. (User, Tweet) for Twitter

4. (Query, Webpage) for Search

5. (Searcher, Webpage) for Search

In all the above scenarios, the retrieval and ranking of results for a particular user (or query) are mostly about predicting how close they are. Therefore, having an embedding model that projects these documents in the same embedding space can vastly help in the retrieval and ranking tasks of recommendation, search, feed-based, and many other ML systems.

We can generate embeddings for both the above-discussed pairs of entities in the same space by creating a two-tower neural network model that tries to encode each item using their raw features. The model optimizes the inner product loss such that positive pairs from entity interactions have a higher score and random pairs have a lower score. Let’s say the selected pairs of entities (from a graph or based on interactions) belong to set A. We then select random pairs for negative examples. The expression below is an informal score-separation objective: it increases the positive-pair scores relative to the sampled negative-pair scores. Read the “max” notation as an objective to maximize, despite its “Loss” label; it does not specify a complete loss implementation to minimize.

$Loss=max({\sum }_{(u,v)\in A}dot(u,v)-{\sum }_{(u,v)\notin A}dot(u,v))$

---