Word Embeddings: From One-Hot Encoding to Word2Vec

Overview

Computers work with numbers, not words directly. To process language, we must represent text numerically. A word embedding maps a word to a vector—a sequence of numbers that can capture useful relationships between words.

Before exploring learned embeddings, let's examine simpler representations: integer encoding, TF-IDF, one-hot vectors, and bag-of-words. Each is useful, but each has limitations that motivate dense, learned representations.

1. Single-Number Representation

Suppose our vocabulary is a set $V$ containing $n$ words:

We can sort the vocabulary (for example, alphabetically) and assign each word a unique integer from $1$ to $n$.

This gives each word an identifier, but the numerical distance between identifiers does not express similarity. Word 10 is not necessarily more closely related to word 11 than to word 500.

2. TF-IDF Vectorization

TF-IDF (term frequency–inverse document frequency) is a statistical technique that measures how informative a term is in a document relative to a collection of documents (a corpus).

Let $D = \{d_1, d_2, \ldots, d_k\}$ be a corpus of $k$ documents, and let $t$ be a term.

Term Frequency (TF)

Term frequency measures how often $t$ appears in document $d$. A simple definition is:

Other implementations normalize the count by document length.

Inverse Document Frequency (IDF)

Document frequency $\operatorname{DF}(t,D)$ counts the documents containing $t$. A common IDF definition is:

For example, suppose the corpus contains 10,000 documents and the term appears in 100 of them:

Very common words, such as the, tend to have lower IDF scores because they appear in many documents.

Combining TF and IDF

TF-IDF is valuable for representing documents and highlighting informative terms. However, it does not directly learn semantic relationships between words.

3. One-Hot Vector Representation

In one-hot encoding, every word is represented by a vector of length $|V|$. Exactly one position is $1$; all other positions are $0$.

Suppose the vocabulary is:

Then:

The vector's position identifies the word, but the representation contains no information about what the word means.

4. Bag of Words (BoW)

A bag-of-words representation combines word vectors to describe an entire document, ignoring the order in which words appear.

If a document $d$ contains $T$ tokens $w_1,\ldots,w_T$, its count-based bag-of-words vector is:

A weighted variation is:

The weights might reflect term frequencies or another measure of importance.

Consider these two sentences:

• Cat eats fish.
• Fish eats cat.

They contain the same words, so a count-based bag-of-words representation is identical, even though their meanings differ.

Limitations of One-Hot and Bag-of-Words Representations

1. High dimensionality: The vector length grows with the vocabulary.
2. Sparsity: Most vector entries are zero. For $m$ documents, a vocabulary-by-document representation has shape $|V|\times m$ and is typically sparse.
3. Lost word order: The two example sentences have the same bag-of-words vector.
4. Lost phrase meaning: The phrase New York has a meaning not captured simply by treating New and York as independent words.
5. No inherent similarity: Distinct one-hot word vectors are orthogonal.

For two different words such as apple and orange:

Both are fruits, but their one-hot vectors cannot express that similarity.

Can N-Grams Help?

An n-gram groups $n$ consecutive words. For cat eats fish:

• Bigrams: cat eats, eats fish
• Trigram: cat eats fish

N-grams preserve some local word-order information. However, they substantially expand the feature vocabulary and do not inherently learn that related words have related meanings.

5. Why Dense Word Embeddings?

Imagine representing words as points in a relatively low-dimensional, continuous vector space. Related words could occupy nearby positions, allowing us to measure similarity and sometimes discover relationships through vector arithmetic.

For intuition, imagine these hand-designed features:

| Feature | King | Queen | Man | Woman | | --- | ---: | ---: | ---: | ---: | | Power | 0.90 | 0.80 | 0.50 | 0.40 | | Wealth | 0.89 | 0.85 | 0.45 | 0.38 | | Technology | 0 | 0 | 0 | 0 | | Gender (illustrative axis) | -1 | 1 | -1 | 1 |

Important: These values are illustrative, not real learned embeddings. A model learns numerical dimensions from data; those dimensions are not necessarily individually interpretable as named attributes.

Some learned embedding spaces exhibit approximate analogies such as:

or:

These relationships are useful intuition, but analogies are not guaranteed for every embedding model or word pair.

6. Word2Vec

Word2Vec is a family of neural approaches for learning word embeddings from large text corpora. It was introduced by Tomas Mikolov and collaborators at Google in 2013.

The underlying intuition is often summarized by linguist J. R. Firth's observation:

You shall know a word by the company it keeps.

Words that appear in similar contexts often have related meanings or functions. Here, related does not necessarily mean synonymous.

Consider the sentence:

The quick brown fox jumped over the lazy dog.

With fox as the center word:

| Window size | Context words | | --- | --- | | 0 | None | | 1 | brown, jumped | | 2 | quick, brown, jumped, over |

Word2Vec commonly uses two training architectures:

• Skip-gram: Predict surrounding context words from a center word.
• Continuous Bag of Words (CBOW): Predict the center word from surrounding context words.

The prediction task is a training objective: our primary goal is to learn useful embedding vectors, not to build a next-word prediction application.

7. Skip-Gram Model

Skip-gram predicts context words given a center word $w_t$.

For a window size $k=2$, the context positions are:

From the example sentence with center word fox, we obtain training pairs:

| Input (center) | Target (context) | | --- | --- | | fox | quick | | fox | brown | | fox | jumped | | fox | over |

Skip-Gram Architecture

A center word is encoded as a one-hot vector and mapped to a learned embedding vector $\mathbf{v}_{w_t}\in\mathbb{R}^d$. A second set of learned output vectors $\mathbf{v}'_w$ is used to score candidate context words.

The full-softmax probability of context word $w_i$ is:

The numerator scores the candidate context word. The denominator normalizes across every word in the vocabulary.

Skip-Gram Training Objective

For a sequence $w_1,w_2,\ldots,w_T$ and context radius $k$, maximize the likelihood of observed context words:

Terms whose positions fall outside the sequence are omitted.

Equivalently, minimize the average negative log-likelihood:

Substituting the softmax expression:

Note: This is a cross-entropy / negative log-likelihood objective for the observed context targets. Although the targets are generated automatically from text, the model is trained using labeled center-context pairs.

Why Is Full Softmax Expensive?

The denominator requires scoring every vocabulary word for each training example. With a large vocabulary, calculating the probabilities and their gradients becomes computationally expensive.

This motivates more efficient training techniques, including negative sampling.

8. Negative Sampling

Instead of computing a probability distribution over the entire vocabulary, negative sampling trains the model to distinguish observed center-context pairs from randomly sampled non-context pairs.

For center word fox:

• Positive pair: (fox, brown) with label $1$.
• Negative pair: (fox, telephone) with label $0$, assuming the latter was sampled as a negative example.

A positive pair is scored with the sigmoid function:

For a negative pair:

With one positive context word $w_o$ and $K$ negative samples $w_{n_1},\ldots,w_{n_K}$, maximize:

Or minimize the negative of this expression.

Negative samples are drawn from a chosen sampling distribution; they need not be uniformly random. Training only scores the positive example and a small number of negatives, rather than the entire vocabulary.

Key takeaway: Negative sampling makes Skip-gram training much more efficient for large vocabularies. It optimizes a binary classification objective rather than the full-softmax probability directly.

9. Continuous Bag of Words (CBOW)

CBOW reverses Skip-gram: it uses context words to predict the center word.

For a context radius of two:

For the sentence:

The quick brown fox jumped over the lazy dog.

The input context is quick, brown, jumped, and over; the target is fox.

CBOW Architecture

1. Look up the learned input embedding for each context word.
2. Average (or sum) the context embeddings to form a hidden representation.
3. Use an output weight matrix and softmax to predict the center word.

If the context contains $n$ words, their embeddings are $\mathbf{v}_{w_1},\ldots,\mathbf{v}_{w_n}$, and the hidden vector is:

The vocabulary probability distribution is:

The training target is the actual center word $w_t$. The model minimizes a prediction loss, such as cross-entropy, and learns embeddings through backpropagation.

Because context vectors are averaged or summed, the order of context words is not preserved in the basic CBOW architecture.

10. Skip-Gram vs. CBOW

| Aspect | Skip-Gram | CBOW | | --- | --- | --- | | Input | Center word | Surrounding context words | | Prediction target | Context words | Center word | | Training examples | One center-context pair at a time | Context group to center word | | Context order | Not modeled explicitly | Not modeled explicitly | | Learned representation | Word embeddings | Word embeddings |

Both approaches learn embeddings by solving context-based prediction tasks.

Conclusion

Word embeddings provide a more expressive way to represent language than arbitrary integer IDs or sparse one-hot vectors.

We started with TF-IDF and bag-of-words, which are useful for document-level features but do not inherently model word meaning. We then introduced dense word embeddings, where related words can acquire related numerical representations.

Finally, we explored Word2Vec, including Skip-gram, CBOW, and negative sampling. These methods learn embeddings from the patterns of words appearing together in large text collections.

The central idea is simple: the contexts in which words appear provide valuable clues about their meanings and relationships.