Tokenization and Embeddings: How Generative AI Understands Language

When people first encounter Generative AI, it can feel almost magical. You type a sentence, and out comes a thoughtful response, a poem, or even working code. But under the hood, there’s no magic at all; just math, probability, and some very clever design. 

In a section of our Introduction to Generative AI Concepts course, we pull back the curtain and explore two foundational ideas that make large language models possible: tokenization and embeddings. 

Understanding these concepts is a turning point for many learners, because it’s where Generative AI stops feeling mysterious and starts making sense. 

From Human Language to Machine Readable Input 

Humans experience language as words and meaning. Machines don’t. Before a large language model (LLM) can do anything useful with text, that text must be transformed into a format a computer can work with. 

This transformation happens in two key steps:

1. Tokenization – breaking text into manageable pieces
2. Embeddings – converting those pieces into numbers that represent meaning

These steps are the entry point for everything an LLM does, from writing emails to summarizing documents.

What Is Tokenization?

At its simplest, tokenization is the process of breaking text into smaller units called tokens. A token might be:

• A whole word
• Part of a word (a subword, like run in running)
• A punctuation mark
• Or even a number 

For example, a sentence you see as a smooth line of text may be split into dozens of tokens behind the scenes. Some words become a single token, while others are broken into multiple pieces, especially longer or less common words. 

Why does this matter? 

Because models don’t “read” text. They process token IDs, or numbers that represent each token in the model’s vocabulary. The way text is tokenized directly affects:

• Cost (more tokens = more computation)
• Accuracy
• How well a model handles numbers, punctuation, and specialized language 

In other words, tokenization shapes how the model perceives your input before it ever generates a response.

Why Subwords Are a Smart Compromise 

You might wonder: why not just tokenize by characters or full words? 

• Characters are too small to capture meaning efficiently. 
• Full words change meaning (bad = good) and language evolves constantly - think of words like selfie, doomscrolling, or prompt engineering.

Subword tokenization strikes a balance. It allows models to understand unfamiliar or rare words, reuse meaningful word fragments, handle variations like run, running, and runner more effectively. This is one reason modern LLMs are so flexible with language they’ve never seen before.

Embeddings: Turning Meaning into Math

Once text is tokenized, the next step is embeddings. An embedding is a numerical representation of a token; essentially a list of numbers (called a vector) that captures aspects of that token’s meaning. These numbers allow the model to compare tokens mathematically and recognize relationships between them. 

This is where things get interesting. With embeddings, models can detect patterns such as:

• Similar words being “close” together
• Words with different meanings separating based on context
• Conceptual relationships like king – man + woman ≈ queen (If you remove the “male” aspect from the concept of king and add the “female” aspect, you end up very close to the concept of queen.)

Embeddings don’t store dictionary definitions. Instead, they encode meaning based on how words are used across massive amounts of training data. That’s why models can understand slang, ambiguity, and context in a surprisingly human like way.

Context Is Everything

Words often have multiple meanings. Think about:

• Bank (money vs. river)
• Bat (animal vs. sports equipment) 

Embeddings allow a model to infer the correct meaning based on surrounding tokens. The same word can “land” in a different region of embedding space depending on context. This ability is foundational to how LLMs generate relevant, coherent responses instead of random text.

Predicting the Next Token

Once text has been turned into embeddings, a large language model does one main thing: it predicts what should come next. Based on the text so far, the model considers several possible next words and estimates how likely each one is. It then chooses one and repeats this process, generating text one word (or token) at a time. 

The model can be more cautious or more creative in how it makes these choices. When it focuses on only the most likely options, the output tends to be more predictable and factual. When it allows a wider range of possibilities, the responses can become more creative and expressive, though sometimes less consistent. This is why the same prompt can produce very different‑sounding results depending on how the model is configured. 

From Theory to Practice: A Hands-On Lab

Concepts like tokenization and embeddings really click when learners can see them in action and when they have an expert to guide them along the way. This is just one of the many real-world, hands-on labs in our curriculum.

In this lab in our Introduction to Generative AI Concepts course, learners actively:

• Experiment with live tokenizers to see how text is split into tokens
• Compare how different inputs (numbers, punctuation, code) are tokenized
• Visualize word embeddings and explore semantic similarity
• Observe how meaning clusters form in embedding space
• Work with visual embeddings, including image-based examples 

Throughout all hands-on labs, an experienced instructor is present to demonstrate concepts, answer questions, and help learners connect what they’re doing to how they work in the real world.

Why This Matters

Understanding tokenization and embeddings changes how you:

• Write prompts
• Evaluate AI output
• Estimate cost and performance
• Troubleshoot unexpected behavior
• Design AI powered applications 

Instead of treating models like black boxes, you start working with their strengths and around their limitations.

Our training philosophy is grounded in learning by doing and directly connecting concepts to real on‑the‑job work. When foundational Generative AI concepts remain abstract, learners often struggle to truly understand how large language models operate. Without seeing how Generative AI works in practice (or having the opportunity to get into the weeds and ask questions in the moment) learners may understand what the concepts are, but not how to apply them to the way work actually gets done.

Any course in our AI & Agentic AI catalog can be customized for your team and delivered live online or at your site; contact us to get started.