Two very different kinds of AI model sit inside most "chat with your documents" systems, and confusing them leads to budgets and architectures that do not hold up.

When a vendor says their product "uses AI to search your company documents," there are usually two separate models involved, doing two unrelated jobs. One writes the answer. The other decides which of your documents the answer should be based on. Both are commonly called "LLMs," which is where the confusion starts — and where a lot of budget gets misallocated. This article explains what an embedding model actually does, how it differs from a general-purpose model like the one behind ChatGPT, and how to decide which one your project needs.

1. What a general-purpose LLM does

A general-purpose LLM is a model that takes text as input and writes new text as output. This is the category most people have used directly: ChatGPT, Claude, Gemini, and the open models teams run on their own servers.

Typical business uses include:

• Drafting a customer email from a few bullet points
• Summarizing a 40-page contract into one page
• Answering a question in conversational language
• Extracting structured fields from a messy invoice
• Writing and reviewing code

A simple way to think about a general-purpose LLM: it is the part of the system that talks.

What it does not do on its own is know anything about your business. Its knowledge comes from public data it was trained on, frozen at a point in time. It has never seen your pricing sheet, your support tickets, or last quarter's board deck. If you ask about them without supplying them, the model will produce a fluent answer built on guesses — the failure mode usually called hallucination.

2. What an embedding model does

An embedding model is a model that takes text as input and outputs a list of numbers that represents the meaning of that text. It does not write, summarize, or answer. It converts.

That list of numbers is called a vector or an embedding. For a common model like OpenAI's text-embedding-3-small, each piece of text becomes a list of 1,536 numbers; the larger text-embedding-3-large produces 3,072. The specific numbers are not meaningful to a human. What matters is the relationship between them: two passages about late delivery penalties will produce vectors that sit close together, even if one says "liquidated damages for delayed shipment" and the other says "we charge a fee if the goods arrive late."

This is the property that makes embeddings useful. Traditional keyword search matches words. Embedding-based search matches meaning, so a customer asking "why was I charged twice" can be matched to a policy document that never uses the word "charged."

A simple way to think about an embedding model: it is the part of the system that files things by meaning, so the right document can be found later.

What an embedding model does not do is generate language. Give it a question and it will not answer — it returns numbers. It also cannot reason, follow instructions, or evaluate whether a retrieved document is actually relevant. It measures similarity, and similarity is not the same as correctness.

3. What RAG is, and where embeddings fit

RAG stands for retrieval-augmented generation. It is the standard pattern for making a general-purpose LLM answer questions about content it was never trained on — your content.

The idea is straightforward: before asking the model to answer, find the handful of internal documents most likely to contain the answer, paste them into the prompt, and instruct the model to answer using only that material. "Retrieval" is the finding step. "Generation" is the answering step.

Embedding models power the retrieval step. In a typical setup:

• Your documents are split into passages of a few hundred words each, a step called chunking.
• Each chunk is run through the embedding model once and stored as a vector in a vector database — a database built to find the nearest vectors to a given vector quickly.
• When a user asks a question, the question is run through the same embedding model, and the database returns the chunks whose vectors sit closest to it.

The general-purpose LLM never touches this process. It only sees the final prompt: the user's question plus the retrieved passages.

One limitation worth naming early: RAG does not make a model factually reliable by itself. If retrieval returns the wrong passages, the model will answer confidently from the wrong passages. In most production systems, retrieval quality — not model quality — is what determines whether users trust the output.

4. How the two models work together

A realistic scenario: a logistics company wants its support team to stop digging through a 900-page carrier agreement archive.

1. 1.

   Preparation, done once. The 900 pages are split into roughly 3,000 chunks. Each chunk is embedded and stored in a vector database, along with metadata such as carrier name and contract date.
2. 2.

   A question arrives. An agent types, "Do we owe anything if the driver waits more than two hours at pickup?"
3. 3.

   Retrieval. The question is embedded and compared against the 3,000 stored vectors. The system returns the ten closest chunks — likely including detention and demurrage clauses that never use the word "wait."
4. 4.

   Filtering. A reranker or metadata filter narrows those ten to the four most relevant for the carrier in question.
5. 5.

   Generation. The general-purpose LLM receives the question plus those four passages, and writes a plain-language answer with references back to the source clauses.
6. 6.

   Verification. The agent sees the cited clauses alongside the answer and can confirm it in seconds rather than searching from scratch.

Steps 1 and 3 use the embedding model. Step 5 uses the general-purpose LLM. Neither can do the other's job.

5. When to use which

Use an embedding model when the task is finding, grouping, or comparing. Concretely:

• Searching across internal documents, tickets, or a knowledge base
• Detecting duplicate or near-duplicate records
• Routing incoming requests to the right team or category
• Clustering customer feedback into themes
• Recommending similar products or articles

Use a general-purpose LLM when the task is writing, reasoning, or transforming. Concretely:

• Answering a question in natural language
• Summarizing, translating, or rewriting text
• Extracting structured data from unstructured documents
• Multi-step reasoning or tool use

Use both when the task is answering questions about your own content. Any RAG system, internal assistant, or document-grounded chatbot needs both, and the two choices are largely independent — you can change the generation model without re-processing your documents.

Two cost points are worth having in mind, because they surprise people. First, embedding models are far cheaper per unit of text than generation models: OpenAI currently lists text-embedding-3-small at $0.02 per million tokens and text-embedding-3-large at $0.13, while its general-purpose models are priced in dollars per million input tokens and higher still on output. Second, embedding is mostly a one-time cost per document, whereas generation is a recurring cost per question. In a mature RAG system, the retrieval side is rarely the expensive part.

The trade-off runs the other way on switching cost. Changing the generation model is usually a configuration change. Changing the embedding model means re-embedding every document in the corpus, because vectors from two different models are not comparable. That is a real migration, and it is worth choosing deliberately rather than defaulting.

6. Why this matters for businesses

The practical gap this creates is that most organisations evaluate AI vendors on the wrong half of the system. Demos showcase the writing — fluent answers, a polished chat interface — while the part that determines whether answers are correct sits in the retrieval layer nobody asked about.

What to look for when assessing a proposal or an internal build:

• Ask how retrieval is evaluated, not just how the model performs. A credible team can tell you what share of questions return the correct source passage in the top few results, measured on a real set of questions from your business.
• Ask what happens when nothing relevant is found. A well-built system says it does not know. A weak one lets the model improvise.
• Ask whether answers cite their sources. Traceability back to a document and section is what makes the output auditable, and in regulated contexts it is usually non-negotiable.
• Ask where the embeddings are stored and computed. Embedding your entire document archive means sending it through a model. If that model is a third-party API, this becomes a data-residency question — and there are capable open embedding models that run inside your own infrastructure when the data cannot leave it.
• Ask about re-indexing. Documents change. If there is no plan for keeping the vector database current, the assistant will confidently quote last year's policy.

The failure pattern we see most often is not a bad model choice. It is a team that spent its effort on the generation layer, used default settings for chunking and retrieval, and ended up with a system that sounds capable and is wrong often enough that people quietly stop using it. Retrieval quality is where the engineering effort pays off.

7. Quick reference

• General-purpose LLM — writes text; the part of the system that talks.
• Embedding model — converts text into numbers representing meaning; the part that files and finds.
• Vector (or embedding) — the list of numbers representing one piece of text.
• Chunking — splitting documents into passages small enough to embed and retrieve usefully.
• Vector database — storage built to find the closest vectors to a query fast.
• RAG — retrieving your own documents and passing them to an LLM so it answers from them rather than from memory.
• Reranker — an optional second pass that re-scores retrieved passages for relevance before generation.

Conclusion

Embedding models and general-purpose LLMs are complements, not competitors: one makes your content findable, the other makes it readable. Teams that understand the split ask better questions of their vendors, budget more accurately, and build systems that stay trustworthy as the document set grows.

At ITGRATE, we build these systems for European businesses from Munich and Ho Chi Minh City — as an MVP when you need a working internal assistant in weeks rather than quarters, or as a dedicated product team embedded alongside your own engineers for the longer build. Either way, we treat retrieval as the part that earns the trust.

A system that writes well but retrieves poorly is a system that is confidently wrong at scale. Getting the retrieval layer right is what separates a demo from something your team actually uses.