RAG is not a product: it's a layer of the solution
In the last two years, RAG has become one of the most commonly used terms in AI commercial proposals in Brazil. Along with it came confusion that is costing companies dearly, as they contracted systems that work in demos but fail in production.
I'll be direct: most of the RAG implementations I've seen in the market do not work well. Not due to a lack of technology, but due to a lack of understanding of what RAG really is and what it is not.
What RAG really is
RAG stands for Retrieval-Augmented Generation. In practical terms: before generating a response, the system retrieves relevant information from an external database and injects that information into the context of the model.
The problem that RAG solves is real and important. LLMs are trained with data up to a certain date and do not have access to your company's specific knowledge, documents, or processes. If you want the model to respond based on internal and updated information, you need some mechanism to bring that knowledge into the context of each call.
RAG is one of those mechanisms. It is the most common and the most studied. It works well when implemented correctly within a well-thought-out architecture.
Implemented haphazardly, which is how most are being implemented, it delivers exactly the result the market is seeing: impressive demo, disappointing production.
Why "we implemented RAG" does not mean the problem is solved
RAG is not an API call. It is a pipeline with at least six critical stages, each with implementation decisions that directly affect the quality of the final result. Making a mistake at any stage compromises the entire system.
Document preparation. How do you split the documents into chunks? Chunk size affects retrieval accuracy and processing cost. Chunks that are too small lose context. Chunks that are too large retrieve irrelevant information along with the relevant. Documents with complex formatting, tables, or cross-references require specific chunking strategies that most implementations ignore.
Embedding generation. Which embedding model do you use? Embeddings from different models capture semantics in different ways. An embedding model predominantly trained in English will have degraded performance in Portuguese without adjustment. This is a decision that affects everything that comes after and rarely receives the attention it deserves.
Indexing strategy. Purely semantic search fails when the query uses terminology different from the document, even if the meaning is identical. Hybrid search, combining vector with lexical, improves recall in many cases but adds complexity in configuration and tuning.
Retrieval. How many chunks do you retrieve per query? Too few and you lose relevant context. Too many and you pollute the context with information that confuses the model and increases cost per call. The right number depends on the use case and needs to be calibrated with real usage data.
Reranking. The chunks retrieved by vector similarity are not necessarily the most relevant for the specific query. Rerankers apply a second layer of evaluation to order the results before injecting them into the context. This is a step that most implementations I have seen skip and that often makes a measurable difference in the quality of the response.
Generation. Only here does the model receive the context and generate the response. If any previous step was poorly implemented, the generation will be based on poor context. The model will do the best it can with what it receives, but it will not compensate for poor quality retrieval. No model does that.
Implementing RAG without attention to each of these steps is building on a weak foundation. It works in the demo because the demo uses clean documents, predictable queries, and the happy path. It shows up in production when real documents, varied queries, and users who do not know exactly what they are looking for arrive.
When RAG is the right choice and when it is not
RAG effectively solves a specific problem: giving the model access to external and updated knowledge dynamically, without retraining the model.
It works well when: the knowledge base changes frequently and needs to be available to the model in real-time; when the knowledge is too vast to fit into the context of a single call; when you need clear traceability (to know where the information that supported each response came from).
It is not the right choice when: the knowledge the model needs is stable and small enough to fit into a well-structured context; when the additional latency of the retrieval step is unacceptable for the use case; when the database is not organized and clean enough to produce reliable retrievals.
This last case is more common than it seems. I worked with a company that wanted to implement RAG over years of emails and internal documents without any organization. The problem was not implementing the retrieval pipeline. It was that the base was so inconsistent that anything you retrieved would be noise as much as signal. RAG over bad data produces bad responses with a lot of apparent confidence, which is worse than admitting you don’t know.
RAG within a real architecture
In well-architected applications, RAG is a component within the context preparation layer. It coexists with other sources of context: data from authenticated users, session state, results from calls to external systems, structured business rules.
Orchestration decides, for each query, which combination of contexts is relevant. Sometimes RAG is the main context. Sometimes it is secondary. Sometimes it is not needed for that specific query.
Treating RAG as the architecture instead of as a component of the architecture leads to systems optimized for one type of query that fail when the use case evolves.
What happens when you sell RAG as a product differentiator
When a company positions "we have RAG" as a differentiator, it is describing the implementation of a technical component as if it were a business capability. The user does not buy RAG. They buy accurate responses, quick access to information, more efficient processes, better-informed decisions.
RAG can help deliver this. What determines whether it will or will not is the quality of the entire pipeline, from document preparation to output validation, not the presence of the technique.
The real differentiator lies in: how well the system understands user queries in their specific domain; how clean and organized the knowledge base that feeds the system is; how accurate the retrieval is for the vocabulary and typical intentions of your use case; and how well calibrated the generation is for the level of detail and format that the user needs.
All of this is product engineering work. RAG is one of the ingredients, not the recipe.
RAG is a powerful technique. When well implemented, within a well-thought-out architecture, with quality data and a calibrated pipeline, it significantly enhances the ability of AI systems to work with specific domain knowledge.
But the technique does not replace engineering. And in today's market, where RAG has become a buzzword in commercial proposals without most users being able to explain the six stages of the pipeline, confusing the two is the fastest way to a system that impresses in presentation and disappoints in real use.
