Why Iterative Thinking is Crucial for Question Answering

The simple argument for Agentic RAG

Let’s say you want your RAG system to answer: “What is the size of the town or village where Joseph Conrad was born”. The natural way to answer that question is a two step process:

1. Find out where Conrad was born

2. Find out how many people live there

A traditional RAG system is a one step process and it cannot do that. Theoretically it could retrieve information about all towns in Europe together with the biography of Conrad and then combine these texts. But it would be a very heavy process even in that simple case, and if we add steps to this procedure it would mean exponential explosion of the required background texts.

That is why traditional one step RAG cannot be the base for any universal solution in Question Answering. For that we need iterative processes. The idea that we can gather all needed background info in one step and then do all the thinking in a seconds step is pretty silly on its face - but somehow it got so popular in LLM applications. It does simplify the code and we needed this simplification at the beginning. Now the RAG setups proposed in blogs and conferences are not simple anymore without being complete. We need to step over to the next phase and do something more complete - an iterative approach such as Agentic RAG.