Agentic RAG: Let the Agent Search

Byadmin

application we build is a RAG app.

The recipe is simple: chunk, embed, retrieve, then answer.

It looks clean on paper. But once you use it on real cases, things get messy very quickly: similarity search finds similar wordings but not necessarily useful chunks, the right evidence never shows up in the retrieved context as it ranks too low, or important context may be split across chunk boundaries.

With insufficient context, the LLM has little room to recover.

So, how about we make retrieval iterative? What if the model can search, read, decide whether it has enough evidence, and search again when needed? Probably we don’t even need the vector embeddings in the first place.

That’s the premise of agentic RAG.

In this post, we’ll build a mini agentic RAG workflow with the OpenAI Agents SDK. We’ll examine how the agent iteratively searches, reads, and grounds its answer.

At the end, we’ll take a step back and briefly discuss the considerations for building a practical agentic RAG solution.

1. Case Study: Answering a Policy Question with Agentic RAG

For our case study, we’ll build a policy RAG agent over a company policy document collection.

1.1 Curating The Document Collection

Here, I created six synthetic company policy docs. They are all markdown files. Each one has a title, an effective date, a short summary, and the policy text.

To be realistic, those docs cover 6 common company policy areas:

  1. approval_matrix.md, containing approval levels for common business travel decisions, effective on July 1, 2025.
  2. conference_guidelines.md, containing rules for attending external events, effective on May 15, 2025.
  3. faq.md, containing informal answers to common travel questions, effective on September 1, 2025.
  4. policy_updates_2026.md, containing updates to lodging, conference travel, and approval timing for 2026, effective on January 1, 2026.
  5. remote_work_policy.md, containing rules for remote work, effective on February 1, 2026.
  6. travel_policy.md, containing standard travel booking rules for flights, lodging, meals, and transportation, effective on March 1, 2025.

We made it intentional that the answer to a policy question may not live in one document. This allows us to see the desired agentic behavior.

You can find the full synthetic documents and the agentic RAG implementation notebook here.

1.2 Defining The Agent

Next, we configure the agent. For that, we use the OpenAI Agents SDK.

At a high level, the agent is just this:

# pip install openai-agents
from agents import Agent

agent = Agent(
    name="Policy research assistant",
    instructions=INSTRUCTIONS,
    model="gpt-5.4",
    tools=[list_docs, search_docs, read_doc],
)

Two parts we need to go through: the agent instruction, and the tools it has access to.

First, the instruction. This is where we define the desired search behavior:

# Note: This instruction is iterated with AI
INSTRUCTIONS = """
[Role]
You are a careful internal policy research assistant.

[Research behavior]
Answer employee policy questions using the document tools.
Find enough relevant evidence to support the answer.
Keep conclusions grounded in the policy documents.

[Expected output]
Give a direct answer first.
Then briefly explain the evidence.
Cite the document filenames used for each important claim.
""".strip()

For this case study, we require that the agent can only touch the docs via three pre-defined tools:

The first one is a tool that gives the agent a quick overview of what documents exist:

@function_tool
def list_docs() -> list[dict]:
    """List available policy documents without returning their body text."""
    return [
        {
            "doc_name": doc["doc_name"],
            "title": doc["title"],
            "effective": doc["effective"],
            "summary": doc["summary"],
        }
        for doc in docs.values()
    ]

The second tool is a keyword-search tool. We keep it simple here: each document is split into paragraph chunks, and each query is matched against those chunks by token overlap:

@function_tool
def search_docs(query: str) -> list[dict]:
    """Search policy documents and return the top three short snippets."""
    query_tokens = tokenize(query)
    scored = []

    for chunk in chunks:
        score = len(query_tokens & chunk["tokens"])
        if score:
            scored.append((score, chunk))

    scored.sort(key=lambda item: item[0], reverse=True)

    results = []
    for score, chunk in scored[:3]:
        snippet = chunk["text"].replace("\n", " ")
        if len(snippet) > 420:
            snippet = snippet[:417].rstrip() + "..."
        results.append({
            "doc_name": chunk["doc_name"],
            "title": chunk["title"],
            "section": chunk["section"],
            "snippet": snippet,
            "score": round(score, 2),
        })

    return results

The last tool is what allows the agent to open one document by filename:

@function_tool
def read_doc(doc_name: str) -> str:
    """Read one policy document by filename."""
    if doc_name not in docs:
        valid = ", ".join(sorted(docs))
        return f"Unknown document: {doc_name}. Valid documents: {valid}"

    return docs[doc_name]["text"]

That’s the full RAG agent.

1.3 Running One Policy Question

Now we test the agent with one concrete question:

“I am attending a conference in Berlin. The conference organizer lists an official hotel, but the nightly rate is above the normal hotel cap. Can I book that hotel, and what approval do I need before booking?“

We run the agent with:

from agents import Runner

result = await Runner.run(agent, PROMPT, max_turns=12)

The agent produced the right answer: yes, the employee can book the official conference hotel if there is a practical business reason. It got that information from conference_guidelines.md.

For the approval part, the agent first identified that approval is needed as the hotel is above the normal cap. Then it gave the corresponding approval conditions. The agent used travel_policy.md, approval_matrix.md, and policy_updates_2026.md to support its answer, which is exactly what we would expect.

The more interesting part is the trace, from which we can learn how the agent thinks. We can show the trace in the following way:

for item in result.new_items:
    print(type(item).__name__, item)

result.new_items contains the intermediate tool calls and tool outputs produced by the agent. In my run, I can see that the agent first called search_docs() with keywords like conference hotel, hotel cap, approval, and Berlin. Then, it called list_docs() to inspect the available policy documents. After that, it opened the relevant files with read_doc(). Only then did it produce the final answer.

This is exactly the agentic loop we wanted to see.

3. What to Decide Before Building Agentic RAG

The case study we just went through only scratched the surface. To really build a practical agentic RAG solution, based on my experience, I suggest you answer the following 5 questions:

Q1: How much freedom should the agent have?

One common option is exactly what we have done in the earlier case study: we exposed a couple of carefully curated tools, and the agent is only allowed to use those tools to do the investigation. This is straightforward in terms of controlling, testing, and auditing.

But we can also give the agent broader access, such as shell and file system. This way, the agent can directly run scripts to search and inspect files, and maybe even do further data processing to generate useful artifacts, all on its own.

This pattern can be much more powerful, but it also increases risk and makes behavior harder to predict.

So for most RAG applications, I’d start with curated tools first, and only add shell/file-system access when the task complexity justifies it.

Q2: Should the agent search raw text only?

Most RAG projects might start with plain text like PDFs, wiki pages, manuals, etc. That’s fine.

But in practice, we can often make retrieval easier by deriving a knowledge layer on top of the raw texts.

Those derived knowledge artifacts can be document metadata, summaries, cross-document links, or we can go further and implement a proper knowledge graph.

These derived knowledge artifacts help the agent navigate the corpus, while the raw texts remain as the source of truth.

Q3: Do we still need embeddings?

Agentic RAG doesn’t necessarily mean embeddings are gone.

Vector embeddings are still an efficient way to find semantically relevant texts, and it often outperforms a pure keyword search strategy.

In agentic RAG, what changed essentially is that the retrieval becomes an “action” the agent can take. Under this framing, “action” can still be powered by an embedding-based retriever, a keyword-based one, or even a hybrid one.

So embeddings can still be useful. They are just one possible way to power the agent’s search tool.

Q4: Should one agent handle everything?

The simplest agentic RAG setup is just one agent that does the search, read, and answer.

But as the task gets more complex, you might want to split the work among multiple agents. More concretely, you might need to adopt a multi-agent strategy.

You can split the work by role. For example, the planner-retriever-writer split, where the planner decides what evidence is needed, the retriever collects it, and the writer produces the final answer by using the collected evidence.

You can also split by source type, where each agent is equipped with customized tools and focuses on one specific type of source.

Just keep in mind: A multi-agent setup adds coordination complexity, and there is no guarantee that it will perform better than a single-agent setup. Empirical testing is very important.

Q5: Should we always use agentic RAG?

Maybe not always.

Just because agentic RAG becomes a trendy topic does not necessarily mean you should always default to it.

Agentic RAG gives more flexibility, but that comes with costs. That cost is not only about latency or token cost, but also less predictable agent behavior.

Always start simple, then add agentic loops when the question actually needs iterative retrieval.

Similar Posts

Leave a Reply