Search-R1: Train your LLMs to reason and call a search engine with reinforcement learning

Search-R1: Training LLMs to Reason and Call a Search Engine with Reinforcement Learning

Introduction

The landscape of large language models (LLMs) is rapidly evolving toward agents that not only generate text but actively reason, plan, and interact with external tools. Among the most exciting directions is teaching LLMs to call search engines and use retrieved information to reason more accurately. This is the core idea behind Search-R1, a reinforcement learning (RL) framework designed to train reasoning-and-searching interleaved LLMs. By integrating tool use (such as search) directly into the model’s reasoning loop, Search-R1 aims to produce agents that can navigate complex information tasks with improved accuracy and interpretability.

At its heart, Search-R1 is an open-source RL training pipeline that builds on veRL and extends the DeepSeek-R1 family of ideas by enabling interleaved access to search engines. It positions itself as a robust alternative and open solution to proprietary approaches, inviting researchers and developers to explore tool-augmented LLM reasoning in a transparent, configurable environment. The project emphasizes flexibility: multiple RL algorithms, various LLM backbones, and a range of search engines from local retrievers to online services. The result is a modular framework that makes it practical to train, evaluate, and compare reasoning-plus-search strategies across different setups.

[Logo image included below to anchor the post visually]

What is Search-R1? Search-R1 is a reinforcement learning framework for training LLMs to reason and call search engines in a coordinated, multi-turn fashion. It treats the problem as an interactive loop where the model can:

Decide when and what to search
Use retrieved information to refine its reasoning
Execute subsequent tool calls or queries based on updated context
Learn through RL signals to improve both decision quality and search policies

The framework is built to accommodate a spectrum of components:

RL algorithms: PPO, GRPO, REINFORCE (and more in future iterations)
LLM backbones: llama3, Qwen2.5, and other contenders
Search engines: local sparse retrievers (e.g., BM25), local dense retrievers (flat or ANN indexing), and online search APIs (Google, Bing, etc.)
Retrievers and rerankers: off-the-shelf neural rerankers and task-tuned components to improve answer quality
Data and experiments: a fully open pipeline with logs, datasets, and reproducible experiments

The project is designed as an open alternative to proprietary tool-use frameworks, enabling researchers to study how reasoning and tool use can be jointly trained and evaluated. It provides a full end-to-end stack—from data processing and indexing to RL training, inference, and evaluation—while maintaining compatibility with a variety of search backends and model sizes.

[Single-turn usage illustration]

Core Features at a Glance

Search-R1 brings together several capabilities that make tool-augmented reasoning practical and extensible:

Local and external search integration
Support for local sparse retrievers (e.g., BM25)
Support for local dense retrievers (flat indexing and ANN indexing)
Optional online search APIs (Google, Bing, Brave, etc.)
Flexible RL methods
PPO, GRPO, REINFORCE (and more)
Diverse LLM compatibility
Works with models such as llama3, Qwen2.5, and beyond
Tool-augmented reasoning
Interleaved reasoning and search calls
End-to-end RL training with tool use as part of the policy
Easy experimentation and reproducibility
Open-source codebase and logs for different experiment variants
Logs and dashboards for monitoring progress and ablations
Extensible data pipelines
Simple guidelines for adding your own QA data and corpora
JSONL-based corpora format with id and contents fields
Multinode training support (for large models)
Scalable training infrastructure to handle 30B+ LLMs
Clear documentation and community references
Links to papers, code repositories, and community logs
Acknowledgments and related work
Inspirations from related research and tool-use RL systems
Integration with veRL for multiturn tool use and search interactions

Key Images and Visual Aids

The project logo anchors the post and identifies the effort:
An illustrative diagram of the single-turn interaction helps readers picture the core loop:
Preliminary results visuals show the impact of the approach on model behavior:
llm-3b-base with search integration:
Multi-turn search-and-reasoning interaction:

How Search-R1 Works: Architecture and Flow

Search-R1 treats tool use as a trained behavior. The agent learns to reason about whether to search, what to search for, how to interpret retrieved results, and how to incorporate evidence into subsequent reasoning steps. A typical training and inference loop comprises the following stages:

1) Data preparation and indexing

Corpora can be any text collection; the recommended format is jsonl lines with an id and contents field.
The contents field typically contains the passage, with the title and the body, to provide rich context for retrieval.
Optional indexing steps enable both sparse and dense retrieval.
If you have a local retriever, you can index your corpus using the provided scripts, enabling fast retrieval during RL rollouts.

2) Retriever and search engine integration

Local retrievers: BM25 for sparse retrieval; dense retrievers with flat or ANN indexing for faster, semantically aware search.
Online search: API hooks to Google, Bing, Brave, and other engines, enabling live web search when needed.
The LLM calls a retriever service via a defined API endpoint (e.g., http://127.0.0.1:8000/retrieve). The retriever runs in a separate process or service to decouple search from RL training.

3) RL training loop

The core loop uses an RL objective to optimize the agent’s ability to reason and search effectively.
Supported RL methods include PPO, GRPO, and REINFORCE, enabling experimentation with different policy optimization strategies.
The agent’s actions include selecting search queries, issuing search calls, processing retrieved passages, and generating final answers with justification.

4) Inference and evaluation

After training, you can run inference against your own questions.
A local retrieval server is launched, and the trained model is asked to respond with reasoning and search steps.
You can tailor the question prompts and the evaluation setup to align with your tasks.

5) Data formats and prompts

For QA datasets used in training, each sample includes:
data_source: source of data
prompt: an array of user messages (e.g., the question)
ability: a descriptor such as "fact-reasoning"
reward_model: style and ground-truth solution
extra_info: split and index metadata
A concrete example and data processing script (nq_search.py) illustrate how to prepare data for Search-R1.

6) Use-your-own dataset and corpora

You can adapt your own data by following the provided schema and scripts.
Corpora are encouraged to be in jsonl format, with each line including an id and contents field.
Examples show including quotes around titles and content to preserve structure and enable effective retrieval.

Implementation and Quick Start

If you are ready to try Search-R1, the project provides a quick-start guide with concrete steps. The installation and setup are designed to be modular, allowing you to mix-and-match retrievers, LLMs, and search engines.

Installation Overview

Create and activate the primary environment for Search-R1:
conda create -n searchr1 python=3.9
conda activate searchr1
Core dependencies:
torch 2.4.0 with CUDA 12.1
vllm 0.6.3 (or compatible versions 0.5.4, 0.4.2, 0.3.1)
verl (the reinforcement learning framework)
flash-attn
wandb for experiment logging
Optional: a separate environment for retrievers
conda create -n retriever python=3.10
conda activate retriever
Install PyTorch with CUDA for FAISS-GPU
faiss-gpu 1.8.0
pip install transformers datasets pyserini
Install API support: uvicorn, fastapi
Quick-start example tasks:
Download indexing data and corpus
Process the NQ dataset
Launch a local retrieval server
Run RL training with a selected LLM (e.g., Llama-3.2-3b-base)

Sample installation and quick-start commands (as cited in the source)

Installation notes:
Concreate commands to set up the environment and dependencies are provided in the official docs, including the steps above.
Quick start workflow: 1) Download indexing and corpus
- save_path=/the/path/to/save
- python scripts/download.py --savepath $savepath
- cat $savepath/part* > $savepath/e5Flat.index
- gzip -d $save_path/wiki-18.jsonl.gz 2) Process the NQ dataset
- python scripts/dataprocess/nqsearch.py 3) Launch a local retrieval server
- conda activate retriever
- bash retrieval_launch.sh 4) Run RL training (PPO) with a chosen model
- conda activate searchr1
- bash train_ppo.sh

Preliminary Results and What They Show

Search-R1’s initial experiments demonstrate meaningful improvements in how LLMs reason and interact with search engines:

Base model llama3.2-3b-base learns to call the search engine and achieve improved performance
Visualization of performance gains is provided in the project visuals:
- llama-3b image:
Base model Qwen2.5-7b-base exhibits multi-turn search engine calling with RL
The multi-turn reasoning and search interplay is showcased in:
- multi-turn image:

Inference and Playing with Search-R1

Once a trained Search-R1 model is available, you can test its reasoning-and-searching abilities with your own questions:

Step 1: Start the retrieval server
conda activate retriever
bash retrieval_launch.sh
Step 2: Run inference
conda activate searchr1
python infer.py
Customizing your question
You can edit the question variable within the inference script or supply a new question to spark a search-driven reasoning path

Use Your Own Dataset and Corpora QA data

For each QA sample, define a dictionary with:
data_source
prompt: a list containing user content (e.g., the question)
ability: "fact-reasoning"
reward_model: with style and ground-truth solution
extra_info: split and index
A concrete example and a reference data processing script (nq_search.py) are provided to help you prepare your own data.

Corpora

It is recommended that corpora be stored as jsonl, with each line containing:
id: passage id
contents: the passage text (often a combination of title and content)
Example format (simplified): {"id": "0", "contents": "\"Evan Morris\" Evan L. Morris (January 26, 1977 – July 9, 2015) was a lobbyist for Genentech and its parent corporation Roche in Washington."} … {"id": "100", "contents": "Three years later, when the United States Exploring Expedition…"}

Indexing and Custom Retrievers

You can index your own corpus with searchr1/search/buildindex.sh
You can swap retrievername and retrievermodel to match your preferred off-the-shelf retriever systems
The codebase supports a range of local retrievers (sparse and dense) as well as online search engines

Use Your Own Search Engine

The architecture separates the search engine from the RL training loop, allowing you to:
Run a local sparse retriever (BM25)
Run a local dense retriever with GPU acceleration or CPU-based ANN indexing
Use external online search engines via accessible APIs
The Search-R1 codebase provides a retriever_server.py example to launch a local search server, enabling the LLM to call the search endpoints as part of its reasoning

What to Expect: Features Recap

Local sparse retrievers (BM25) supported
Local dense retrievers with both flat and ANN indexing supported
Online search API support (Google, Bing, Brave, etc.)
Off-the-shelf neural rerankers compatibility
Multiple RL methods available
A variety of LLM backbones supported
End-to-end data processing and experiment logging
Multinode training for large-scale LLMs
Extensible documentation and community resources

Acknowledgments and Inspirations

Search-R1 draws inspiration from several related efforts and open-source projects. The concept of training LLMs to reason with search is influenced by DeepSeek-R1 and TinyZero, while its implementation leans on veRL and related tool-use RL frameworks like RAGEN. The project acknowledges the contributions of these teams to open-source AI research, and it builds upon their ideas to deliver a practical, extensible RL toolkit for tool-augmented reasoning.

DeepSeek-R1
TinyZero
veRL
RAGEN

Additionally, the project surfaces a wide ecosystem of related works that explore the idea of empowering LLMs with search, reasoning, and tool use. Readers are encouraged to explore the listed resources and the ongoing experiments to see how different design choices influence performance and interpretability.

News and Milestones

The Search-R1 project has documented a series of milestones and community updates that highlight its momentum:

2025.10: Featured by Thinking Machines Lab’s first product Tinker; documentation available for tool-use recipes including search
2025.07: Supported by SkyRL with detailed examples and documentation
2025.06: Integrated with the latest veRL features to harness multiturn tool use
2025.05: Second paper detailing empirical studies published
2025.04: Multinode training and various search engine integrations introduced
2025.03: First Search-R1 paper and initial logs released
2025.02: Open-sourced codebase with preliminary results

Notes on Citations

For readers who want to cite Search-R1 in academic work, the project provides BibTeX entries for two key papers:

@article{jin2025search, title={Search-r1: Training llms to reason and leverage search engines with reinforcement learning}, author={Jin, Bowen and Zeng, Hansi and Yue, Zhenrui and Yoon, Jinsung and Arik, Sercan and Wang, Dong and Zamani, Hamed and Han, Jiawei}, journal={arXiv preprint arXiv:2503.09516}, year={2025} }
@article{jin2025empirical, title={An Empirical Study on Reinforcement Learning for Reasoning-Search Interleaved LLM Agents}, author={Jin, Bowen and Yoon, Jinsung and Kargupta, Priyanka and Arik, Sercan O and Han, Jiawei}, journal={arXiv preprint arXiv:2505.15117}, year={2025} }

Citations are provided to encourage readers to explore the research continuum and the related empirical work that underpins Search-R1.

Conclusion

Search-R1 represents a meaningful step toward practical, tool-augmented LLM agents. By combining robust reinforcement learning techniques with flexible search engines and diverse LLM backbones, it enables researchers to study how reasoning and search can be coordinated, how tools can be integrated into multi-turn conversations, and how such capabilities can be evaluated in reproducible, open settings. The project’s design emphasizes modularity, extensibility, and openness, making it possible to adapt and extend the framework to new tasks, new search modalities, and new model architectures as the field evolves.

If you’re exploring tool-based reasoning with LLMs, Search-R1 offers a ready-to-run pipeline that you can customize for your datasets, corpora, and preferred search tooling. The combination of open-source code, clear documentation, and a modular architecture helps accelerate experimentation, enabling researchers to probe questions like: How does interleaved reasoning with search affect answer quality? Which RL objective best promotes efficient tool use? How does the choice of retriever influence the agent’s decision to search? What is the role of multi-turn dialogue in robust information gathering? These questions sit at the heart of building more capable and trustworthy AI agents, and Search-R1 provides a practical platform to begin answering them.