Reader: Read and Search for LLMs

Reader by Jina AI: A Deep Dive into Turning the Web into LLM-Ready Input

Introduction

In the ever-growing world of large language models, the quality of input matters as much as the model itself. Reader from Jina AI is designed to bridge the gap between raw web content and the needs of modern LLMs. It converts any URL into an LLM-friendly input, enabling smoother reasoning, retrieval, and multiturn conversations with your agents and RAG systems. It also offers robust web search capabilities so your models can access the latest knowledge from the open web. This post explores what Reader is, what it can read, how to use it effectively, and how you can self-host and develop with it—backed by hands-on details and practical guidance.

Overview: Read and Search at a Glance

Reader sits at the intersection of content ingestion and search, offering two core capabilities:

• Read: URL to LLM-friendly input

• How it works: Prepend https://r.jina.ai/ to any URL to obtain a streamlined, model-friendly representation of the page.

• Benefit: Improve outputs for agents and RAG systems with minimal fuss and no cost.

• Example: https://r.jina.ai/https://en.wikipedia.org/wiki/Artificial_intelligence

• Search: Web-wide queries for fresh knowledge

• How it works: Use https://s.jina.ai/your+query to fetch top results, then Reader visits each URL and applies r.jina.ai to extract content.

• Benefit: Access up-to-date information beyond static documents.

Live demos and status

Reader offers live demonstrations and an active development cycle. See the live demo at the official page and explore the capabilities firsthand:

• Live demo: https://jina.ai/reader#demo

The project is actively maintained as one of Jina AI’s core products, with a focus on stability, scalability, and accessibility. For visibility into quality, you can observe the project’s status via external badges such as the code coverage badge:

• Code coverage:

Support and community guidance

Reader is complemented by community-driven insights and documentation from DeepWiki, which provides a badge that points to the Jina AI/Reader project:

• DeepWiki badge:

What Reader can read: content types and how they’re processed

Reader is designed to handle a variety of content formats while preserving useful structure and context for downstream LLMs.

• Web pages

• Rendering options: Reader chooses intelligently between headless Chrome rendering and lightweight curl-impersonation.

• Outcome: A readable, SEO-friendly representation of page content suitable for LLM processing.

• PDFs

• Processing: Any URL ending in .pdf is parsed with PDF.js and converted to markdown.

• Example reference: NASA’s PDF result demonstrates the transformation from the original to a markdown-friendly form.

• Benefit: Complex documents become digestible text for LLMs.

• MS Office documents (Word, Excel, PowerPoint)

• Processing: Converted via LibreOffice, then processed as HTML or PDF for robust downstream consumption.

• Images

• Vision-language captioning: Images are captioned by a vision-language model, giving your text-only LLM enough context to reason about the visuals without embedding heavy image data.

Usage: Quick start guides and practical examples

Using Reader is straightforward, with clear entry points for single URLs, entire sites, and search workflows.

Single URL fetching (r.jina.ai)

• Concept: Prefix a URL with https://r.jina.ai/ to obtain an LLM-ready rendition.
• Example:
  https://r.jina.ai/https://en.wikipedia.org/wiki/Artificial_intelligence
• Why it works: This path minimizes rendering complexity while preserving essential content, formatting, and links where appropriate.

Full website fetching (Google Colab example)

• Practical guidance: For larger tasks, you can use a Colab notebook to fetch an entire site or run batch operations via r.jina.ai. This allows you to prototype large-scale reads without local setup friction.
• Colab link: Provided in the Reader docs (example here for quick access).

Web search with s.jina.ai

• Concept: Use https://s.jina.ai/Your+Query to search the web and retrieve top results.
• Example:
  https://s.jina.ai/Who%20will%20win%202024%20US%20presidential%20election%3F
• Process: Reader gathers the top 5 results, visits each URL, and applies r.jina.ai to extract content, giving you a thorough reading path without manual page scrapes.

In-site search with s.jina.ai

• Example syntax: curl 'https://s.jina.ai/When%20was%20Jina%20AI%20founded%3F?site=jina.ai&site=github.com'
• Benefit: Focus searches within specific domains, enabling precise knowledge retrieval.

Interactive code exploration: Builder and examples

• Interactive Code Snippet Builder: A hands-on tool to explore parameter combinations for the Reader API.
• Visual cue: The builder is presented with an illustrative image to guide experimentation.
• Builder image:

Header controls: Tailoring Reader’s behavior

Reader supports a rich set of request headers to control the output format, latency, content, and formatting. The defaults are designed for usability, but advanced users can fine-tune behavior for production systems. The core categories include output format, engine selection, caching, and content retention.

Key headers and their effects (highlights)

• x-respond-with: Choose the output format:
• markdown: Markdown output, without readability filtering
• html: raw documentElement.outerHTML
• text: plain text from the body
• screenshot: URL of the webpage’s screenshot
• pageshot: whole-page screenshot
• frontmatter: Markdown with YAML front matter
• markdown+frontmatter: full-page Markdown with front matter
• x-engine: Enforces the fetching engine (browser, curl, or auto)
• x-proxy-url: Route traffic through a custom proxy
• x-cache-tolerance: Acceptable staleness for cached pages (seconds)
• x-no-cache: Bypass cache entirely
• x-target-selector: Content extraction within a specific CSS selector
• x-wait-for-selector: Wait for a specific element to render
• x-timeout: Maximum wait time (seconds; up to 180)
• x-max-tokens: Cap on output length (minimum 500)
• x-token-budget: Rejection threshold if content would exceed tokens
• x-respond-timing: When to return results (html, visible-content, mutation-idle, resource-idle, media-idle, network-idle)
• x-with-generated-alt: Caption images with VLM-generated alt text
• x-retain-images: How images appear in output (all, none, alt, text, image, html)
• x-retain-links: How links appear (all, none, text)
• gpt-oss: Citations format for OSS-friendly workflows
• x-with-links-summary / x-with-images-summary: Append deduplicated footers with links or images
• x-markdown-chunking: Return content as structured chunks (JSON array or delimited text)
• x-preset: Apply predefined option bundles (reader, index, research, agent, spider)
• x-detach-invisibles: Remove invisible elements before snapshot
• x-md-*: Fine-tune Markdown output parameters
• And many more: The docs provide an exhaustive reference for these options.

SPA (Single Page Application) support and tips

Reader’s architecture natively supports SPAs via Puppeteer and headless Chrome. Some SPA-specific challenges require tailored approaches:

• Hash-based routing: Use a POST with the URL in the body if the path includes a hash fragment (e.g., #/route).
• Preloaded content: Some apps render preload content; you can mitigate by waiting for network idle or a specific element:
• curl 'https://r.jina.ai/https://example.com/' -H 'x-timeout: 10'
• curl 'https://r.jina.ai/https://example.com/' -H 'x-wait-for-selector: #content'
• Combined: curl 'https://r.jina.ai/https://example.com/' -H 'x-timeout: 30' -H 'x-wait-for-selector: non-existent-element'
• JSON mode and other advanced fetch options also apply to SPA content.

Generated alt text for images

Reader can auto-caption images using a Vision-Language Model (VLM) to provide context-friendly alt text for downstream LLMs:

• Example: curl -H "X-With-Generated-Alt: true" https://r.jina.ai/https://en.m.wikipedia.org/wiki/Main_Page

Cookbooks: recipes for pipelines and use cases

The Reader project includes a set of cookbooks that provide pipeline-specific recipes for tasks such as:

• RAG and semantic indexing
• Deep research workflows
• Agentic browsing
• Visual snapshots
• PDF/Office/HTML uploads Each cookbook offers concise curl examples with the header combinations best suited for that scenario, plus trade-off notes.

Self-hosting with Docker: getting started

Reader ships a prebuilt open-source image that bundles headless Chrome, LibreOffice, and CJK fonts for a ready-to-run experience.

• Pull the OSS image:

• docker pull ghcr.io/jina-ai/reader:oss

• Quick run (no caching):

• docker run --rm -p 3000:8081 ghcr.io/jina-ai/reader:oss

• Then: curl http://localhost:3000/https://example.com

• Full deployment with h2c support (production-grade):

• docker run --rm -p 3000:8080 -p 3001:8081 ghcr.io/jina-ai/reader:oss

• Caching with S3-compatible storage:

• docker run --rm -p 3000:8081 \

  -e GCPSTORAGEENDPOINT=https://s3.example.com \ -e GCPSTORAGEBUCKET=reader-cache \ -e GCPSTORAGEACCESSKEY=… \ -e GCPSTORAGESECRETKEY=… \ ghcr.io/jina-ai/reader:oss

Local development and contribution

• Requirements: Node.js environment (nvm) and Docker (optional for bucket caching)
• Steps:
• git clone https://github.com/jina-ai/reader.git
• cd reader
• npm install
• Optional: docker compose up -d (for a local bucket cache)
• Run the dev server: npm run dev
• For deeper exploration of the codebase, consult architecture.md and CONTRIBUTING.md for development workflows, environment variables, testing, and deployment guidelines.

Licensed assets and build-time resources

Reader includes some licensed assets required for rendering and fonts. The repository provides scripts to download these assets when needed:

• GeoLite2-City.mmdb and geolite2-asn.mmdb (MaxMind)
• SourceHanSansSC-Regular.otf (Source Han Sans for CJK rendering)
• gsa_useragents.txt (user-agent list for curl engine)
• Asset download helper: npm run assets:download

The assets script is idempotent and designed to be robust against partial network failures, with options to force re-download or skip entirely.

How Reader works, and what powers it

Reader is built to provide a cohesive, production-ready pathway from any URL to a usable, model-friendly input. It leverages a combination of rendering strategies, content extraction logic, and output formatting controls. The Open Source branch mirrors the SaaS codebase behind the r.jina.ai and s.jina.ai endpoints, with stateless operation by default and optional bucket caching to accelerate repeated fetches.

Having trouble on some websites? Practical knobs Not all sites cooperate with automated fetchers. Reader offers practical knobs to increase success:

• Use an API key for higher quotas and access to internal proxy capabilities.
• Bypass the cache with -H 'x-no-cache: true' to force fresh fetches.
• Force the browser engine with -H 'x-engine: browser' when JS-heavy sites render content only to browsers.
• Route traffic through the hosted SaaS proxy with -H 'x-proxy: auto' (key required). You can pin a country for geo-specific behavior.
• Bring your own proxy via -H 'x-proxy-url: https://user:pass@host:port'
• If all else fails, open an issue with the URL and headers you tried so the team can investigate.

Legal and licensing

Reader is backed by Jina AI and licensed under Apache-2.0. The project emphasizes open collaboration and practical tooling for developers who rely on robust, scalable content ingestion for LLMs.

Visuals and contextual imagery

To complement the narrative and offer quick visual anchors, Reader includes a few representative visuals sourced from the project assets:

• Code coverage and project badges illustrating status and community trust:
• Code coverage badge: https://codecov.io/gh/jina-ai/reader/branch/main/graph/badge.svg
• Live demonstrations and community cues:
• DeepWiki badge: https://deepwiki.com/badge.svg
• Handy visuals for developer walkthroughs and interactive tooling:
• Interactive Code Snippet Builder image: https://github.com/jina-ai/reader/assets/2041322/a490fd3a-1c4c-4a3f-a95a-c481c2a8cc8f
• Additional project visuals illustrating repository structure or examples:
• Image 1: https://github.com/jina-ai/reader/assets/2041322/2067c7a2-c12e-4465-b107-9a16ca178d41
• Image 2: https://github.com/jina-ai/reader/assets/2041322/675ac203-f246-41c2-b094-76318240159f

Conclusion: A practical bridge from the web to intelligent agents

Reader offers a pragmatic, battle-tested approach to transforming web content into actionable, context-rich inputs for LLMs. By supporting multiple content types (web pages, PDFs, Office docs, and images with captions), providing two complementary modes (reading a single URL or searching the web), and exposing a rich set of headers and configuration options, Reader empowers developers to tailor their pipelines to match the needs of RAG, semantic indexing, AI research, agentic browsing, and more.

If you’re building a system that relies on timely knowledge, precise document understanding, or robust content ingestion, Reader provides a solid foundation. Its open-source nature means you can adapt, extend, and host it in your own environment to meet stringent latency, privacy, and scale requirements. Explore the live demos, experiment with the code builder, and consider the Docker-based self-host path to bring your own Reader-powered pipelines into production.

Images in this post are drawn from the input materials to illustrate key concepts and workflow moments:

• Code coverage and project status: https://codecov.io/gh/jina-ai/reader/branch/main/graph/badge.svg
• DeepWiki guidance: https://deepwiki.com/badge.svg
• Interactive code builder: https://github.com/jina-ai/reader/assets/2041322/a490fd3a-1c4c-4a3f-a95a-c481c2a8cc8f
• Visual cues from the Reader repository: https://github.com/jina-ai/reader/assets/2041322/2067c7a2-c12e-4465-b107-9a16ca178d41 and https://github.com/jina-ai/reader/assets/2041322/675ac203-f246-41c2-b094-76318240159f

This combination of features, practical guidance, and open tooling makes Reader a compelling choice for teams looking to streamline the ingestion-to-insight workflow for modern LLM-powered applications.

[![Featured on TechLogHub](https://techloghub.com/badge-neutral.svg)](https://techloghub.com/open-source/reader-read-and-search-for-llms?utm_source=techloghub_badge&utm_medium=referral&utm_campaign=readme)