OpenViking: AI Agent Context Database
OpenViking: A Revolutionary Context Database for AI Agents
Introduction to OpenViking
The rapid advancement of artificial intelligence has introduced a new challenge: managing context effectively. While vast amounts of data are available, high-quality contextual information remains elusive. Developers building AI agents face several critical issues:
- Fragmented Context: Memories exist in code, resources in vector databases, and skills scattered across different systems.
- Surging Context Demand: Long-running tasks generate continuous context, making simple truncation or compression ineffective.
- Poor Retrieval Effectiveness: Traditional Retrieval-Augmented Generation (RAG) relies on flat storage, lacking a global understanding of information.
- Unobservable Context: The implicit retrieval chain in RAG operates as a black box, making debugging difficult.
- Limited Memory Iteration: Current memory systems only record user interactions, missing agent-specific task memories.
OpenViking addresses these challenges by introducing an innovative context database framework designed specifically for AI agents. It eliminates the need to manage fragmented vector storage and adopts a "file system paradigm" to unify structured organization of memories, resources, and skills. This approach allows developers to build an AI agent’s "brain" akin to managing local files, enabling precise context manipulation through intuitive commands like ls, find, and grep.
Core Features and Innovations
1. Filesystem Management Paradigm
OpenViking redefines how context is structured by treating it as an abstract virtual filesystem. Instead of viewing data as flat text slices, all memories, resources, and skills are mapped to unique URIs under the viking:// protocol.
Example Virtual File Structure:
viking://
├── resources/ # Project documentation, repositories, web pages
│ ├── my_project/
│ │ ├── docs/
│ │ │ ├── api/
│ │ │ │ ├── .abstract # L0 Layer: Quick summary (~100 tokens)
│ │ │ │ └── .overview # L1 Layer: Core information (~2k tokens)
│ │ │ │ ├── auth.md # Full content (L2 Layer)
│ │ │ │ └── endpoints.md
│ │ │ └── tutorials/
│ │ └── src/
├── user/ # User-specific preferences and habits
│ └── memories/
│ ├── preferences/
│ │ ├── writing_style
│ │ └── coding_habits
├── agent/ # Agent skills, instructions, task memories
├── skills/
│ ├── search_code
│ └── analyze_data
└── memories/
└── instructions/
This hierarchical structure enables agents to locate and manipulate information deterministically through standard filesystem commands (ls, find, grep), transforming vague semantic matching into intuitive, traceable operations.
2. Tiered Context Loading
OpenViking introduces a three-tiered context loading system (L0/L1/L2) to optimize token efficiency:
- L0 (Abstract): A concise one-sentence summary for quick retrieval and relevance checking.
- L1 (Overview): Core information and usage scenarios, enabling agents to make decisions during planning phases.
- L2 (Details): Full original content, loaded on demand when deeper analysis is required.
Advantages:
- Reduces token consumption by avoiding unnecessary data loading.
- Prevents model window overflow and noise introduction from excessive context dumping.
3. Directory Recursive Retrieval
To address the limitations of single-vector retrieval, OpenViking implements a "Directory Recursive Retrieval Strategy" that integrates multiple retrieval methods:
- Intent Analysis: Generates multiple retrieval conditions based on user intent.
- Initial Positioning: Uses vector retrieval to quickly locate high-scoring directories where initial context slices are stored.
- Refined Exploration: Performs secondary retrieval within the identified directory, updating high-score results iteratively.
- Recursive Drill-Down: If subdirectories exist, recursively repeats the process layer by layer.
- Result Aggregation: Combines the most relevant context fragments for final output.
This strategy ensures agents retrieve semantically accurate information while maintaining a global understanding of where data resides, improving retrieval accuracy and efficiency.
4. Visualized Retrieval Trajectory
OpenViking’s hierarchical filesystem structure provides a clear, observable path for retrieving context. Unlike traditional flat RAG systems, which operate as black boxes, OpenViking preserves the entire retrieval trajectory—from initial directory positioning to final file selection.
Visualization Example:
Retrieval Path:
1. User query: "Explain Python APIs"
2. Intent analysis → Targets `viking://resources/my_project/docs/api`
3. Vector search → High-score directory: `viking://resources/my_project/docs/api/auth.md`
4. Secondary retrieval → Refined results include `.abstract` and `.overview`
5. Final output: Full content of `auth.md`
This visualization allows developers to trace errors, optimize retrieval logic, and enhance debugging.
5. Automatic Session Management
OpenViking introduces an "automatic session management" loop that enables agents to learn from interactions:
- User Memory Update: Extracts preferences and habits (e.g., writing style, coding habits) for personalized responses.
- Agent Experience Accumulation: Analyzes task execution results and user feedback to update memories in the
agentdirectory. This includes extracting operational tips, tool usage insights, and decision-making patterns.
Result: Agents become "smarter with use," evolving through iterative interactions with the world.
Technical Implementation
Prerequisites
Before deploying OpenViking, ensure your environment meets these requirements:
- Python: 3.10 or higher
- Go: 1.22 or higher (for building AGFS components)
- C++ Compiler: GCC 9+ or Clang 11+
- Operating System: Linux, macOS, Windows
- Network Connection: Required for downloading dependencies and accessing model services
Installation
Python Package Installation
pip install openviking --upgrade --force-reinstall
Rust CLI (Optional)
curl -fsSL https://raw.githubusercontent.com/volcengine/OpenViking/main/crates/ov_cli/install.sh | bash
Or build from source:
cargo install --git https://github.com/volcengine/OpenViking ov_cli
Model Preparation
OpenViking supports multiple Vision-Language Model (VLM) providers:
| Provider | Description |
|----------------|-------------------------------------------------------------------------------------------------|
| volcengine | Volcengine Doubao Models |
| openai | OpenAI Official API |
| litellm | Unified access to third-party models (Anthropic, DeepSeek, Gemini, etc.) |
Example Configuration for Volcengine:
{
"vlm": {
"provider": "volcengine",
"model": "doubao-seed-2-0-pro-260215",
"api_key": "your-api-key",
"api_base": "https://ark.cn-beijing.volces.com/api/v3"
}
}
Example Configuration for OpenAI:
{
"vlm": {
"provider": "openai",
"model": "gpt-4o",
"api_key": "your-openai-api-key",
"api_base": "https://api.openai.com/v1"
}
}
Example Configuration for LiteLLM:
{
"vlm": {
"provider": "litellm",
"model": "claude-3-5-sonnet-20240620",
"api_key": "your-anthropic-api-key"
}
}
Environment Configuration
Create a configuration file ~/.openviking/ov.conf with the following structure:
{
"storage": {
"workspace": "/home/your-name/openviking_workspace"
},
"log": {
"level": "INFO",
"output": "stdout" // or "file"
},
"embedding": {
"dense": {
"api_base": "",
"api_key": "",
"provider": "volcengine", // or "openai"
"dimension": 1024,
"model": ""
},
"max_concurrent": 10
},
"vlm": {
"api_base": "",
"api_key": "",
"provider": "volcengine",
"model": "",
"max_concurrent": 100
}
}
Note: Supported providers for embedding models include volcengine, openai, and jina. For VLM models, supported providers are volcengine, openai, and litellm.
Running OpenViking
Launch Server
openviking-server
For background execution:
nohup openviking-server > /data/log/openviking.log 2>&1 &
CLI Commands
# Check server status
ov status
# Add a resource (e.g., GitHub repo)
ov add-resource https://github.com/volcengine/OpenViking --wait
# List resources
ov ls viking://resources/
# Explore directory structure recursively
ov tree viking://resources/volcengine -L 2
# Search for "openviking"
ov find "what is openviking"
# Grep within a specific URI
ov grep "openviking" --uri viking://resources/volcengine/OpenViking/docs/zh
Deployment and Scaling
Production Deployment
For high-performance, scalable environments, OpenViking can be deployed as a standalone HTTP service. The recommended platform is Volcengine Elastic Compute Service (ECS) with the veLinux operating system.
Step-by-Step Guide:
- Follow the detailed deployment guide: Server Deployment & ECS Setup Guide
OpenClaw Memory Plugin
OpenViking integrates with OpenClaw, an AI agent framework, to enhance memory management. The plugin evaluates its effectiveness using the LoCoMo10 dataset (1,540 long-range dialogue cases).
Experimental Results: | Experimental Group | Task Completion Rate | Cost: Input Tokens (Total) | |----------------------------------------|---------------------|---------------------------| | OpenClaw(memory-core) | 35.65% | 24,611,530 | | OpenClaw + LanceDB (-memory-core) | 44.55% | 51,574,530 | | OpenClaw + OpenViking Plugin (-memory-core)| 52.08% | 4,264,396 | | OpenClaw + OpenViking Plugin (+memory-core) | 51.23% | 2,099,622 |
Key Insights:
- With native memory enabled: 43% improvement over original OpenClaw with a 91% reduction in token cost; 15% improvement over LanceDB with a 96% reduction.
- With native memory disabled: 49% improvement over original OpenClaw with an 83% reduction; 17% improvement over LanceDB with a 92% reduction.
Plugin Documentation: OpenClaw Memory Plugin
Core Concepts and Advanced Features
1. Viking URI
OpenViking’s unique URI format enables precise context manipulation:
viking://<path>
Example:
viking://resources/my_project/docs/api/auth.md
Documentation: Viking URI
2. Context Types
OpenViking categorizes context into three types:
| Type | Description | |-----------|-------------------------------------------------------------------------------------------------| | Resources | Project documentation, repositories, web pages | | User | Personal preferences and habits | | Agent | Skills, instructions, task memories |
Documentation: Context Types
3. Context Layers
OpenViking’s three-tiered context structure ensures efficient retrieval:
- L0 (Abstract): Quick relevance check (~100 tokens)
- L1 (Overview): Core information for decision-making (~2k tokens)
- L2 (Details): Full content, loaded on demand
Documentation: Context Layers
4. Retrieval Mechanism
OpenViking’s retrieval process is designed for accuracy and efficiency:
- Intent analysis → Generate multiple retrieval conditions.
- Initial positioning → Vector search to locate high-score directories.
- Refined exploration → Secondary retrieval within identified directories.
- Recursive drill-down → Repeat for subdirectories if available.
- Result aggregation → Combine most relevant context fragments.
Documentation: Retrieval Mechanism
5. Session Management
OpenViking’s automatic session management loop enables agents to learn from interactions:
- Updates user memories based on preferences and habits.
- Extracts operational tips, tool usage insights, and decision-making patterns for agent improvement.
Documentation: Session Management
Getting Started with VikingBot
VikingBot is an AI agent framework built on OpenViking. Here’s how to get started:
Installation
# Option 1: Install from PyPI (recommended)
pip install "openviking[bot]"
# Option 2: Install from source (for development)
uv pip install -e ".[bot]"
Launch VikingBot Server
openviking-server --with-bot
Interactive Chat
ov chat
Community and Contributions
OpenViking is an open-source project with a growing community. Here’s how you can get involved:
Join the Community:
📱 Lark Group: Scan QR code to join → View QR Code
💬 WeChat Group: Add assistant → View QR Code
🎮 Discord: Join Server
🐦 X (Twitter): Follow us → Follow Us
Contribute: OpenViking welcomes contributions, whether it’s fixing bugs or adding new features. Every line of code helps shape the future of AI agent context management.
License
OpenViking is licensed under the Apache License 2.0. For more details, see the LICENSE file.
Conclusion
OpenViking represents a paradigm shift in AI agent context management by introducing an innovative "file system paradigm" that unifies memories, resources, and skills into a structured virtual filesystem. With its tiered context loading, recursive retrieval strategy, visualized trajectories, and automatic session management, OpenViking addresses the core challenges developers face when building intelligent agents.
By leveraging OpenViking’s advanced features, developers can build AI agents with precise context manipulation, efficient token usage, and self-improving capabilities. Whether deployed in production environments or used as a standalone tool, OpenViking empowers developers to create smarter, more adaptive AI agents that evolve alongside their users.
Join the community today and contribute to shaping the future of AI agent technology! 🚀
Image References:
- OpenViking Banner – Project introduction.
- [Filesystem Structure Diagram] (Conceptual visualization of
viking://filesystem). - [Retrieval Trajectory Visualization] (Illustration of directory recursive retrieval process).
Enjoying this project?
Discover more amazing open-source projects on TechLogHub. We curate the best developer tools and projects.
Repository:https://github.com/volcengine/OpenViking
GitHub - volcengine/OpenViking: OpenViking: AI Agent Context Database
OpenViking is an open-source AI agent context database that provides a revolutionary file system paradigm for managing memories, resources, and skills. It offer...
github - volcengine/openviking