mcp DEEPwebresearch

Local 2025-08-31 23:18:32 0

A Model Context Protocol server that enables Claude to perform advanced web research with intelligent search queuing, enhanced content extraction, and deep research capabilities.

A Model Context Protocol (MCP) server for advanced web research.

Latest Changes

Added visit_page tool for direct webpage content extraction
Optimized performance to work within MCP timeout limits
Reduced default maxDepth and maxBranching parameters
Improved page loading efficiency
Added timeout checks throughout the process
Enhanced error handling for timeouts

This project is a fork of mcp-webresearch by mzxrai, enhanced with additional features for deep web research capabilities. We're grateful to the original creators for their foundational work.

Bring real-time info into Claude with intelligent search queuing, enhanced content extraction, and deep research capabilities.

Features

Intelligent Search Queue System
Batch search operations with rate limiting
Queue management with progress tracking
Error recovery and automatic retries
Search result deduplication
Enhanced Content Extraction
TF-IDF based relevance scoring
Keyword proximity analysis
Content section weighting
Readability scoring
Improved HTML structure parsing
Structured data extraction
Better content cleaning and formatting
Core Features
Google search integration
Webpage content extraction
Research session tracking
Markdown conversion with improved formatting

Prerequisites

Node.js >= 18 (includes npm and npx)
Claude Desktop app

Installation

Global Installation (Recommended)

# Install globally using npm
npm install -g mcp-deepwebresearch

# Or using yarn
yarn global add mcp-deepwebresearch

# Or using pnpm
pnpm add -g mcp-deepwebresearch

Local Project Installation

# Using npm
npm install mcp-deepwebresearch

# Using yarn
yarn add mcp-deepwebresearch

# Using pnpm
pnpm add mcp-deepwebresearch

Claude Desktop Integration

After installing the package, add this entry to your claude_desktop_config.json:

Windows

{
  "mcpServers": {
    "deepwebresearch": {
      "command": "mcp-deepwebresearch",
      "args": []
    }
  }
}

Location: %APPDATA%Claudeclaude_desktop_config.json

macOS

{
  "mcpServers": {
    "deepwebresearch": {
      "command": "mcp-deepwebresearch",
      "args": []
    }
  }
}

Location: ~/Library/Application Support/Claude/claude_desktop_config.json

This config allows Claude Desktop to automatically start the web research MCP server when needed.

First-time Setup

After installation, run this command to install required browser dependencies:

npx playwright install chromium

Usage

Simply start a chat with Claude and send a prompt that would benefit from web research. If you'd like a prebuilt prompt customized for deeper web research, you can use the agentic-research prompt that we provide through this package. Access that prompt in Claude Desktop by clicking the Paperclip icon in the chat input and then selecting Choose an integration → deepwebresearch → agentic-research.

Tools

deep_research
Performs comprehensive research with content analysis

Arguments:

{
  topic: string;
  maxDepth?: number;      // default: 2
  maxBranching?: number;  // default: 3
  timeout?: number;       // default: 55000 (55 seconds)
  minRelevanceScore?: number;  // default: 0.7
}

Returns:

{
  findings: {
    mainTopics: Array<{name: string, importance: number}>;
    keyInsights: Array<{text: string, confidence: number}>;
    sources: Array<{url: string, credibilityScore: number}>;
  };
  progress: {
    completedSteps: number;
    totalSteps: number;
    processedUrls: number;
  };
  timing: {
    started: string;
    completed?: string;
    duration?: number;
    operations?: {
      parallelSearch?: number;
      deduplication?: number;
      topResultsProcessing?: number;
      remainingResultsProcessing?: number;
      total?: number;
    };
  };
}

parallel_search
Performs multiple Google searches in parallel with intelligent queuing
Arguments: { queries: string[], maxParallel?: number }
Note: maxParallel is limited to 5 to ensure reliable performance
visit_page
Visit a webpage and extract its content
Arguments: { url: string }

Returns:

{
  url: string;
  title: string;
  content: string;  // Markdown formatted content
}

Prompts

`agentic-research`

A guided research prompt that helps Claude conduct thorough web research. The prompt instructs Claude to: - Start with broad searches to understand the topic landscape - Prioritize high-quality, authoritative sources - Iteratively refine the research direction based on findings - Keep you informed and let you guide the research interactively - Always cite sources with URLs

Configuration Options

The server can be configured through environment variables:

MAX_PARALLEL_SEARCHES: Maximum number of concurrent searches (default: 5)
SEARCH_DELAY_MS: Delay between searches in milliseconds (default: 200)
MAX_RETRIES: Number of retry attempts for failed requests (default: 3)
TIMEOUT_MS: Request timeout in milliseconds (default: 55000)
LOG_LEVEL: Logging level (default: 'info')

Error Handling

Common Issues

Rate Limiting
Symptom: "Too many requests" error
Solution: Increase SEARCH_DELAY_MS or decrease MAX_PARALLEL_SEARCHES
Network Timeouts
Symptom: "Request timed out" error
Solution: Ensure requests complete within the 60-second MCP timeout
Browser Issues
Symptom: "Browser failed to launch" error
Solution: Ensure Playwright is properly installed (npx playwright install)

Debugging

This is beta software. If you run into issues:

Check Claude Desktop's MCP logs:

# On macOS
tail -n 20 -f ~/Library/Logs/Claude/mcp*.log

# On Windows
Get-Content -Path "$env:APPDATAClaudelogsmcp*.log" -Tail 20 -Wait

Enable debug logging:
```
export LOG_LEVEL=debug
```

Development

Setup

# Install dependencies
pnpm install

# Build the project
pnpm build

# Watch for changes
pnpm watch

# Run in development mode
pnpm dev

Testing

# Run all tests
pnpm test

# Run tests in watch mode
pnpm test:watch

# Run tests with coverage
pnpm test:coverage

Code Quality

# Run linter
pnpm lint

# Fix linting issues
pnpm lint:fix

# Type check
pnpm type-check

Contributing

Fork the repository
Create your feature branch (git checkout -b feature/amazing-feature)
Commit your changes (git commit -m 'Add some amazing feature')
Push to the branch (git push origin feature/amazing-feature)
Open a Pull Request

Coding Standards

Follow TypeScript best practices
Maintain test coverage above 80%
Document new features and APIs
Update CHANGELOG.md for significant changes
Follow semantic versioning

Performance Considerations

Use batch operations where possible
Implement proper error handling and retries
Consider memory usage with large datasets
Cache results when appropriate
Use streaming for large content

Requirements

Node.js >= 18
Playwright (automatically installed as a dependency)

Verified Platforms

macOS
Windows
Linux

License

MIT

Credits

This project builds upon the excellent work of mcp-webresearch by mzxrai. The original codebase provided the foundation for our enhanced features and capabilities.

Author

qpd-v