---
id: web-forager
version: "3.0.1"
license: MIT
license_treatment: permissive
maintenance: active
---

# web-forager — A search-and-fetch toolkit for AI agents — MCP server and standalone Agent Skills powered by DuckDuckGo and Jina Reader

License: permissive · Maintenance: active · Downloads: 328.9K/mo

## What it is and what it does

Web Forager is a search-and-fetch toolkit designed for AI agents and LLM workflows. It wraps DuckDuckGo web search, DuckDuckGo news search, and Jina Reader into a unified interface that can run as an MCP server (for integration with Claude, Cursor, and other MCP clients) or as standalone command-line tools and Python Agent Skills. The package converts web pages and search results into markdown or JSON formats optimized for LLM consumption, handling the common friction points of web scraping — cookie banners, dynamic content, and inconsistent HTML structure — by falling back through multiple fetching strategies.

The toolkit ships with six runtime dependencies: mcp and fastmcp for MCP server functionality, requests and urllib3 for HTTP transport, beautifulsoup4 for HTML parsing, and ddgs for DuckDuckGo integration. It supports Python 3.10 through 3.13 and is actively maintained. The package is positioned as a research and fact-checking aid for AI agents, with specialized Agent Skills for competitive analysis and technology evaluation.

Use it for:

- Integrate web search into Claude Desktop or Cursor via MCP to give LLMs real-time web access.
- Build autonomous research agents that fetch and summarize news articles and web pages.
- Fact-check claims by searching DuckDuckGo and converting results into structured data for LLM analysis.
- Monitor competitive products and technologies by automating news search and page fetching.
- Use standalone Agent Skills in Python scripts for specialized research workflows without MCP overhead.

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

Web Forager is a search-and-fetch toolkit for AI agents that retrieves web content via DuckDuckGo, news search, and Jina Reader, then converts results into LLM-friendly formats.

Yes, if you need web search and page fetching for AI agents. The package is actively maintained, has low install friction, carries a permissive MIT license, and integrates cleanly with MCP clients like Claude Desktop. No known vulnerabilities. The main gotcha is Python version support (3.10–3.13 only) and potential rate-limiting or API stability issues with DuckDuckGo and Jina Reader, which the fact sheet does not detail.

## Install

pip install web-forager
uv add web-forager
poetry add web-forager

## Installing web-forager

Before you install: Low install friction with a pure Python wheel. Requires Python 3.10 or higher. Actively maintained with a recent commit on 2026-08-09 and six runtime dependencies (mcp, fastmcp, requests, urllib3, beautifulsoup4, ddgs) that are themselves well-established.

License in practice: MIT license is permissive, allowing use in commercial and private projects with minimal restrictions — only requiring attribution.

Quickstart:

pip install web-forager

# CLI usage
web-forager search "your query" --max-results 5
web-forager news "your query" --max-results 10
web-forager fetch "https://example.com" --format markdown

# Or start as MCP server
web-forager serve

Requires Python 3.10 or higher; Python 3.14 is not yet supported.

Verify before relying:

- Whether Jina Reader API access requires authentication or has rate limits.
- DuckDuckGo API stability and any terms-of-service constraints for automated queries.
- Performance characteristics when handling large result sets or long-form page content.

## Package facts

- License: MIT (permissive)
- Python support: supports_current
- Install friction: low
- Maintenance: active
- Downloads: 328.9K/month (top 15,000 on PyPI)
- Known vulnerabilities: none known

## Tags

web search for ai agents, mcp server duckduckgo, fetch and convert web pages, llm web scraping toolkit, agent research tools, news search integration, jina reader wrapper, mcp-server, ai-agents, web-scraping

[View on SkillFed](https://skillfed.io/packages/web-forager) · [View on PyPI](https://pypi.org/project/web-forager/)