spider-client

What it is and what it does

spider-client is a Python SDK that wraps the Spider Cloud API, a managed web scraping and crawling service. It provides methods to scrape individual URLs, crawl entire websites, extract links, capture screenshots, transform HTML to markdown, and bypass anti-bot protections. The SDK also includes AI-powered methods (requiring a separate subscription) for guided scraping and crawling using natural language prompts, plus an Unlimited plan tier that bills by concurrency seats rather than per-request credits.

The package handles the HTTP communication with Spider Cloud's backend, manages streaming responses, supports multiple output formats (JSON, CSV, XML, JSONL), and formats scraped content for compatibility with language models. It relies on requests for HTTP calls, ijson for streaming JSON parsing, tenacity for retry logic, aiohttp for async operations, and spider-browser for browser automation features.

Use it for:

• Scrape product listings or pricing data from e-commerce sites and format for LLM ingestion.
• Crawl documentation sites or knowledge bases to build training datasets for AI models.
• Extract all links from a website for SEO analysis or site mapping.
• Bypass anti-bot protections on protected sites using the unblocker feature with AI extraction.
• Automate screenshot capture of web pages for visual monitoring or testing.
• Stream large crawl results in chunks to handle websites with hundreds or thousands of pages.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Python SDK for the Spider Cloud API that enables website scraping, crawling, link extraction, screenshot capture, and LLM-compatible data collection through a managed cloud service.

Yes, if you need a managed web scraping solution with built-in anti-bot handling and LLM-ready output formatting. The SDK is actively maintained, has no known vulnerabilities, and low install friction. However, it requires a Spider Cloud account and API key—this is a client library for a paid service, not a standalone scraper. Suitable for production use when the service cost aligns with your scraping volume.

Install

spider-client on PyPI

pip install spider-client

uv add spider-client

poetry add spider-client

Installing spider-client

Before you install

Low friction installation with five runtime dependencies (requests, ijson, tenacity, aiohttp, spider-browser). Package is actively maintained with recent releases; last commit on 2026-07-19 and version 0.1.93 released 2026-07-18.

License in practice

MIT license permits commercial and private use with minimal restrictions. You may use, modify, and distribute the package freely provided you include the license notice.

Quickstart

pip install spider_client

from spider import Spider

app = Spider(api_key='your_api_key')
scraped_data = app.scrape_url('https://example.com')

Requires a Spider Cloud API key (obtain from spider.cloud); set as SPIDER_API_KEY environment variable or pass directly to Spider class.

Verify before relying

• Whether spider-browser dependency requires system libraries or browser binaries to be pre-installed.
• Specific Python version support (requires_python not specified in metadata).
• Rate limiting or quota behavior when using the Unlimited plan with concurrent requests.

Package facts

Evidence: spider_client-0.1.93-py3-none-any.whl

Tags

More Python Modules packages