scrapegraph-py

What it is and what it does

scrapegraph-py is an official Python client for ScrapeGraphAI's managed web scraping API. It wraps a cloud service that handles web page fetching, JavaScript rendering, LLM-based content extraction, and anti-bot measures on your behalf. You send a URL and optional format/extraction prompt; the API returns structured results (markdown, HTML, JSON, screenshots, summaries, or extracted data) without managing browsers, proxies, or LLM keys yourself.

The SDK provides both sync and async interfaces for six main operations: scrape (fetch pages in multiple formats), extract (AI-powered structured data from URLs or raw HTML), search (web search with optional extraction), crawl (multi-page site traversal with depth/link limits), monitor (scheduled change detection via cron), and history (request audit trail). All methods return an ApiResult wrapper with status, data, error, and elapsed_ms fields—no exceptions to catch. It differs from the open-source scrapegraphai library, which runs locally and requires you to manage LLMs, browsers, and infrastructure.

Use it for:

• Scrape product listings and prices from e-commerce sites without writing site-specific parsers.
• Extract structured data (JSON schema) from unstructured web content using AI prompts.
• Monitor competitor websites or pricing pages on a schedule and receive webhook notifications.
• Crawl documentation or blog sites to collect all pages in markdown format for indexing.
• Search the web and extract key information from results in a single API call.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Client SDK for the ScrapeGraphAI managed API, enabling web scraping, structured data extraction, search, crawling, and monitoring through a cloud service with built-in LLM and browser rendering.

Yes, if you need web scraping with minimal infrastructure setup and can absorb the API costs. The SDK is actively maintained, has low install friction, and offers a straightforward path to production. No if you require on-premises data handling, want to avoid per-request billing, or need fine-grained control over LLM selection and browser configuration—use the open-source scrapegraphai library instead.

Install

scrapegraph-py on PyPI

pip install scrapegraph-py

uv add scrapegraph-py

poetry add scrapegraph-py

Installing scrapegraph-py

Before you install

Low install friction with only two runtime dependencies (httpx, pydantic). Active maintenance with a release 14 days ago and recent commits. Beta status and early project age (first release November 2024) mean the API surface may still shift.

License in practice

MIT license permits commercial use, modification, and distribution with minimal restrictions. The SDK itself is open-source, though the underlying API service is a paid cloud offering.

Quickstart

pip install scrapegraph-py

from scrapegraph_py import ScrapeGraphAI

sgai = ScrapeGraphAI()  # reads SGAI_API_KEY from env
result = sgai.scrape("https://example.com")
if result.status == "success":
    print(result.data["results"]["markdown"]["data"])

Requires SGAI_API_KEY environment variable or explicit api_key parameter; the API is a paid cloud service, not free.

Verify before relying

• Rate limits, quota tiers, or per-request costs for the managed API service.
• Whether async client supports all methods mentioned in sync documentation.
• Supported Python versions beyond 3.12 and 3.13 (requires_python states >=3.12 but classifiers list only 3.12 and 3.13).

Package facts

Evidence: scrapegraph_py-2.3.1-py3-none-any.whl

Keywords: ai, api, scraping, sdk, web-scraping

Tags

More Python Modules packages