scrapfly-sdk

What it is and what it does

Scrapfly SDK is a Python client library for the Scrapfly cloud web scraping service. It wraps three main API endpoints—Web Scraping, Extraction, and Screenshot—allowing developers to scrape web pages, extract structured data, and capture screenshots programmatically. The SDK handles authentication, request configuration, and response parsing, with built-in support for advanced features like JavaScript rendering, anti-bot bypass (ASP), proxy pool selection, and custom JavaScript execution.

The package integrates with LlamaIndex and LangChain for RAG (Retrieval-Augmented Generation) workflows, enabling developers to scrape web content and feed it directly into LLM training pipelines. It depends on decorator, requests, python-dateutil, loguru, urllib3, and backoff for HTTP handling, logging, and retry logic. Optional extras add asyncio/threading support, Scrapy integration, and a Flask-based webhook server for event handling.

Use it for:

• Scrape JavaScript-heavy websites by enabling cloud-based headless browser rendering and anti-bot bypass.
• Extract structured data from web pages and feed it into LLM pipelines via LlamaIndex or LangChain for RAG systems.
• Rotate through proxy pools and geographic locations to bypass IP-based blocking and geo-restrictions.
• Capture full-page screenshots of websites for visual monitoring or archival purposes.
• Build automated data collection pipelines with retry logic and error handling for unreliable or blocking websites.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Python SDK for the Scrapfly web scraping service, providing access to web scraping, extraction, and screenshot APIs with support for JavaScript rendering, proxy rotation, and anti-bot bypass.

Yes, if you need cloud-based web scraping with anti-bot features and have a Scrapfly account. The SDK is actively maintained, has low install friction, carries a permissive license, and integrates well with LLM frameworks. Install only if you plan to use the Scrapfly service; it is a client library, not a standalone scraper.

Install

scrapfly-sdk on PyPI

pip install scrapfly-sdk

uv add scrapfly-sdk

poetry add scrapfly-sdk

Installing scrapfly-sdk

Before you install

Low install friction with a pure Python wheel and six common runtime dependencies. Active maintenance with a recent release (61 days ago) and ongoing repository activity.

License in practice

BSD permissive license allows commercial and private use with minimal restrictions.

Quickstart

pip install scrapfly-sdk

from scrapfly_sdk import ScrapeConfig, ScrapflyClient

client = ScrapflyClient(api_key="your_api_key")
config = ScrapeConfig(url="https://example.com")
result = client.scrape(config)

Requires a valid Scrapfly API key from https://www.scrapfly.io/ to make actual requests.

Verify before relying

• Whether the optional extras (seepdup, concurrency, scrapy, webhook-server) add meaningful value for typical use cases.
• Performance characteristics and rate limits of the Scrapfly service itself.
• Pricing model and free tier limitations for API usage.

Package facts

Evidence: scrapfly_sdk-0.11.1-py3-none-any.whl

Keywords: scraping, web scraping, data, extraction, scrapfly, sdk, cloud, scrapy

Tags

More Python Modules packages