scrapydo
Crochet-based blocking API for Scrapy.
Decision gist · record as of 2026-08-14
No. The package is abandoned with last release on 2017-02-24 and high install friction. Dependency compatibility with modern Scrapy and Crochet versions is highly uncertain. If you need synchronous Scrapy integration, consider using Scrapy's modern async/await patterns directly or evaluating actively maintained alternatives.AI-flagged interpretation of the facts on this page — verify before relying
Before you install
- Requires Scrapy and Crochet as dependencies; package is abandoned and may not work with current versions of those libraries.
- High install friction and abandoned maintenance status.
- Last commit was 2017-02-24 with no updates since.
License · maintenance · safety
MIT (permissive) — MIT license is permissive and poses no restrictions on use or redistribution.
last release 2017-02-24 (3458 days) · last repo commit 2017-02-24 · 47 stars
0 known vulnerabilities (OSV.dev, 2026-08-14) · 87,727 downloads/mo, #13,774 on PyPI
Alternatives
Verify before relying
import scrapydo
scrapydo.setup()
response = scrapydo.fetch("http://example.com")
# Or crawl with a callback:
items = scrapydo.crawl("http://example.com", parse_callback)- Whether scrapydo works with modern versions of Scrapy and Crochet given the time since last release.
- Whether the high install friction is due to compiled dependencies or dependency resolution issues.
- Current compatibility with modern Python versions (requires_python is unspecified in metadata).
What it is and what it does
ScrapyDo wraps Scrapy's asynchronous reactor with Crochet to expose a blocking, function-based API for web scraping. Instead of managing Scrapy's event loop directly, you call simple functions like `fetch()` and `crawl()` that handle reactor initialization and teardown internally. This is useful when you want to use Scrapy from a synchronous context—such as a Jupyter notebook or a regular Python script—without having to understand or manage Twisted's reactor lifecycle.
The package provides three main operations: fetching a single URL to get a response object, crawling a URL with a callback function to extract and yield items, and running an existing spider class with custom arguments. It also includes a `highlight()` utility for syntax-highlighting code in IPython notebooks. However, the package has been abandoned since its latest release on 2017-02-24, so compatibility with current versions of Scrapy, Crochet, and Python is uncertain.
Use it for
- Running Scrapy spiders from Jupyter notebooks without managing the Twisted reactor directly.
- Fetching and parsing a single URL synchronously in a regular Python script without async/await boilerplate.
- Crawling a site with a callback function to extract structured data in a blocking, sequential manner.
- Integrating Scrapy into a synchronous application or API endpoint that needs to scrape web content on demand.
Worth the install?
AI-flagged interpretation of the facts on this page. Verify before relying on it.
No.
The package is abandoned with last release on 2017-02-24 and high install friction. Dependency compatibility with modern Scrapy and Crochet versions is highly uncertain. If you need synchronous Scrapy integration, consider using Scrapy's modern async/await patterns directly or evaluating actively maintained alternatives.
Install
scrapydo on PyPI
Before you install
High install friction and abandoned maintenance status. Last commit was 2017-02-24 with no updates since. Depends on Scrapy and Crochet, which may have evolved significantly since the package was last maintained.
Requires Scrapy and Crochet as dependencies; package is abandoned and may not work with current versions of those libraries.
License in practice
MIT license is permissive and poses no restrictions on use or redistribution.
Quickstart
import scrapydo
scrapydo.setup()
response = scrapydo.fetch("http://example.com")
# Or crawl with a callback:
items = scrapydo.crawl("http://example.com", parse_callback)Verify before relying
- Whether scrapydo works with modern versions of Scrapy and Crochet given the time since last release.
- Whether the high install friction is due to compiled dependencies or dependency resolution issues.
- Current compatibility with modern Python versions (requires_python is unspecified in metadata).
Package facts
| License | MIT permissive |
| Python support | Not specified |
| Install friction | High. Source build required |
| Runtime dependencies | None |
| Maintenance | Abandoned 3,458 days since the last release |
| Last repo commit | |
| First released | |
| Downloads | 87,727 / month, #13,774 on PyPI 30-day window, as of 2026-08-14 |
| Known vulnerabilities | None known OSV.dev, checked 2026-08-14 |
| Classifiers | Development Status :: 4 - BetaIntended Audience :: DevelopersProgramming Language :: Python |
Evidence: scrapydo-0.2.2.tar.gz
Tags
Let your AI agent find packages like this
Example. Real query, live index.
You found this page by searching. An agent finds it by wishing: SkillFed indexes 14,416 PyPI packages by what they can do, searchable in plain language.
wish › “scrapy blocking api”
- scrapydoProvides a blocking API to run Scrapy spiders synchronously, wrapping…
- scrapy-zyte-apiScrapy plugin that integrates Zyte API for web scraping, allowing you…
- scrapfly-sdkPython SDK for the Scrapfly web scraping service, providing access to…
Give your agent the search over MCP, or paste the wish link into any chat.
More WWW/HTTP packages
urllib3 is an HTTP client library that provides thread-safe connection pooling, SSL/TLS verification, multipart file uploads, request retries, compression support, and proxy handling for Python applications.
Requests is a Python HTTP library that simplifies sending HTTP/1.1 requests with automatic handling of headers, authentication, cookies, and response parsing.
h11 is a pure-Python HTTP/1.1 protocol implementation that handles parsing and serializing HTTP messages without any built-in I/O, letting you integrate it with any network layer you choose.
HTTPX is a fully featured HTTP client library for Python that provides both sync and async APIs, with support for HTTP/1.1 and HTTP/2, plus an integrated command-line client.
Install it if you are building new projects or modernizing existing ones that rely on HTTP.
A minimal low-level HTTP client library that sends HTTP requests with thread-safe and task-safe connection pooling, supporting HTTP/1.1, HTTP/2, proxies, and both sync and async interfaces.
aiohttp is an async HTTP client and server framework built on asyncio, supporting both WebSockets and middleware-based routing for building concurrent web applications.
Install it if you need async HTTP client or server capabilities in asyncio-based applications.
See also crochet · scrapling · Scrapy · scrapy-playwright · spider-client · pytest-twisted · scrapy-zyte-api · syncer · scrapfly-sdk · threadloop