scrapydo

What it is and what it does

ScrapyDo wraps Scrapy's asynchronous reactor with Crochet to expose a blocking, function-based API for web scraping. Instead of managing Scrapy's event loop directly, you call simple functions like `fetch()` and `crawl()` that handle reactor initialization and teardown internally. This is useful when you want to use Scrapy from a synchronous context—such as a Jupyter notebook or a regular Python script—without having to understand or manage Twisted's reactor lifecycle.

The package provides three main operations: fetching a single URL to get a response object, crawling a URL with a callback function to extract and yield items, and running an existing spider class with custom arguments. It also includes a `highlight()` utility for syntax-highlighting code in IPython notebooks. However, the package has been abandoned since its latest release on 2017-02-24, so compatibility with current versions of Scrapy, Crochet, and Python is uncertain.

Use it for:

• Running Scrapy spiders from Jupyter notebooks without managing the Twisted reactor directly.
• Fetching and parsing a single URL synchronously in a regular Python script without async/await boilerplate.
• Crawling a site with a callback function to extract structured data in a blocking, sequential manner.
• Integrating Scrapy into a synchronous application or API endpoint that needs to scrape web content on demand.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Provides a blocking API to run Scrapy spiders synchronously, wrapping Scrapy's asynchronous reactor with Crochet to allow straightforward function calls for fetching URLs and crawling sites.

No. The package is abandoned with last release on 2017-02-24 and high install friction. Dependency compatibility with modern Scrapy and Crochet versions is highly uncertain. If you need synchronous Scrapy integration, consider using Scrapy's modern async/await patterns directly or evaluating actively maintained alternatives.

Install

scrapydo on PyPI

pip install scrapydo

uv add scrapydo

poetry add scrapydo

Installing scrapydo

Before you install

High install friction and abandoned maintenance status. Last commit was 2017-02-24 with no updates since. Depends on Scrapy and Crochet, which may have evolved significantly since the package was last maintained.

License in practice

MIT license is permissive and poses no restrictions on use or redistribution.

Quickstart

import scrapydo
scrapydo.setup()
response = scrapydo.fetch("http://example.com")
# Or crawl with a callback:
items = scrapydo.crawl("http://example.com", parse_callback)

Requires Scrapy and Crochet as dependencies; package is abandoned and may not work with current versions of those libraries.

Verify before relying

• Whether scrapydo works with modern versions of Scrapy and Crochet given the time since last release.
• Whether the high install friction is due to compiled dependencies or dependency resolution issues.
• Current compatibility with modern Python versions (requires_python is unspecified in metadata).

Package facts

Evidence: scrapydo-0.2.2.tar.gz

Tags

More WWW/HTTP packages