---
id: scrapydo
version: "0.2.2"
license: MIT
license_treatment: permissive
maintenance: abandoned
---

# scrapydo — Crochet-based blocking API for Scrapy.

License: permissive · Maintenance: abandoned · Downloads: 87.7K/mo

## What it is and what it does

ScrapyDo wraps Scrapy's asynchronous reactor with Crochet to expose a blocking, function-based API for web scraping. Instead of managing Scrapy's event loop directly, you call simple functions like `fetch()` and `crawl()` that handle reactor initialization and teardown internally. This is useful when you want to use Scrapy from a synchronous context—such as a Jupyter notebook or a regular Python script—without having to understand or manage Twisted's reactor lifecycle.

The package provides three main operations: fetching a single URL to get a response object, crawling a URL with a callback function to extract and yield items, and running an existing spider class with custom arguments. It also includes a `highlight()` utility for syntax-highlighting code in IPython notebooks. However, the package has been abandoned since its latest release on 2017-02-24, so compatibility with current versions of Scrapy, Crochet, and Python is uncertain.

Use it for:

- Running Scrapy spiders from Jupyter notebooks without managing the Twisted reactor directly.
- Fetching and parsing a single URL synchronously in a regular Python script without async/await boilerplate.
- Crawling a site with a callback function to extract structured data in a blocking, sequential manner.
- Integrating Scrapy into a synchronous application or API endpoint that needs to scrape web content on demand.

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

Provides a blocking API to run Scrapy spiders synchronously, wrapping Scrapy's asynchronous reactor with Crochet to allow straightforward function calls for fetching URLs and crawling sites.

No. The package is abandoned with last release on 2017-02-24 and high install friction. Dependency compatibility with modern Scrapy and Crochet versions is highly uncertain. If you need synchronous Scrapy integration, consider using Scrapy's modern async/await patterns directly or evaluating actively maintained alternatives.

## Install

pip install scrapydo
uv add scrapydo
poetry add scrapydo

## Installing scrapydo

Before you install: High install friction and abandoned maintenance status. Last commit was 2017-02-24 with no updates since. Depends on Scrapy and Crochet, which may have evolved significantly since the package was last maintained.

License in practice: MIT license is permissive and poses no restrictions on use or redistribution.

Quickstart:

import scrapydo
scrapydo.setup()
response = scrapydo.fetch("http://example.com")
# Or crawl with a callback:
items = scrapydo.crawl("http://example.com", parse_callback)

Requires Scrapy and Crochet as dependencies; package is abandoned and may not work with current versions of those libraries.

Verify before relying:

- Whether scrapydo works with modern versions of Scrapy and Crochet given the time since last release.
- Whether the high install friction is due to compiled dependencies or dependency resolution issues.
- Current compatibility with modern Python versions (requires_python is unspecified in metadata).

## Package facts

- License: MIT (permissive)
- Python support: unspecified
- Install friction: high
- Maintenance: abandoned
- Downloads: 87.7K/month (top 15,000 on PyPI)
- Known vulnerabilities: none known

## Tags

scrapy blocking api, synchronous scrapy wrapper, scrapy synchronous crawling, crochet scrapy integration, scrapy fetch and crawl, blocking scrapy spider runner, web-scraping, abandoned

[View on SkillFed](https://skillfed.io/packages/scrapydo) · [View on PyPI](https://pypi.org/project/scrapydo/)