{"categories":[{"label":"WWW/HTTP","url":"https://skillfed.io/packages/category/internet-www-http/7"}],"enrichment":{"capability":"Provides a blocking API to run Scrapy spiders synchronously, wrapping Scrapy's asynchronous reactor with Crochet to allow straightforward function calls for fetching URLs and crawling sites.","skillfed_tags":["web-scraping","abandoned"],"use_cases":["Running Scrapy spiders from Jupyter notebooks without managing the Twisted reactor directly.","Fetching and parsing a single URL synchronously in a regular Python script without async/await boilerplate.","Crawling a site with a callback function to extract structured data in a blocking, sequential manner.","Integrating Scrapy into a synchronous application or API endpoint that needs to scrape web content on demand."],"what_it_does":"ScrapyDo wraps Scrapy's asynchronous reactor with Crochet to expose a blocking, function-based API for web scraping. Instead of managing Scrapy's event loop directly, you call simple functions like `fetch()` and `crawl()` that handle reactor initialization and teardown internally. This is useful when you want to use Scrapy from a synchronous context\u2014such as a Jupyter notebook or a regular Python script\u2014without having to understand or manage Twisted's reactor lifecycle.\n\nThe package provides three main operations: fetching a single URL to get a response object, crawling a URL with a callback function to extract and yield items, and running an existing spider class with custom arguments. It also includes a `highlight()` utility for syntax-highlighting code in IPython notebooks. However, the package has been abandoned since its latest release on 2017-02-24, so compatibility with current versions of Scrapy, Crochet, and Python is uncertain.","worth_installing":"No. The package is abandoned with last release on 2017-02-24 and high install friction. Dependency compatibility with modern Scrapy and Crochet versions is highly uncertain. If you need synchronous Scrapy integration, consider using Scrapy's modern async/await patterns directly or evaluating actively maintained alternatives."},"id":"scrapydo","links":{"html":"https://skillfed.io/packages/scrapydo","md":"https://skillfed.io/packages/scrapydo.md","pypi":"https://pypi.org/project/scrapydo/"},"maintenance":{"status":"abandoned"},"meta":{"latest_release":"2017-02-24","license_spdx":null,"license_treatment":"permissive","name":"scrapydo","python_support":"unspecified","summary":"Crochet-based blocking API for Scrapy."},"popularity":{"monthly_downloads":87727,"position":13774,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"0.2.2"}