icrawler

What it is and what it does

icrawler is a Python framework for writing multi-threaded web crawlers that focus on downloading media—images, videos, and other files—from websites. It abstracts away the complexity of thread management, exception handling, and queue coordination, letting you concentrate on what to crawl. The framework uses a three-stage pipeline: a Feeder puts page URLs into a queue, a Parser extracts media URLs from those pages, and a Downloader fetches and saves the files. Each stage runs in its own thread pool, so you can tune concurrency independently.

The package ships with ready-to-use crawlers for Google Images, Bing Images, Baidu Images, and Flickr, so you can start downloading images with a few lines of code. You can also extend the framework to build crawlers for custom websites by implementing your own Feeder, Parser, and Downloader classes. It depends on beautifulsoup4, lxml, pillow, requests, and pyyaml—standard tools for HTTP requests, HTML parsing, and image handling.

Use it for:

• Download batches of images from Google Images, Bing, or Baidu for dataset creation or research.
• Build a custom crawler to scrape product images or media from a specific website.
• Automate media collection from Flickr or other image-hosting services with configurable thread pools.
• Extract and download images from search results with filters (size, color, date, license).
• Prototype a web scraper without the overhead of learning Scrapy's full architecture.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

icrawler is a lightweight, modular web crawler framework that downloads images, videos, and other media from websites using built-in crawlers for popular image sites and search engines, or custom crawlers you write yourself.

Yes, if you need a lightweight image crawler and can tolerate aging maintenance. The package is stable for straightforward use cases (downloading from Google Images, Bing, or Flickr), has no known vulnerabilities, and installs cleanly. However, be aware that the last release was 549 days ago—built-in crawlers may break if target websites change their HTML or API structure, and you may need to fork or patch the code yourself. Best suited for one-off scripts or prototypes rather than production systems that require active support.

Install

icrawler on PyPI

pip install icrawler

uv add icrawler

poetry add icrawler

Installing icrawler

Before you install

Low install friction with a pure-Python wheel distribution. Maintenance is aging—last release was 549 days ago—but the repository remains active and the package supports current Python versions (3.7–3.12).

License in practice

MIT license permits unrestricted use, modification, and distribution. No restrictions on commercial or proprietary use.

Quickstart

pip install icrawler

from icrawler.builtin import GoogleImageCrawler

google_crawler = GoogleImageCrawler(storage={'root_dir': 'images'})
google_crawler.crawl(keyword='cat', max_num=100)

Requires Python 3.7 or later. Built-in crawlers depend on external site APIs and HTML structures that may change without notice.

Verify before relying

• Whether built-in crawlers for Google, Bing, Baidu, and Flickr still work reliably given the 549-day gap since last release.
• Whether the package handles modern anti-scraping measures (rate limiting, user-agent blocking, JavaScript rendering).
• Performance and stability when running with high thread counts on large crawl jobs.

Package facts

Evidence: icrawler-0.6.10-py3-none-any.whl

Keywords: Crawler

Tags

More Python Modules packages