django-chunkator

What it is and what it does

django-chunkator provides a simple iterator that breaks a Django QuerySet into smaller batches, fetching and processing them one chunk at a time instead of loading the entire result set into memory. This is useful when you need to process large numbers of records but don't have enough RAM to hold them all at once. The package exports two main functions: chunkator() for iterating over individual items in chunks, and chunkator_page() for working with entire pages of results as lists.

The tool works by repeatedly querying the database with offset and limit, fetching each batch sequentially. It requires that your model has a unique primary key (which Django provides by default) and works with both standard integer PKs and UUIDs. The package has been tested with PostgreSQL and SQLite. Note that chunking does not speed up the overall process—it trades a single large query for multiple smaller ones, with the main benefit being reduced memory consumption during iteration.

Use it for:

• Bulk-update large numbers of records in a Django model without exhausting available RAM.
• Export large datasets to CSV or other formats by processing rows in manageable batches.
• Perform background tasks on all records in a table by chunking the queryset and dispatching each chunk to a task queue.
• Migrate or transform data from one table to another when the result set is too large to fit in memory.
• Generate reports or analytics by iterating over a large queryset in controlled increments.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Iterates over large Django QuerySets in configurable chunks to avoid loading entire result sets into memory at once.

No. The package is abandoned (last release 2020-06-22, last commit 2022-07-11) and untested against Django versions beyond 3.0. Compatibility with modern Django and Python versions is unverified. For new projects, consider using Django's built-in QuerySet.iterator() with chunk_size parameter, which provides similar functionality and is actively maintained.

Install

django-chunkator on PyPI

pip install django-chunkator

uv add django-chunkator

poetry add django-chunkator

Installing django-chunkator

Before you install

Low install friction with a single Django dependency. However, the package is abandoned—last release was 2020-06-22, with no commits since 2022-07-11.

License in practice

MIT license is permissive and imposes no restrictions on use, modification, or distribution in proprietary or open-source projects.

Quickstart

pip install django-chunkator

from chunkator import chunkator
for item in chunkator(LargeModel.objects.all(), 200):
    do_something(item)

Your Django model must define a unique pk field (the default), and the queryset must include it if using values().

Verify before relying

• Compatibility with Django versions released after 3.0 and Python versions after 3.8.
• Performance characteristics and memory savings on very large datasets.
• Whether the package functions correctly with modern dependency versions.

Package facts

Evidence: django_chunkator-2.0.0-py3-none-any.whl

Tags

More WWW/HTTP packages