---
id: courlan
version: "1.4.0"
license: Apache-2.0
license_treatment: permissive
maintenance: active
---

# courlan — Clean, filter and sample URLs to optimize data collection – includes spam, content type and language filters.

License: permissive · Maintenance: active · Downloads: 13.2M/mo

## What it is and what it does

Courlan is a URL processing library designed to improve web crawler efficiency and document collection quality. It provides validation, normalization, filtering, and sampling functions that work together to identify high-value pages and discard low-quality content before crawling begins. The library filters out spam, tracking parameters, and platform-specific pages, and can apply language-aware heuristics to target specific locales.

The package is built on three runtime dependencies (babel, tld, urllib3) and offers both programmatic Python functions and command-line utilities. It includes specialized tools for crawl frontier management—distinguishing navigation pages from content pages, detecting non-crawlable URLs, and handling relative link resolution—making it useful for anyone building a web crawler, scraper, or bulk document collection pipeline.

Use it for:

- Filter out tracker parameters and spam domains before crawling to reduce bandwidth waste and improve document quality.
- Sample representative URLs from a large domain to avoid redundant crawling and focus on diverse content.
- Extract and prioritize links from HTML with language filtering to target content in specific locales.
- Normalize and deduplicate URLs across multiple sources to prevent re-crawling the same page.
- Identify navigation and overview pages separately from content pages to optimize crawl scheduling.

## Worth the install?

AI-flagged interpretation of the facts above — verify before relying.

Validates, normalizes, filters, and samples URLs for web crawling and document collection, removing spam, trackers, and low-value pages while respecting language and content-type constraints.

Yes. Courlan is actively maintained, has no known vulnerabilities, and offers a focused, well-scoped toolkit for URL handling in crawling workflows. The permissive Apache-2.0 license and low install friction make it a practical choice for web scraping and document collection projects. Install it if you are building a crawler or bulk collection pipeline and need to filter, normalize, or sample URLs at scale.

## Install

pip install courlan
uv add courlan
poetry add courlan

## Installing courlan

Before you install: Low install friction with three lightweight runtime dependencies (babel, tld, urllib3). Active maintenance with a recent release 74 days ago and ongoing repository activity.

License in practice: Apache-2.0 permissive license allows commercial and private use with minimal restrictions; suitable for most projects.

Quickstart:

pip install courlan

from courlan import check_url, sample_urls, extract_links

# Validate and clean a URL
url, domain = check_url('https://example.org/page?utm_source=twitter')

# Sample URLs from a collection
sampled = sample_urls(['https://example.org/' + str(i) for i in range(100)], 10)

# Extract and filter links from HTML
links, priority_links = extract_links('<a href="/page">link</a>', 'https://example.org')

Requires Python 3.10 or higher; earlier versions (3.6–3.9) require older courlan releases.

Verify before relying:

- Performance characteristics when processing large URL collections (millions of URLs).
- Accuracy of language detection and filtering heuristics across different URL patterns.
- Whether the package handles internationalized domain names (IDN) correctly.

## Package facts

- License: Apache-2.0 (permissive)
- Python support: supports_current
- Install friction: low
- Maintenance: active
- Downloads: 13.2M/month (top 5,000 on PyPI)
- Known vulnerabilities: none known

## Tags

url validation and normalization, web crawler url filtering, spam and tracker removal, language-aware url filtering, url deduplication and sampling, crawl frontier management, url canonicalization, web-crawling, url-processing, data-quality

[View on SkillFed](https://skillfed.io/packages/courlan) · [View on PyPI](https://pypi.org/project/courlan/)