{"categories":[{"label":"WWW/HTTP","url":"https://skillfed.io/packages/category/internet-www-http"},{"label":"Security","url":"https://skillfed.io/packages/category/security"},{"label":"Information Analysis","url":"https://skillfed.io/packages/category/scientific-engineering-information-analysis"},{"label":"Linguistic","url":"https://skillfed.io/packages/category/text-processing-linguistic"},{"label":"Filters","url":"https://skillfed.io/packages/category/text-processing-filters"}],"enrichment":{"capability":"Validates, normalizes, filters, and samples URLs for web crawling and document collection, removing spam, trackers, and low-value pages while respecting language and content-type constraints.","skillfed_tags":["web-crawling","url-processing","data-quality"],"use_cases":["Filter out tracker parameters and spam domains before crawling to reduce bandwidth waste and improve document quality.","Sample representative URLs from a large domain to avoid redundant crawling and focus on diverse content.","Extract and prioritize links from HTML with language filtering to target content in specific locales.","Normalize and deduplicate URLs across multiple sources to prevent re-crawling the same page.","Identify navigation and overview pages separately from content pages to optimize crawl scheduling."],"what_it_does":"Courlan is a URL processing library designed to improve web crawler efficiency and document collection quality. It provides validation, normalization, filtering, and sampling functions that work together to identify high-value pages and discard low-quality content before crawling begins. The library filters out spam, tracking parameters, and platform-specific pages, and can apply language-aware heuristics to target specific locales.\n\nThe package is built on three runtime dependencies (babel, tld, urllib3) and offers both programmatic Python functions and command-line utilities. It includes specialized tools for crawl frontier management\u2014distinguishing navigation pages from content pages, detecting non-crawlable URLs, and handling relative link resolution\u2014making it useful for anyone building a web crawler, scraper, or bulk document collection pipeline.","worth_installing":"Yes. Courlan is actively maintained, has no known vulnerabilities, and offers a focused, well-scoped toolkit for URL handling in crawling workflows. The permissive Apache-2.0 license and low install friction make it a practical choice for web scraping and document collection projects. Install it if you are building a crawler or bulk collection pipeline and need to filter, normalize, or sample URLs at scale."},"id":"courlan","links":{"html":"https://skillfed.io/packages/courlan","md":"https://skillfed.io/packages/courlan.md","pypi":"https://pypi.org/project/courlan/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2026-06-01","license_spdx":"Apache-2.0","license_treatment":"permissive","name":"courlan","python_support":"supports_current","summary":"Clean, filter and sample URLs to optimize data collection \u2013 includes spam, content type and language filters."},"popularity":{"monthly_downloads":13207347,"position":1292,"tier":"top_5000"},"security":{"n_vulnerabilities":0},"version":"1.4.0"}