ultimate-sitemap-parser

What it is and what it does

Ultimate Sitemap Parser is a Python library that discovers and parses sitemaps in all common formats—XML, RSS, Atom, plain text, and Google News/Image variants—and extracts URLs into an object tree. It handles malformed sitemaps gracefully, discovers sitemaps linked from robots.txt, and uses memory-efficient Expat XML parsing to avoid loading entire hierarchies into memory at once.

The library is designed for web crawlers and indexing workflows. You give it a homepage URL, and it returns a tree of sitemap objects you can iterate over to get all discovered pages. It has been field-tested with approximately 1 million URLs as part of the Media Cloud project and depends only on python-dateutil and requests, making installation straightforward on any modern Python environment.

Use it for:

• Discover all pages on a website by parsing its sitemap hierarchy for web crawling or SEO audits.
• Extract URLs from Google News or Image sitemaps for specialized content indexing.
• Build a site map inventory by recursively following nested sitemap references.
• Integrate sitemap discovery into a web scraper to respect site structure and robots.txt directives.
• Analyze sitemap coverage to identify missing or orphaned pages in a website.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Parses and crawls sitemaps in multiple formats (XML, RSS, Atom, plain text, Google News/Image) and extracts URLs efficiently without loading entire trees into memory.

Yes. The library is actively maintained, has low install friction, no known vulnerabilities, and solves a specific problem well—parsing diverse sitemap formats reliably. The copyleft GPL-3.0-or-later license is the main constraint: use it freely in open-source projects, but review licensing implications before bundling into proprietary software.

Install

ultimate-sitemap-parser on PyPI

pip install ultimate-sitemap-parser

uv add ultimate-sitemap-parser

poetry add ultimate-sitemap-parser

Installing ultimate-sitemap-parser

Before you install

Low friction: pure Python wheel with only two stable runtime dependencies (python-dateutil and requests). Actively maintained as of 2026-06-16 with 256 repository stars.

License in practice

GPL-3.0-or-later (copyleft): you must license any derivative work or bundled application under compatible terms; suitable for open-source projects but requires legal review before use in proprietary software.

Quickstart

pip install ultimate-sitemap-parser

from ultimate_sitemap_parser.tree import sitemap_tree_for_homepage

tree = sitemap_tree_for_homepage('https://www.example.org/')
for page in tree.all_pages():
    print(page.url)

Requires Python 3.10 or later; network access to fetch sitemaps from target URLs.

Verify before relying

• Whether the library handles redirects or authentication when fetching sitemaps from protected URLs.
• Memory consumption profile on very large sitemap hierarchies beyond the ~1 million URLs mentioned in testing.
• Whether custom web client support extends to proxy configuration or certificate handling.

Package facts

Evidence: ultimate_sitemap_parser-1.8.1-py3-none-any.whl

Keywords: sitemap, crawler, indexing, xml, rss, atom, google news

Tags

More XML packages