MainContentExtractor

What it is and what it does

MainContentExtractor wraps trafilatura to extract the main article or content body from HTML pages while preserving structural information like links and headers. Unlike trafilatura alone, it can output in HTML format (via an XML intermediate step), plain text, or Markdown—formats more suitable for feeding into LLM frameworks. It depends on trafilatura for core extraction, html2text for text conversion, and beautifulsoup4 for HTML parsing.

The library addresses a specific gap: trafilatura is effective at identifying main content but cannot output HTML directly and sometimes loses necessary data. MainContentExtractor compensates by converting trafilatura's XML output to HTML, though this conversion is not perfectly reversible. It's designed for workflows where you need both structured content and multiple output formats, particularly when preparing web data for language models.

Use it for:

• Prepare web articles for ingestion into LangChain or LlamaIndex by extracting clean content in Markdown or text format.
• Extract main content from HTML while preserving link and header hierarchy for downstream analysis or indexing.
• Convert web pages to Markdown for easier consumption by LLM prompts or knowledge bases.
• Remove boilerplate, navigation, and ads from web pages before processing with NLP tools.
• Build web scraping pipelines that output multiple formats from a single extraction pass.

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Extracts main content from HTML pages and outputs it in HTML, text, or Markdown format, preserving element hierarchy and link information for use with LLM frameworks.

Yes, if you need HTML-format output from main content extraction for LLM workflows. The library fills a real gap and has low install friction. However, proceed with caution: the package has been dormant since 2023-12-10 with no updates, maintenance status is unclear, and the XML-to-HTML conversion is lossy. Evaluate whether trafilatura alone or an alternative meets your needs before committing to a package with uncertain long-term support.

Install

maincontentextractor on PyPI

pip install maincontentextractor

uv add maincontentextractor

poetry add maincontentextractor

Installing MainContentExtractor

Before you install

Low install friction with three straightforward dependencies. Package is dormant since its single release on 2023-12-10, with no updates despite a recent commit on 2024-05-16; maintenance status is unclear and future updates are uncertain.

License in practice

MIT license is permissive and imposes no restrictions on use, modification, or distribution in proprietary or open-source projects.

Quickstart

pip install MainContentExtractor

from main_content_extractor import MainContentExtractor
extracted_html = MainContentExtractor.extract(html_content)
extracted_markdown = MainContentExtractor.extract(html_content, output_format="markdown")

Verify before relying

• Whether the May 2024 commit represents ongoing development or a one-time fix after the December 2023 release.
• How the XML-to-HTML conversion affects data fidelity in practice and whether the irreversibility is a significant limitation.
• Whether output quality and format preservation meet requirements compared to using trafilatura directly.

Package facts

Evidence: MainContentExtractor-0.0.4-py3-none-any.whl

Tags

More HTML packages