{"categories":[{"label":"HTML","url":"https://skillfed.io/packages/category/text-processing-markup-html"}],"enrichment":{"capability":"Extracts main content from HTML pages and outputs it in HTML, text, or Markdown format, preserving element hierarchy and link information for use with LLM frameworks.","skillfed_tags":["web-scraping","llm-data-prep","content-extraction"],"use_cases":["Prepare web articles for ingestion into LangChain or LlamaIndex by extracting clean content in Markdown or text format.","Extract main content from HTML while preserving link and header hierarchy for downstream analysis or indexing.","Convert web pages to Markdown for easier consumption by LLM prompts or knowledge bases.","Remove boilerplate, navigation, and ads from web pages before processing with NLP tools.","Build web scraping pipelines that output multiple formats from a single extraction pass."],"what_it_does":"MainContentExtractor wraps trafilatura to extract the main article or content body from HTML pages while preserving structural information like links and headers. Unlike trafilatura alone, it can output in HTML format (via an XML intermediate step), plain text, or Markdown\u2014formats more suitable for feeding into LLM frameworks. It depends on trafilatura for core extraction, html2text for text conversion, and beautifulsoup4 for HTML parsing.\n\nThe library addresses a specific gap: trafilatura is effective at identifying main content but cannot output HTML directly and sometimes loses necessary data. MainContentExtractor compensates by converting trafilatura's XML output to HTML, though this conversion is not perfectly reversible. It's designed for workflows where you need both structured content and multiple output formats, particularly when preparing web data for language models.","worth_installing":"Yes, if you need HTML-format output from main content extraction for LLM workflows. The library fills a real gap and has low install friction. However, proceed with caution: the package has been dormant since 2023-12-10 with no updates, maintenance status is unclear, and the XML-to-HTML conversion is lossy. Evaluate whether trafilatura alone or an alternative meets your needs before committing to a package with uncertain long-term support."},"id":"maincontentextractor","links":{"html":"https://skillfed.io/packages/maincontentextractor","md":"https://skillfed.io/packages/maincontentextractor.md","pypi":"https://pypi.org/project/maincontentextractor/"},"maintenance":{"status":"dormant"},"meta":{"latest_release":"2023-12-10","license_spdx":null,"license_treatment":"permissive","name":"MainContentExtractor","python_support":"supports_current","summary":"A library to extract the main content from html. Developed for information on LLM and for feeding data into LangChain and LlamaIndex."},"popularity":{"monthly_downloads":79301,"position":14367,"tier":"top_15000"},"security":{"n_vulnerabilities":0},"version":"0.0.4"}