{"categories":[{"label":"WWW/HTTP","url":"https://skillfed.io/packages/category/internet-www-http"},{"label":"Utilities","url":"https://skillfed.io/packages/category/utilities/2"},{"label":"Security","url":"https://skillfed.io/packages/category/security"},{"label":"Information Analysis","url":"https://skillfed.io/packages/category/scientific-engineering-information-analysis"},{"label":"Linguistic","url":"https://skillfed.io/packages/category/text-processing-linguistic"},{"label":"HTML","url":"https://skillfed.io/packages/category/text-processing-markup-html"},{"label":"XML","url":"https://skillfed.io/packages/category/text-processing-markup-xml"},{"label":"Markdown","url":"https://skillfed.io/packages/category/text-processing-markup-markdown"},{"label":"Text Processing","url":"https://skillfed.io/packages/category/text-editors-text-processing"}],"enrichment":{"capability":"Trafilatura extracts main text, metadata, and structured content from web pages and HTML, converting raw HTML into clean, usable data in multiple output formats.","skillfed_tags":["web-scraping","text-extraction","nlp-data-prep"],"use_cases":["Build a text corpus from news websites or blogs by crawling sitemaps and extracting article text and publication dates.","Convert downloaded HTML files to clean JSON or CSV for data analysis, removing boilerplate and preserving only article content.","Extract metadata (title, author, date, categories) from web pages for indexing or cataloging.","Process large batches of HTML documents in parallel to prepare training data for NLP or machine learning models.","Scrape and structure web content for research purposes while respecting politeness and deduplication rules."],"what_it_does":"Trafilatura is a Python library and command-line tool for discovering, downloading, and extracting text and metadata from web pages. It handles the full pipeline: crawling via sitemaps and feeds, downloading HTML, parsing and cleaning content, and exporting results in formats like JSON, CSV, Markdown, XML, and HTML. The core extraction logic balances precision (removing noise like headers, footers, navigation) against recall (preserving valid content), and it supports optional extraction of comments, links, images, and tables alongside main text and metadata.\n\nThe package is designed for both one-off extraction and bulk processing, with parallel handling of online URLs and offline HTML files. It integrates widely in academic and commercial projects (HuggingFace, IBM, Microsoft Research, Stanford, Allen Institute) and is actively maintained with regular updates. Runtime dependencies include lxml for parsing, courlan for URL management, htmldate for date extraction, and justext for text segmentation.","worth_installing":"Yes. Trafilatura is actively maintained, widely adopted, has no known vulnerabilities, and offers low install friction. It solves a real problem (converting messy HTML to clean text and metadata) with a mature, well-documented API. Use it if you need reliable web content extraction for research, data pipelines, or corpus building."},"id":"trafilatura","links":{"html":"https://skillfed.io/packages/trafilatura","md":"https://skillfed.io/packages/trafilatura.md","pypi":"https://pypi.org/project/trafilatura/"},"maintenance":{"status":"active"},"meta":{"latest_release":"2026-07-31","license_spdx":"Apache-2.0","license_treatment":"permissive","name":"trafilatura","python_support":"supports_current","summary":"Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML."},"popularity":{"monthly_downloads":13364051,"position":1284,"tier":"top_5000"},"security":{"n_vulnerabilities":0},"version":"2.2.0"}