newspaper4k

What it is and what it does

Newspaper4k is a continuation of the unmaintained newspaper3k library, designed to extract structured article data from web pages and news websites. It downloads and parses HTML to recover article text, authors, publish dates, top images, and embedded media, then applies NLP techniques to extract keywords and generate summaries. The library supports multiple languages through automatic meta-tag detection or explicit configuration, and can build a Source object to discover and download all articles from a news website's front page and RSS feeds using multithreaded downloads.

The package maintains backward compatibility with newspaper3k's API, so existing code continues to work. It includes a command-line interface for one-off article extraction and special integration with Google News via an optional extra. Core dependencies are standard web-scraping and parsing libraries: beautifulsoup4, lxml, requests, pillow for image handling, feedparser for RSS, and python-dateutil for timestamp parsing.

Use it for:

• Extract article text and metadata from a single URL for content aggregation or archival
• Build a news source crawler to discover and download all articles from a website's homepage and category pages
• Automatically summarize and extract keywords from articles for content curation dashboards
• Parse multilingual news sites by auto-detecting language from meta tags or specifying it explicitly
• Integrate Google News as a source to fetch recent articles by country and topic

Worth the install?

AI-flagged interpretation of the facts on this page — verify before relying

Extracts article text, metadata, and images from web pages and news sources, with support for multiple languages and automatic summarization.

Yes. The package is actively maintained, has low install friction, carries no known vulnerabilities, and uses a permissive MIT license. It is well-suited for article scraping and curation tasks where you need structured extraction from HTML. The main gotcha is that bulk scraping can be slow and may trigger IP blocks on news sites; start with small batches and respect robots.txt.

Install

newspaper4k on PyPI

pip install newspaper4k

uv add newspaper4k

poetry add newspaper4k

Installing newspaper4k

Before you install

Low friction installation with 11 runtime dependencies. Actively maintained with recent release 26 days ago; requires Python 3.10 or later.

License in practice

MIT license permits commercial and private use with minimal restrictions; suitable for most projects.

Quickstart

pip install newspaper4k

from newspaper4k import article

obj = article('https://example.com/article')
print(obj.title)
print(obj.authors)
print(obj.text)

Requires Python 3.10 or later. Article extraction quality depends on correct language detection; if the wrong language is specified, text extraction may fail.

Verify before relying

• Whether multithreaded download performance scales well with large numbers of articles
• Robustness of language auto-detection across non-Latin scripts
• Rate-limiting or IP-blocking mitigation strategies when bulk-scraping news sites

Package facts

Evidence: newspaper4k-0.9.6-py3-none-any.whl

Keywords: article, curation, extraction, newspaper, nlp, scraping

Tags

More Python Modules packages