scrapling
scrapling enables automated data extraction from sites defended by Cloudflare and similar anti-scraping systems. It uses sophisticated evasion methods to bypass protection layers and retrieve content that would otherwise be inaccessible to standard scrapers. Ideal for researchers and developers who need reliable access to protected web data.
scrapling employs sophisticated evasion methods to bypass Cloudflare and similar anti-scraping defenses, enabling automated data extraction from protected sites. The tool uses stealth browsing techniques and anti-detection capabilities to retrieve content that standard scrapers cannot access, making it suitable for researchers and developers who require reliable access to protected web data.
AI-generated summary based on this skill's SKILL.md
Install
NousResearch/hermes-agent/scrapling · repository language: Python
git clone https://github.com/NousResearch/hermes-agent
cp -r hermes-agent/optional-skills/research/scrapling ~/.claude/skills/scraplingFrequently asked questions
AI-generated answers based on this skill's SKILL.md and metadata
How does scrapling scrape websites with Cloudflare protection?
scrapling employs sophisticated evasion methods to bypass Cloudflare and similar anti-scraping defenses, enabling automated data extraction from protected sites. The tool uses stealth browsing techniques and anti-detection capabilities to retrieve content that standard scrapers cannot access, making it suitable for researchers and developers who require reliable access to protected web data.
Can scrapling handle JavaScript-rendered and dynamic content?
Yes, scrapling is designed to scrape JavaScript-rendered pages and dynamic content from single pages. It includes browser automation capabilities that allow it to wait for elements to load, handle asynchronous rendering, and extract data after the page has fully rendered, rather than just parsing static HTML.
What data extraction methods does scrapling support?
scrapling supports multiple structured data extraction approaches, including CSS selectors and XPath queries. These methods allow you to precisely target and extract specific elements from HTML documents, giving you fine-grained control over which data points you capture from scraped pages.
Does scrapling support multi-page web crawling with automated link following?
Yes, scrapling includes crawling capabilities that enable you to traverse multiple pages across a site with automated link following. This allows you to systematically collect data across entire domains or sections of websites rather than being limited to individual pages.
What browser automation features does scrapling provide?
scrapling offers comprehensive browser automation capabilities including scrolling, clicking, and waiting for elements to appear. These features enable you to automate complex user interactions and handle sites that require dynamic engagement beyond simple page requests.
Is scrapling available as an open-source tool?
Yes, scrapling is released under the MIT license, making it open-source and freely available for use, modification, and distribution. This permissive license allows developers to integrate scrapling into their projects and customize it for their specific data extraction needs.
SKILL.md
rendered from the published skill — quoted content, verbatim
Scrapling
Scrapling is a web scraping framework with anti-bot bypass, stealth browser automation, and a spider framework. It provides three fetching strategies (HTTP, dynamic JS, stealth/Cloudflare) and a full CLI.
This skill is for educational and research purposes only. Users must comply with local/international data scraping laws and respect website Terms of Service.
When to Use
- Scraping static HTML pages (faster than browser tools)
- Scraping JS-rendered pages that need a real browser
- Bypassing Cloudflare Turnstile or bot detection
- Crawling multiple pages with a spider
- When the built-in
web_extracttool does not return the data you need
Installation
```bash pip install "scrapling[all]" scrapling
(truncated - see the full file via the links below)
Read as markdown · JSON record · Browse the source repository
File tree — 1 file
optional-skills/research/scrapling/SKILL.md