web-scraping
Web-scraping lets you pull structured data from any URL using plain English descriptions of what you need. Choose from AI-powered extraction with natural language prompts, high-volume batch processing, or schema-based data structuring—then combine methods for comprehensive site crawling and automation.
Web-scraping extracts structured data from websites using natural language prompts and AI-powered extraction methods.
AI-generated summary based on this skill's SKILL.md
Install
gooseworks-ai/goose-skills/web-scraping · repository language: Python
git clone https://github.com/gooseworks-ai/goose-skills
cp -r goose-skills/skills/research-tools/capabilities/web-scraping ~/.claude/skills/web-scrapingFrequently asked questions
AI-generated answers based on this skill's SKILL.md and metadata
What can web-scraping extract from websites?
Web-scraping lets you pull structured data from any URL using plain English descriptions of what you need. You can extract information from web pages including product details, pricing, brand assets like logos and colors, design system information, and any other structured content. The tool supports AI-powered extraction with natural language prompts, making it easy to specify exactly what data you want without writing code.
How does web-scraping handle multiple pages or entire sites?
Web-scraping supports scrape and crawl multiple pages or entire sites automatically. You can process bulk URLs in batch with consistent output schemas, allowing you to gather data from many pages at once. The tool combines crawling capabilities with automated data extraction to handle comprehensive site scraping efficiently.
Can web-scraping automate browser interactions for complex tasks?
Yes, web-scraping automates browser interactions for complex scraping tasks. This capability enables you to handle dynamic content, JavaScript-rendered pages, and multi-step workflows that require simulating user actions. Combined with the tool's other features, browser automation lets you tackle sophisticated data extraction scenarios beyond simple HTML parsing.
How do I extract structured data from websites using natural language?
Web-scraping's primary strength is extracting structured data from websites using natural language prompts. Instead of writing selectors or code, you describe in plain English what data you need—like 'get product names and prices' or 'extract all contact information'—and the AI-powered tool handles the extraction and formatting automatically.
What output formats does web-scraping support for batch processing?
Web-scraping processes bulk URLs in batch with consistent output schemas. This means you can scrape multiple URLs and receive results in a standardized format, making it easy to integrate the extracted data into your workflows, databases, or downstream applications without manual reformatting.
Is web-scraping open source and what license does it use?
Web-scraping is released under the MIT license, making it open source and freely available for both commercial and personal use. The MIT license allows you to use, modify, and distribute the tool with minimal restrictions.
SKILL.md
rendered from the published skill — quoted content, verbatim
Scrape — General-Purpose Web Scraping & Data Extraction
Setup
Read your credentials from ~/.gooseworks/credentials.json:
export GOOSEWORKS_API_KEY=$(python3 -c "import json;print(json.load(open('$HOME/.gooseworks/credentials.json'))['api_key'])")
export GOOSEWORKS_API_BASE=$(python3 -c "import json;print(json.load(open('$HOME/.gooseworks/credentials.json')).get('api_base','https://api.gooseworks.ai'))")
If ~/.gooseworks/credentials.json does not exist, tell the user to run: npx gooseworks login
All endpoints use Bearer auth: -H "Authorization: Bearer $GOOSEWORKS_API_KEY"
Scrape websites, extract structured data, and automate browser interactions. Pick the best API for the task — or combine several for comprehensive extraction.
1. Scrapegraph — AI-Powered Scraping with Natural Language
Best for: Extracting data using plain English prompts, converting pages to markdown, crawling with AI extraction, and search-based scraping.
**AI-powered
(truncated - see the full file via the links below)
Read as markdown · JSON record · Browse the source repository
File tree — 2 files
skills/research-tools/capabilities/web-scraping/SKILL.md
skills/research-tools/capabilities/web-scraping/skill.meta.json