web-scraper
web-scraper enables you to harvest structured information from websites by defining selectors and parsing patterns. Perfect for automating data collection tasks, this skill transforms raw HTML into organized, usable datasets without manual intervention.
web-scraper enables you to harvest structured information from websites by defining selectors and parsing patterns. You specify CSS selectors or XPath expressions to target the content you want, and web-scraper extracts and organizes that data into structured formats. This approach transforms raw HTML into usable datasets without requiring manual intervention, making it ideal for automating repetitive data collection tasks at scale.
AI-generated summary based on this skill's SKILL.md
Install
AIDotNet/MoYuCode/web-scraper · repository language: TypeScript
git clone https://github.com/AIDotNet/MoYuCode
cp -r MoYuCode/skills/community/web-scraper ~/.claude/skills/web-scraperFrequently asked questions
AI-generated answers based on this skill's SKILL.md and metadata
How do I scrape a website using web-scraper?
web-scraper enables you to harvest structured information from websites by defining selectors and parsing patterns. You specify CSS selectors or XPath expressions to target the content you want, and web-scraper extracts and organizes that data into structured formats. This approach transforms raw HTML into usable datasets without requiring manual intervention, making it ideal for automating repetitive data collection tasks at scale.
Can web-scraper parse website data with CSS selectors?
Yes, web-scraper supports parsing website data using CSS selectors as one of its core extraction methods. By defining CSS selector rules, web-scraper identifies and retrieves specific content from HTML pages. This selector-based approach is one of the primary ways web-scraper enables you to extract structured data from web pages, allowing you to target elements with precision and collect exactly the information you need.
What license does web-scraper use?
web-scraper is released under the MIT license, which is a permissive open-source license. This means you can freely use, modify, and distribute web-scraper in both personal and commercial projects, provided you include a copy of the license and copyright notice. The MIT license makes web-scraper accessible for a wide range of web data extraction and automation scenarios.
How can web-scraper help me scrape product prices from ecommerce sites?
web-scraper is well-suited for scraping product information, prices, and metadata from e-commerce sites. You can define selectors to target product names, prices, descriptions, and other relevant fields within the HTML structure. web-scraper then automates the extraction of this data across multiple product pages, allowing you to build comprehensive price monitoring systems or competitive analysis datasets without manual data entry.
Does web-scraper support rate limiting and error handling?
web-scraper includes support for implementing rate limiting and error handling to ensure robust web scraping operations. These features help you avoid overwhelming target servers, respect rate limits, and gracefully handle network failures or parsing errors. By incorporating rate limiting and error handling with web-scraper, you can build reliable, production-grade data collection systems that respect website policies and maintain data integrity.
What are the main use cases for web-scraper?
web-scraper excels at extracting structured data from web pages using selectors and parsing, automating repetitive web data collection tasks at scale, parsing HTML to retrieve specific content from websites, and scraping product information from e-commerce platforms. Whether you need to monitor prices, collect articles, harvest metadata, or build datasets from multiple websites, web-scraper transforms raw HTML into organized, actionable information through its selector-based extraction and automation capabilities.
SKILL.md
rendered from the published skill — quoted content, verbatim
Web Scraper Skill
Description
Extract and process data from web pages with intelligent parsing capabilities.
Trigger
/scrapecommand- User requests web data extraction
- User needs to parse HTML
Prompt
You are a web scraping expert that extracts data efficiently and ethically.
Puppeteer Scraper (TypeScript)
```typescript import puppeteer from 'puppeteer';
interface Product { name: string; price: number; rating: number; url: string; }
async function scrapeProducts(url: string): Promise<Product[]> { const browser = await puppeteer.launch({ headless: 'new' }); const page = await browser.newPage();
// Set user agent to avoid detection await page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36');
await page.goto(url, { waitUntil: 'networkidle2' });
// Wait for products to load await page.waitForSelector('.product-card');
const products = await page.evaluate(() => { const items = document.querySelectorAll('.product-card'); return Array.from(items).map(item => ({
(truncated - see the full file via the links below)
Read as markdown · JSON record · Browse the source repository
File tree — 1 file
skills/community/web-scraper/SKILL.md