{"enrichment":{"faq":[{"a":"web-scraping enables ethical content extraction by prioritizing compliance with robots.txt, implementing respectful rate limiting, and honoring access controls. The skill is designed for journalists, researchers, and media professionals who need dependable data collection workflows that respect site policies. Always validate that you have permission to scrape a target site, check its robots.txt file, and implement delays between requests to avoid overwhelming servers.","q":"How can I scrape websites ethically using web-scraping?"},{"a":"web-scraping employs multiple fallback strategies and robust error management to reliably extract public web content. It integrates tools like Trafilatura for clean text extraction and Playwright for browser automation, allowing you to handle complex pages that require JavaScript rendering. The skill includes proper error handling to gracefully manage network failures, timeouts, and unexpected page structures, ensuring your extraction pipeline remains stable.","q":"What methods does web-scraping use to extract content from web pages safely?"},{"a":"Yes, web-scraping supports downloading and archiving social media content from YouTube, Instagram, and TikTok. It integrates with specialized tools like yt-dlp for YouTube videos and Instaloader for Instagram posts, enabling media professionals to archive content systematically. The skill handles the authentication and API quirks of these platforms while maintaining respectful rate limiting to avoid detection and blocking.","q":"Can web-scraping help download content from YouTube, Instagram, and TikTok?"},{"a":"web-scraping includes detection mechanisms for paywalls, CAPTCHAs, and anti-bot protections, allowing you to identify when a page is behind access controls and respond appropriately. Rather than attempting to bypass these protections, the skill helps you recognize when content is restricted and handle the access-denied scenario gracefully. This approach respects publisher business models while helping you determine whether alternative sources or legitimate access methods are available.","q":"How does web-scraping detect and handle paywalls, CAPTCHAs, and anti-bot protections?"},{"a":"web-scraping enables discovery of undocumented public APIs by guiding you through browser developer tools to inspect network requests and identify hidden endpoints. Once you locate a public API call, the skill helps you understand its parameters and call it directly, often providing more reliable data extraction than parsing HTML. This approach is particularly useful for social media platforms and news sites that load content dynamically.","q":"How can I discover and call undocumented public APIs using web-scraping?"},{"a":"web-scraping implements comprehensive error handling for redirects, access denied errors, and other HTTP failures. The skill validates URLs before fetching, follows redirects appropriately, and distinguishes between temporary failures (which may warrant retries) and permanent access restrictions. This layered approach ensures your data collection workflows remain robust even when individual requests fail, allowing you to integrate web-scraping seamlessly into Claude-based journalism and academic research pipelines.","q":"What error handling does web-scraping provide for redirects and access denied responses?"}],"shadow_tags":["ethical-scraping","access-control-bypass","bot-detection","content-extraction","social-media-archival","legal-compliance","browser-automation","api-discovery","rate-limiting","data-validation"],"summary_rewrite":"This skill enables reliable extraction of public web content using layered fallback approaches and robust error management. Built for journalists, researchers, and media professionals who need dependable data collection workflows. Integrates seamlessly with Claude-based journalism and academic research pipelines."},"files":[{"bytes":28922,"path":"dev-toolkit/skills/web-scraping/SKILL.md","sha256":"fe487f06b27bfd8f0617a76925746ec8f5dc3113909f693363f571229ac15116","url":"https://skillfed.io/files/jamditis/claude-skills-journalism/web-scraping/d062c2e1/SKILL.md"}],"id":"jamditis/claude-skills-journalism/web-scraping","links":{"html":"https://skillfed.io/jamditis/claude-skills-journalism/web-scraping","md":"https://skillfed.io/jamditis/claude-skills-journalism/web-scraping.md","repo":"https://github.com/jamditis/claude-skills-journalism"},"meta":{"agents_supported":[],"first_seen":"2026-07-28","forks":58,"language":"Python","last_updated":"2026-07-25","license":"MIT","name":"web-scraping","publisher":"jamditis","stars":342},"relations":{"similar":[{"id":"nicepkg/auto-company/web-scraping"},{"id":"martinholovsky/claude-skills-generator/browser-automation"},{"id":"martinholovsky/claude-skills-generator/dbus"},{"id":"datadrivenconstruction/DDC_Skills_for_AI_Agents_in_Construction/change-order-processor"},{"id":"datadrivenconstruction/DDC_Skills_for_AI_Agents_in_Construction/change-order-manager"},{"id":"datadrivenconstruction/DDC_Skills_for_AI_Agents_in_Construction/claims-documentation"},{"id":"datadrivenconstruction/DDC_Skills_for_AI_Agents_in_Construction/lien-waiver-tracker"},{"id":"martinholovsky/claude-skills-generator/os-keychain"},{"id":"foryourhealth111-pixel/Vibe-Skills/architecture-patterns"},{"id":"HermeticOrmus/LibreUIUX-Claude-Code/architecture-patterns"}]},"slug":{"owner":"jamditis","repo":"claude-skills-journalism","skill":"web-scraping"},"version":"d062c2e1"}