web-scraping-olostep
web-scraping-olostep provides API endpoints to scrape pages, crawl websites, and extract structured answers from web content. Use it to initiate scrapes with custom formatting, perform AI-driven searches across multiple pages, map site URLs, or batch-process large crawls with filtering and depth controls.
web-scraping-olostep extracts and retrieves content from web pages at scale through API endpoints for scraping, crawling, and AI-powered answer extraction.
AI-generated summary based on this skill's SKILL.md
Install
gooseworks-ai/goose-skills/web-scraping-olostep · repository language: Python
git clone https://github.com/gooseworks-ai/goose-skills
cp -r goose-skills/skills/research-tools/capabilities/web-scraping-olostep ~/.claude/skills/web-scraping-olostepFrequently asked questions
AI-generated answers based on this skill's SKILL.md and metadata
How does web-scraping-olostep extract data from web pages at scale using API?
web-scraping-olostep provides API endpoints that let you scrape pages and extract structured data at scale. You can initiate scrapes with custom formatting, process multiple URLs in batches, and retrieve web content with filtering options. The API handles large crawls efficiently, making it ideal for bulk URL scraping and automated web scraping workflows.
What is web-scraping-olostep's approach to web crawling and URL discovery?
web-scraping-olostep crawls websites and discovers all URLs with built-in filtering options and depth controls. You can map entire site structures, get all links from a website, and apply custom rules to filter results. The crawler API supports both targeted and comprehensive site exploration depending on your needs.
Can web-scraping-olostep use AI to find answers by searching and browsing the web?
Yes, web-scraping-olostep integrates AI-driven search capabilities that let you find answers by searching and browsing web content across multiple pages. This AI-powered web scraping feature goes beyond simple data extraction to deliver intelligent insights from the pages you crawl.
How does web-scraping-olostep handle batch processing of multiple URLs?
web-scraping-olostep processes multiple URLs in batches with structured data parsing and filtering controls. You can submit large lists of URLs for bulk scraping, apply depth and filtering rules, and retrieve parsed results efficiently. This batch capability makes it practical for enterprise-scale web content extraction projects.
Can web-scraping-olostep monitor and analyze competitor websites?
web-scraping-olostep supports website monitoring and competitor analysis by crawling and extracting content from target sites. You can track content changes, extract structured data from competitor pages, and monitor site updates over time using the API's crawl and extraction features.
What license does web-scraping-olostep use?
web-scraping-olostep is released under the MIT license, allowing free use, modification, and distribution for both personal and commercial projects with minimal restrictions.
SKILL.md
rendered from the published skill — quoted content, verbatim
Olostep - Web Scraping & Crawling API
Setup
Read your credentials from ~/.gooseworks/credentials.json:
export GOOSEWORKS_API_KEY=$(python3 -c "import json;print(json.load(open('$HOME/.gooseworks/credentials.json'))['api_key'])")
export GOOSEWORKS_API_BASE=$(python3 -c "import json;print(json.load(open('$HOME/.gooseworks/credentials.json')).get('api_base','https://api.gooseworks.ai'))")
If ~/.gooseworks/credentials.json does not exist, tell the user to run: npx gooseworks login
All endpoints use Bearer auth: -H "Authorization: Bearer $GOOSEWORKS_API_KEY"
Powerful web scraping, crawling, and AI-powered content extraction.
Capabilities
- Create Scrape: Initiate a web page scrape
- Create Answer: The AI will perform actions like searching and browsing web pages to find the answer to the provided task
- Maps: This endpoint allows users to get all the urls on a certain website
- Start Crawl: Starts a new crawl
- Start Batch: Starts a new batch
- Batch Items: Retrieves the list of items processed for
(truncated - see the full file via the links below)
Read as markdown · JSON record · Browse the source repository
File tree — 2 files
skills/research-tools/capabilities/web-scraping-olostep/SKILL.md
skills/research-tools/capabilities/web-scraping-olostep/skill.meta.json