managed-ingestion-service
Scale document ingestion into Amazon OpenSearch using OSIS pipelines. Upload pre-chunked JSONL to S3 for indexing, or send raw PDFs to the cloud for automatic parsing and embedding. Supports both OpenSearch Service domains and Serverless collections.
Managed Ingestion Service ingests documents into Amazon OpenSearch at scale via OSIS pipelines from S3.
AI-generated summary based on this skill's SKILL.md
Install
opensearch-project/opensearch-agent-skills/managed-ingestion-service · repository language: Python
git clone https://github.com/opensearch-project/opensearch-agent-skills
cp -r opensearch-agent-skills/skills/opensearch-skills/cloud/managed-ingestion-service ~/.claude/skills/managed-ingestion-servicenpx skillfed install opensearch-project/opensearch-agent-skills/managed-ingestion-serviceFrequently asked questions
AI-generated answers based on this skill's SKILL.md and metadata
How to ingest documents into OpenSearch at scale?
managed-ingestion-service enables large-scale document ingestion through OSIS pipelines that connect S3 sources directly to OpenSearch domains or Serverless collections. You can upload pre-chunked JSONL files or raw PDFs; the service handles parsing, chunking, and indexing automatically with optional semantic enrichment including automatic embeddings.
What is an OSIS pipeline and how do I set one up?
managed-ingestion-service uses OpenSearch Ingestion Service (OSIS) pipelines as managed connectors between cloud storage and OpenSearch. Set up involves configuring a pipeline via YAML that specifies your S3 bucket as source, document processing options (PDF/DOCX extraction, chunking), and your target OpenSearch domain or Serverless collection as sink.
Can I upload PDF to Amazon OpenSearch from S3?
Yes. managed-ingestion-service supports uploading raw PDF and DOCX files from S3 buckets to OpenSearch. The OSIS pipeline automatically extracts text, applies semantic enrichment, and indexes documents into your OpenSearch domain or Serverless collection without manual preprocessing.
How does managed-ingestion-service handle semantic enrichment?
managed-ingestion-service can automatically apply semantic enrichment during ingestion, including automatic sparse embeddings and dense vector generation. This enrichment happens within the OSIS pipeline, transforming raw documents into searchable, semantically-aware indexed content ready for hybrid or vector search queries.
What document formats does managed-ingestion-service support?
managed-ingestion-service handles pre-chunked JSONL files and raw cloud documents in PDF and DOCX formats. JSONL chunks can be uploaded directly from S3 for immediate indexing, while raw PDFs and DOCX files are automatically extracted, parsed, and chunked before indexing into OpenSearch.
Can I migrate my local document workflow to managed-ingestion-service?
Yes. managed-ingestion-service simplifies migration by replacing local ingestion infrastructure with managed OSIS pipelines. Upload your documents to S3, configure the pipeline to point to your bucket and target OpenSearch domain or Serverless collection, and the service handles all cloud-based processing and indexing automatically.
SKILL.md
rendered from the published skill — quoted content, verbatim
Managed Ingestion Service
Overview
Ingest documents into Amazon OpenSearch at scale via OSIS pipelines. Supports both Amazon OpenSearch Service (AOS) domains and Amazon
(truncated - see the full file via the links below)
Read as markdown · JSON record · Browse the source repository
File tree — 2 files
skills/opensearch-skills/cloud/managed-ingestion-service/SKILL.md
skills/opensearch-skills/cloud/managed-ingestion-service/iam-setup.md