Amazon OpenSearch Ingestion

This page was generated from content adapted from the AWS Developer Guide

Setting up roles and users

  • Note The S3 access role that you specify must be the same as the pipeline role. Therefore, your pipeline role must contain two separate permissions policies—one to write to a sink, and one to pull from the S3 source. You must use the same sts_role_arn in all pipeline components.

Getting started with OpenSearch Ingestion

Pipeline features overview

Creating pipelines

  • Note If you're writing to a domain that uses fine-grained access control, there are extra steps you need to complete. See Fine-grained access control.
  • Note In addition, the first user to create a pipeline in an account must have permissions for the iam:CreateServiceLinkedRole action. For more information, see pipeline role resource.
  • Note OpenSearch Ingestion doesn't immediately support new versions of Data Prepper as soon as they're released. There will be some lag between when a new version is publicly available and when it's supported in OpenSearch Ingestion. In addition, OpenSearch Ingestion might explicitly not support certain major or minor versions altogether. For a comprehensive list, see Supported Data Prepper versions.
  • Note If you specify multiple sinks within a YAML pipeline definition, they must all be the same OpenSearch Service domain. An OpenSearch Ingestion pipeline can't write to multiple different domains.

Supported plugins and options

  • Note OpenSearch Ingestion doesn't support any buffer plugins because it automatically configures a default buffer. You receive a validation error if you include a buffer in your pipeline configuration.
  • Note OpenSearch Ingestion doesn't support any buffer plugins because it automatically configures a default buffer. You receive a validation error if you include a buffer in your pipeline configuration.

Sending data to pipelines

  • Note To use the role for all pipelines, replace the ARN in the Resource element with a wildcard (*).
  • Note You can only provide cross-account ingestion access for public pipelines, not VPC pipelines.