Working with Pipelines

This page was generated from content adapted from the

Scheduling Pipelines

Note You can find the Default object on the Architect page in the Other section.
Note The minimum scheduling interval is 15 minutes.
Note You can find the Default object on the Architect page in the Other section.

Cloning Your Pipeline

Note You can't clone a pipeline using the command line interface (CLI).

Deleting Your Pipeline

Important You can't restore a pipeline after you delete it, so be sure that you won't need the pipeline in the future before you delete it.

Staging Data and Tables with Activities

Note Staging only functions when the stage field is set to true on an activity, such as ShellCommandActivity. For more information, see .
Note This scenario only works as described if your data inputs and outputs are S3DataNode objects. Additionally, output data staging is allowed only when directoryPath is set on the output S3DataNode object.
Note This scenario only works as described if your data inputs and outputs are S3DataNode or MySqlDataNode objects. Table staging is not supported for DynamoDBDataNode.
Note In this example, the table name variable has the # (hash) character prefix because AWS Data Pipeline uses expressions to access the tableName or directoryPath. For more information about how expression evaluation works in AWS Data Pipeline, see .

Using Resources in Multiple Regions

Pipeline Definition File Syntax

Note On user-defined fields, AWS Data Pipeline only checks for valid references to other pipeline components, not any custom field string values that you add.

Working with the API

PreviousAWS Data Pipeline Concepts NextPipeline Expressions and Functions

Last updated 1 year ago

Was this helpful?

Staging Data and Tables with Activities

Note Staging only functions when the stage field is set to true on an activity, such as ShellCommandActivity. For more information, see .

Note This scenario only works as described if your data inputs and outputs are S3DataNode objects. Additionally, output data staging is allowed only when directoryPath is set on the output S3DataNode object.

Note This scenario only works as described if your data inputs and outputs are S3DataNode or MySqlDataNode objects. Table staging is not supported for DynamoDBDataNode.

Note In this example, the table name variable has the # (hash) character prefix because AWS Data Pipeline uses expressions to access the tableName or directoryPath. For more information about how expression evaluation works in AWS Data Pipeline, see .