> For the complete documentation index, see [llms.txt](https://awsnotes.dendron.so/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://awsnotes.dendron.so/analytics/aws-data-pipeline/topics/working-with-pipelines.md).

# Working with Pipelines

{% hint style="info" %}
This page was generated from content adapted from the [AWS Developer Guide](https://github.com/awsdocs/aws-data-pipeline-developer-guide.git)
{% endhint %}

## Scheduling Pipelines

* **Note**\
  You can find the Default object on the Architect page in the **Other** section.
* **Note**\
  The minimum scheduling interval is 15 minutes.
* **Note**\
  You can find the Default object on the Architect page in the **Other** section.

## Cloning Your Pipeline

* **Note**\
  You can't clone a pipeline using the command line interface (CLI).

## Deleting Your Pipeline

* **Important**\
  You can't restore a pipeline after you delete it, so be sure that you won't need the pipeline in the future before you delete it.

## Staging Data and Tables with Activities

* **Note**\
  Staging only functions when the `stage` field is set to `true` on an activity, such as `ShellCommandActivity`. For more information, see [ShellCommandActivity](https://github.com/kevinslin/aws-reference-notes/blob/main/services/aws_data_pipeline/dp-object-shellcommandactivity.md).
* **Note**\
  This scenario only works as described if your data inputs and outputs are `S3DataNode` objects. Additionally, output data staging is allowed only when `directoryPath` is set on the output `S3DataNode` object.
* **Note**\
  This scenario only works as described if your data inputs and outputs are `S3DataNode` or `MySqlDataNode` objects. Table staging is not supported for `DynamoDBDataNode`.
* **Note**\
  In this example, the table name variable has the # (hash) character prefix because AWS Data Pipeline uses expressions to access the `tableName` or `directoryPath`. For more information about how expression evaluation works in AWS Data Pipeline, see [Expression Evaluation](https://github.com/kevinslin/aws-reference-notes/blob/main/services/aws_data_pipeline/dp-pipeline-expressions.md#dp-datatype-functions).

## Using Resources in Multiple Regions

* **Note**\
  The following list includes regions in which AWS Data Pipeline can orchestrate workflows and launch Amazon EMR or Amazon EC2 resources. AWS Data Pipeline may not be supported in these regions. For information about regions in which AWS Data Pipeline is supported, see [AWS Regions and Endpoints](https://docs.aws.amazon.com/general/latest/gr/rande.html#datapipeline_region).

## Pipeline Definition File Syntax

* **Note**\
  On user-defined fields, AWS Data Pipeline only checks for valid references to other pipeline components, not any custom field string values that you add.

## Working with the API

* **Note**\
  If you are not writing programs that interact with AWS Data Pipeline, you do not need to install any of the AWS SDKs. You can create and run pipelines using the console or command-line interface. For more information, see [Setting up for AWS Data Pipeline](https://github.com/kevinslin/aws-reference-notes/blob/main/services/aws_data_pipeline/dp-get-setup.md)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://awsnotes.dendron.so/analytics/aws-data-pipeline/topics/working-with-pipelines.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.