Provisioners let you automate cluster lifecycle management when running pipelines across different cloud environments. They create the required compute cluster before a run begins and tear it down after the pipeline finishes.
Each provisioner provides configuration settings to control cluster properties and authenticate with target cloud platforms. For example, provisioners for Managed Service for Apache Spark and Amazon EMR configure cluster size and the credentials required to provision compute nodes.
Supported provisioners in Cloud Data Fusion
Cloud Data Fusion supports the following provisioners:
- Managed Service for Apache Spark
- A fast, easy-to-use, and fully-managed cloud service for running Apache Spark and Apache Hadoop clusters.
- Amazon Elastic MapReduce (EMR)
- Provides a managed Hadoop framework that processes vast amounts of data across dynamically scalable Amazon EC2 instances.
- Remote Hadoop
- Runs jobs on a pre-existing Hadoop cluster, either on-premises or in the cloud.