Replication API reference

A replication job is a CDAP application, and can be managed with CDAP REST APIs.

Create a replication job

Use the CDAP application deployment API to create a replication job.

PUT -H "Authorization: Bearer ${AUTH_TOKEN}" "${CDAP_ENDPOINT}/v3/namespaces/namespace-id/apps/replicator-name"
Parameter Description
namespace-id The namespace in which to create the replication job. If your pipeline belongs to a Basic edition instance, the namespace ID is always default. If your pipeline belongs to an Enterprise edition instance, you can create a namespace.
replicator-name Replicator name

Example Config

The following (partial) config of the app is sent in the API request body:

{
    "name": "my-sample-pipeline",
    "artifact": {
        "name": "delta-app",
        "version": "0.6.2",
        "scope": "SYSTEM"
    },
    "config": {
        "connections": [
            {
                "from": "Oracle (by Datastream)",
                "to": "BigQuery"
            }
        ],
        "stages": [
            {
                "name": "Oracle (by Datastream)",
                "plugin": {
                    "name": "OracleDatastream",
                    "type": "cdcSource",
                    "artifact": {
                        "name": "datastream-delta-plugins",
                        "version": "0.4.2",
                        "scope": "SYSTEM"
                    },
                    "properties": {
                        "usingExistingStream": "false",
                        "region": "us-central1",
                        "connectivityMethod": "ip-allowlisting",
                        "port": "1521",
                        "sid": "ORCL",
                        "replicateExistingData": "true",
                        "project": "auto-detect",
                        "dsServiceAccountKey": "auto-detect",
                        "gcsServiceAccountKey": "auto-detect",
                        "host": "",
                        "user": "",
                        "password": ""
                    }
                }
            },
            {
                "name": "BigQuery",
                "plugin": {
                    "name": "bigquery",
                    "type": "cdcTarget",
                    "artifact": {
                        "name": "bigquery-delta-plugins",
                        "version": "0.6.3",
                        "scope": "SYSTEM"
                    },
                    "properties": {
                        "project": "auto-detect",
                        "serviceAccountKey": "auto-detect",
                        "stagingBucketLocation": "us",
                        "loadInterval": "90",
                        "stagingTablePrefix": "_staging_",
                        "requireManualDrops": "false",
                        "softDeletes": "false",
                        "datasetName": "mysampledataset"
                    }
                }
            }
        ],
        "tables": [
            {
                "database": "ORCL",
                "table": "MYTABLE1",
                "schema": "HR"
            },
            {
                "database": "ORCL",
                "table": "MYTABLE2",
                "schema": "HR"
            }
        ],
        "parallelism": {
            "numInstances": 1
        },
        "tableTransformations": []
    }
}

Contents of the replication job config depend on the Cloud Data Fusion instance and plugin versions. To get started, export an existing replication job or draft to use as a template for creating the request body.

See the CDAP application deployment API for more information.

List replication jobs

Use the CDAP application list API to list replication jobs.

GET -H "Authorization: Bearer ${AUTH_TOKEN}" "${CDAP_ENDPOINT}/v3/namespaces/namespace-id/apps"
Parameter Description
namespace-id The namespace of the replication jobs to list. If your pipeline belongs to a Basic edition instance, the namespace ID is always default.

See the CDAP application list API for more information.

View replication job details

Use the CDAP application detail API to view the details of a replication job.

GET -H "Authorization: Bearer ${AUTH_TOKEN}" "${CDAP_ENDPOINT}/v3/namespaces/namespace-id/apps/replicator-name"
Parameter Description
namespace-id The namespace of the replication job to view. If your pipeline belongs to a Basic edition instance, the namespace ID is always default.
replicator-name Replicator name

See the CDAP application detail API for more information.

Start a replication job

Use the CDAP program start API to start a replication job.

POST -H "Authorization: Bearer ${AUTH_TOKEN}" "${CDAP_ENDPOINT}/v3/namespaces/namespace-id/apps/replicator-name/workers/DeltaWorker/start"
Parameter Description
namespace-id The namespace of the replication job to start. If your pipeline belongs to a Basic edition instance, the namespace ID is always default.
replicator-name Replicator name

See the CDAP program start API for more information.

Stop a replication job

Use the CDAP program stop API to stop a replication job.

POST -H "Authorization: Bearer ${AUTH_TOKEN}" "${CDAP_ENDPOINT}/v3/namespaces/namespace-id/apps/replicator-name/workers/DeltaWorker/stop"
Parameter Description
namespace-id The namespace of the replication job to stop. If your pipeline belongs to a Basic edition instance, the namespace ID is always default.
replicator-name Replicator name

See the CDAP program stop API for more information.

What's next