Cloud Data Fusion version 6.11.1.2 is generally available (GA).

This release includes the following changes:

• Fixed the triggers panel in the pipeline details page to display the correct triggers count on initial load (CDAP-21230).

• Updated the GraphQL query mechanism to cache the pipelines list and fix the long loading screen in the deployed pipelines list page (CDAP-21229).

The SAP ODP plugin version 0.12.1 is available in Cloud Data Fusion version 6.10 and later. This release includes the following features and changes:

• Hierarchy data sources are now supported for extraction.

• ODQ fields are populated with TSN, package number, and record number for full extraction.

• Structured SAP messages are now displayed according to the message type (for example, type E as error and type I as information). Earlier, all messages from SAP displayed as errors.

• You can now use the TVARVC table entries with parameter /GOOG/ODP_TRANSFER_MODE (for all data sources) or /GOOG/M_DatasourceName (for individual data sources) to determine if data is transferred in CHAR or XML format. Maintain 'C' to transfer data in CHAR format. Maintain an empty value to transfer data in XML format.

Cloud Data Fusion version 6.11.1.1 is generally available (GA). This release includes the following feature:

• InstanceV3 monitored-resource: Introduced datafusion.googleapis.com/InstanceV3 as the default monitored resource for instance-level metrics and system service logs. This resource excludes the org_id and namespace labels found in InstanceV2. Emission of InstanceV2 metrics and logs is disabled by default for new and upgraded instances but can be re-enabled using the REST API.

  For more information, see Metrics overview and View pipeline logs.

Fixed

Fixed in Cloud Data Fusion 6.11.1.1:

• Fixed retries in message publishing when the messaging service is temporarily unavailable (CDAP-21043).
• Fixed a security vulnerability where user-provided code in preview runners could access sensitive data from other preview runs (CDAP-21211).
• Fixed an issue where internal task workers running user code could hang indefinitely. The system now forces completed tasks to exit and uses a health check to restart unresponsive workers (CDAP-21213).
• Fixed an issue where the list apps API endpoint failed to return all deployed pipelines when used with pagination (CDAP-21220).

The SAP Table Reader plugin version 0.11.5 is available in Cloud Data Fusion version 6.10 and later.

This release includes the following change:

• Fixed an issue in the SAP Table Reader where running in Snapshot mode caused the Clean Up RFM process to generate an unnecessary Job abort failed error message in the detail logs.

The Salesforce plugin version 1.7.0 is available in Cloud Data Fusion version 6.8.0 and later. This release includes the following change:

• Upgrade of Salesforce Bulk API V1 version from 62.0 to 64.0 (PLUGIN-1926).

Salesforce has deprecated certain fields in the API version 64.0. Upgrading to Salesforce plugin version 1.7.0 might cause pipelines that use these fields, to fail. To ensure your pipelines continue to work, you must manually update your pipeline schema to either load a new schema or remove the deprecated fields. For more information, see Prerequisites for upgrading to Salesforce plugin version 1.7.0.

The ServiceNow plugin version 1.2.7 is available in Cloud Data Fusion version 6.10.1. This release includes the following change:

• Fixed an issue related to schema backward compatibility while upgrading from plugin version 1.1.0 (PLUGIN-1902).

Cloud Data Fusion version 6.11.1 is generally available (GA). This release includes the following features:

• Added support for HTTP access tokens (Bearer authentication) in Bitbucket Server for source control management (CDAP-21049).
• A new API is available to retrieve the application count for each namespace (CDAP-21161).

Fixed

Fixed in Cloud Data Fusion 6.11.1:

• To prevent storage issues on static Dataproc clusters, temporary pipeline data is automatically deleted after a successful run (CDAP-21076).
• Fixed an issue that prevented Expression Language (EL) from being used within user-defined directives (CDAP-21204).
• Fixed an issue where a pipeline would fail when reading an encrypted file from Cloud Storage using Tink (PLUGIN-1717).

Deprecated

Dataproc 2.0 is no longer supported in Cloud Data Fusion version 6.11.1 and later.

Change

Changes in Cloud Data Fusion 6.11.1:

• The Java runtime environment is upgraded from Java 8 to Java 11 (CDAP-21184).
• To create ephemeral Dataproc clusters, Cloud Data Fusion uses the Dataproc 2.3 image by default (CDAP-21187).
• The pipeline JSON size limit for creating new pipelines and importing pipelines as JSON is increased to 5MB (previously 2MB) (CDAP-21194).
• On the Pipeline details page, the inbound triggers sidebar features a paginated list of pipelines where you can select the pipelines you want to add to the trigger. Additionally, a refresh button is added to update the existing list of triggers and pipelines (CDAP-21195).

You can now attach tags to Cloud Data Fusion instances during instance creation. This ensures immediate metadata for better organization, cost tracking, and policy automation. For more information, see Attach tags during instance creation.

The Cloud Data Fusion version 6.10.1.5 patch revision is generally available (GA). 6.10.1.5 includes reliability fixes.

The SAP ODP plugin version 0.12 is available in Cloud Data Fusion version 6.10 and later. This release includes the following changes:

• Support for old SAP ODP RFCs is removed to align with SAP Note 3255746.
• Messages have been updated when attempting to extract data from unsupported hierarchy data sources.

We recommend upgrading all your pipelines to the latest SAP ODP plugin version, as earlier versions are no longer compliant with SAP Guidelines.

The Oracle plugin version 1.12.3 is available in Cloud Data Fusion (via Hub) versions 6.11.0 and later, and 1.11.8 is available in Cloud Data Fusion (via Hub) version 6.10.

This release provides backward compatibility for recent schema changes, including the following:

• Precisionless numbers: In version 1.10.0, precisionless numbers were handled as strings.
• Improved timestamp handling: Version 1.11.0 introduced improvements to timestamp handling.

To address backward compatibility for these changes, two new hidden fields are introduced in Oracle batch source configurations: treatPrecisionlessNumAsDeci and treatAsOldTimestamp. Both flags default to false. To enable these flags, edit the respective values in your exported connection JSON (if using connections) or pipeline JSON (if not using connections) before re-importing or re-deploying (PLUGIN-1893).

The Elasticsearch plugin version 1.11.0 is available in Cloud Data Fusion version 6.11.0. This release includes the following change:

• Upgraded Hadoop version for Elasticsearch plugin compatibility (PLUGIN-1881).

The HTTP plugin version 1.4.4 is available in Cloud Data Fusion version 6.10.1. This release includes the following changes:

• Implemented the Client Credentials Grant flow for HTTP OAuth2, enabling authorized clients to securely access data using the client_credentials grant type. Client credentials can be passed through Basic Authentication header, in the request body, or as query parameters (PLUGIN-1872).

• Fixed an issue causing the HTTP Source plugin to throw a NullPointerException when the BasePageIterator received a null response (PLUGIN-1894).

Cloud Data Fusion is available in the northamerica-south1 (Mexico) region. For more information, see Pricing.

The Salesforce plugin version 1.6.10 is available in Cloud Data Fusion versions 6.10.1 and 6.11.0. This release includes the following changes:

• Upgraded Salesforce API version from 53.0 to 62.0 (PLUGIN-1891).
• Added a retry mechanism for all Salesforce API calls from the Salesforce plugin (PLUGIN-1892).

The DB plugin versions 1.11.7 and 1.12.2 are available in Cloud Data Fusion versions 6.10.1 and 6.11.0 respectively. This release includes the following change:

• Added the TRANSACTION_ISOLATION_LEVEL property to DB Plugins (PostgreSQL, MySQL, and MSSQL).

The MariaDB Batch Source plugin versions 1.11.7 and 1.12.2 are available in Cloud Data Fusion versions 6.10.1 and 6.11.0 respectively. This release includes the following changes:

• Fixed an issue related to the MariaDB YEAR data type causing pipeline failures when the date type was manually changed to DATE. Validation for this specific scenario is now skipped (PLUGIN-1888).

• Fixed an issue that caused pipeline failures with a java.lang.ClassNotFoundException: Class io.cdap.plugin.db.DBRecord not found error, which was due to a class loading problem (PLUGIN-1874).

• Added a committer to Database plugins to manage committing or rolling back entire tasks, depending on whether the task completes successfully or is aborted (PLUGIN-1815).

Cloud Data Fusion version 6.9 is no longer supported. You should upgrade your instances to run in a supported version. For instructions, see Manage version upgrades for instances and pipelines.

The Python Transform plugin (version 2.3.2) is available in CDAP version 6.10.1. This includes bug fix for the deprecated PROTOCOL_TLSv1 on Dataproc 2.0 and later. The issue occurs when earlier TLS versions, such as TLSv1 and TLSv1.1 are disabled by default due to security concerns. Applications relying on ssl.PROTOCOL_TLSv1 in Python might fail and requires updates to use ssl.PROTOCOL_TLSv1_2 or later.

Cloud Data Fusion version 6.11.0 is available in Preview.

Feature

Cloud Data Fusion 6.11.0 offers high availability with reduced upgrade downtime.

Feature

When a pipeline run fails, you can retrieve detailed error information on the pipeline details page of the Cloud Data Fusion web interface.

Cloud Data Fusion classifies pipeline errors by category, reason, and message. This classification speeds up resolution and reduces the need to examine complex logs. For more information, see Retrieve error information for a failed pipeline run.

Feature

You can view instance metrics and pipeline metrics in Cloud Monitoring and in the dashboard provided by Cloud Data Fusion. For more information, see Metrics overview and Monitor Cloud Data Fusion system, instance, and pipeline health.

Feature

You can view instance logs and pipeline logs in Cloud Logging, and in the dashboard provided by Cloud Data Fusion. For more information, see View Cloud Data Fusion logs.

Deprecated

The following APIs for downloading system service and pipeline run logs are deprecated in 6.11.0:

• GET /v3/namespaces/<NAMESPACE_ID>/apps/<APP_ID>/<PROGRAM_TYPE>/<PROGRAM_ID>/logs
• GET /v3/system/services/<SERVICE_ID>/logs

Deprecated

The ability to retrieve all applications without pagination using the GET /v3/namespaces/<NAMESPACE_ID>/apps endpoint is deprecated in 6.11.0.

Fixed

Fixed in Cloud Data Fusion 6.11.0:

• Fixed a null pointer exception in the BigQuery multi-sink plugin when used without a reference name (PLUGIN-1843).

• Fixed Joiner plugin failures observed on Dataproc 2.2-debian12 instances (CDAP-21075).

• Fixed an issue that prevented pipelines from accepting empty input in Amazon S3 and Google Cloud Storage source plugins (PLUGIN-1742).

• Fixed an issue in RBAC-enabled instances where the pipeline details page displayed an incorrect author name (CDAP-21069).

Change

Changes in Cloud Data Fusion 6.11.0:

• To create ephemeral clusters, Cloud Data Fusion uses the Dataproc 2.2 image by default. For more information about its limitations in Cloud Data Fusion, see Change the Dataproc image to version 2.1.

• The maximum concurrent runs limit for triggers is displayed in the console (CDAP-21072).

• Added support for destination table write preference in the BigQuery Execute plugin (PLUGIN-1438).

Deprecated

The following APIs for searching and querying metrics are deprecated in 6.11.0:

• POST v3/metrics/query
• POST v3/metrics/search

Breaking

A soft limit of 2 MB for pipeline JSON size is introduced in 6.11.0. Pipelines exceeding this size might encounter deployment failures.

The SAP OData plugin version 0.11.6 is available in Cloud Data Fusion version 6.8.0 and later. This release includes the following change:

Fixed an issue causing pipeline deployments to fail due to SAP memory dumps when processing large datasets with macro filters: ERROR Stage 'SAP OData' encountered : CDF_SAP_ODATA_01534 - Service validation failed. Root Cause:Invalid parametertype used at function.

You can use custom constraints with Organization Policy to provide more granular control over specific fields for some Cloud Data Fusion resources. For more information, see Create custom organization policy constraints.

The SAP SuccessFactors plugin version 1.2.4 is available in Cloud Data Fusion version 6.8.0 and later. This release lets you use OAuth 2.0 for ODATA API authentication (PLUGIN-1741).

The Cloud Data Fusion version 6.10.1.2 patch revision is generally available (GA). 6.10.1.2 includes the following changes:

• You can generate audit logs that record data plane activities within your Cloud Data Fusion instance. Data plane audit logging is available in Preview for RBAC-enabled instances.

• To improve the API response time, by default, all program activity records older than 30 days are cleaned up. Any activity older than 30 days isn't visible in the Cloud Data Fusion studio (CDAP-14950).

• When using role-based access control, performing the List Pipelines operation requires datafusion.pipelines.list permission, in addition to datafusion.namespaces.get permission. For more information, see RBAC roles and permissions (CDAP-20931).

• Fixed an issue causing the flow control metric, flowcontrol.launching.count, to overcount in cases where servers were restarted when a pipeline run was in progress (CDAP-21046).

• Fixed an issue causing the flow control metric, flowcontrol.launching.count, to be stale after a restart when no pipelines were running (CDAP-21048).

• Fixed an issue causing the default max concurrent runs limit for triggers not to appear in the web interface, making it difficult to tell if triggers were working as intended (CDAP-21072).

• Fixed an issue causing the top panel of the Studio tab to disappear when you edited a pipeline draft that's based on a pipeline from an earlier Cloud Data Fusion version (CDAP-21073).

• Improved performance by removing a call to the list apps API during pipeline deployment when checking if a pipeline already exists (CDAP-21074).

Cloud Data Fusion supports the CMEK organization policy.

The Snowflake plugin version 1.1.4 is available in Cloud Data Fusion version 6.8.0 and later. This release includes the following changes (PLUGIN-1816):

• Fixed an issue in the Snowflake source causing pipelines to fail if fields contained decimals.
• Fixed an issue in the Snowflake source causing pipelines to fail if data contained a backslash (\). You can set a new escape character using the cdap.snowflake.source.escape runtime argument.

Fixed

The Cloud SQL MySQL plugins version 1.11.5 is available in Cloud Data Fusion versions 6.10.0 and later. This release fixes an issue in the Cloud SQL MySQL sink causing pipelines to fail when the schema contains a MySQL reserved word (PLUGIN-1017).

The HTTP plugin version 1.4.3 is available in Cloud Data Fusion version 6.8.0 and later. This release includes the following changes (PLUGIN-1810):

• In the HTTP streaming source, batch source, and batch sink, a PATCH option was added to the HTTP Method field.
• Fixed an issue in the HTTP sink causing data loss when a pipeline didn't fail by default when a non-2xx response code was received.
• Fixed an issue in the HTTP source causing a pipeline not to fail when the get_schema method was called and a non-2xx response code was received.

The Cloud SQL MySQL plugins version 1.11.5 is available in Cloud Data Fusion versions 6.8.0 and later. This release fixes an issue in the Cloud SQL MySQL sink causing pipelines to fail when the schema contains a MySQL reserved word (PLUGIN-1017). This note is incorrect; see entry for November 27, 2024.

Fixed

The SAP table batch source plugin version 0.11.5 is available in Cloud Data Fusion version 6.8.0 and later. This release fixes an issue causing the following error: Error encountered while configuring the stage: Unable to access Cloud Storage or download JCo libraries from Cloud Storage.

The Multiple table plugin version 1.4.1 is available in Cloud Data Fusion versions 6.10.1 and later. This release fixes an issue causing pipelines to fail if a Multiple database tables batch source's Reference Name field contains spaces. The field no longer accepts spaces (PLUGIN-1752).

The Oracle plugin version 1.11.4 is available in Cloud Data Fusion versions 6.10.1 and later. This release includes the following change:

• Fixed an issue causing pipelines with an Oracle sink that has date columns in the input schema to fail (PLUGIN-1812).