Release notes

This page documents production updates to Manufacturing Data Engine. Check this page for announcements about new or updated features, bug fixes, known issues, and deprecated functionality.

Versions covered in this change log:

Version	Supported for new deployments	Archived	Critical update	API compatible MC versions
v1.5.1		No	Yes	2.22 - 2.26
v1.5.0		No	No	2.22 - 2.26
v1.4.2		No	Yes	2.22 - 2.26
v1.4.1		No	No	2.22 - 2.26
v1.4.0		No	No	2.22 - 2.26
v1.3.8		No	No	2.15 - 2.22
v1.3.7		No	No	2.15 - 2.22
v1.3.6		No	No	2.15 - 2.22
v1.3.5		No	Yes	2.15 - 2.22
v1.3.4		No	No	2.15 - 2.22
v1.3.3		No	No	2.15 - 2.22
v1.3.2		No	Yes	2.15 - 2.22
v1.3.1		Yes	Yes	2.15 - 2.22
v1.3.0		Yes	Yes	2.15

You can see the latest product updates for all of Google Cloud on the Google Cloud page, browse and filter all release notes in the Google Cloud console, or programmatically access release notes in BigQuery.

To get the latest product updates delivered to you, add the URL of this page to your feed reader, or add the feed URL directly.

August 29, 2025

Libraries

Release signature

b0fc163
1.5.1
ffb87d39d343c20abebd2f52df74a2d3

Security

• Improvement (427447932): Brought Docker images to versions without vulnerabilities reported at the time of the release.
• Improvement (361290775): Modified Terraform deployment scripts to enforce TLS v1.2 for an external MDE UI Load Balancer.

Announcement

Release 1.5.1

This release is a critical update if you have an existing deployment of MDE version 1.5.0. This release resolves a bug regarding materialization of metadata instances created prior to MDE 1.5.0. This release also includes other minor improvements and bug fixes.

March 31, 2025

Change

• Metadata Versioning: To enhance the query readability of the metadata instances, MDE v1.5.0 introduced a new field called validFrom that designates the time when a particular metadata instance is effective. MDE uses this field to check which metadata instance picks based on the source message event time, not processing time, enabling accurate historical data representation. For more information, see Versioning metadata buckets.

Feature

• Configuration Packages: Introduces a file-based configuration package system for atomic deployments and GitOps integration. For more information, see Upload configuration package and File content.
• Development Mode: Adds a "Development Mode" to allow deletion of MDE entities and configuration packages, use with caution. For more information, see Development mode.

Announcement

Release 1.5.0

This release is not a critical update, unless you are directly impacted by the bug fixes, you don't need to update, and you can wait for future releases before updating.

December 13, 2024

Announcement

Release 1.4.2

This release is a critical update if you have an existing deployment of MDE version 1.4.0 or 1.4.1. This release resolves a bug regarding the raw messages writing, where the Cloud Storage Avro files were not being written.

Libraries

Release signature

57c3062
1.4.2
ccc9f5b6c30a1c1a42eae04c995f783b

Announcement

Release 1.4.2

This release is a critical update if you have an existing deployment of MDE version 1.4.0 or 1.4.1. This release resolves a bug regarding the raw messages writing, where the Cloud Storage Avro files were not being written.

Fixed

• Fix (381418590): Fixing an issue of the generated BigQuery views for the types when one of the bucket or data field schemas contains arrays.
• Fix (382189410): Fixing issue of the Cloud Storage Writer not writing the raw messages to the Cloud Storage bucket.

Change

Important: If you are upgrading from 1.4.0 or 1.4.1 make sure that you manually drain the Cloud Storage writer dataflow job, which forces the raw message files to be written to Cloud Storage before shutting down the job.

Fixed

• Fix (381418590): Fixing an issue of the generated BigQuery views for the types when one of the bucket or data field schemas contains arrays.
• Fix (382189410): Fixing issue of the Cloud Storage Writer not writing the raw messages to the Cloud Storage bucket.

November 07, 2024

Change

Important: This release requires different runtime versions for Terraform to use the latest Terraform Google Cloud providers and solve multiple deployment bugs. For more information, see the deployment guide.

Fixed

• Fix (375367674): Fixing issue when the edge messages contained fields named event or headers which was colliding with the MDE event, and headers attribute that are used during the message-class resolution. The MDE message headers are now reserved in a new keyword called mdeHeaders.
• Fix (371176757): Fixing issue of losing the monitoring metrics after upgrading from 1.3.x to 1.4.0. parsers.
• Fix (377694402): Upgrading the Apache Beam runtime to the latest version of 2.60.0.
• Fix (377672812): Upgrading Terraform runtime and the required Google Cloud provider versions to solve multiple deployment bugs.

Announcement

Release 1.4.1

This release is not a critical update, unless you are directly impacted by the bug fixes, you don't need to update, and you can wait for future releases before upgrading.

Change

Important: This release requires different runtime versions for Terraform to use the latest Terraform Google Cloud providers and solve multiple deployment bugs. For more information, see the deployment guide.

Change

Helm charts - MDE services containers image tag: 3f285a5. - MDE UI image tag: 2.22.0-dev.0.123.4.

Change

Helm charts - MDE services containers image tag: 3f285a5. - MDE UI image tag: 2.22.0-dev.0.123.4.

Libraries

Release signature

0b539af
1.4.1
0e06a47adf3c1f7efedfcb182235b490

September 05, 2024

Libraries

Release signature

7176648
1.4.0
cf0b7439bf9862b06aedcea13dd783fe

Security

The vulnerability scanner may flag a critical CVE related to the zlib libraries in the base OS of the container images. The binary of this impacted library is not built in the MDE container images, and it is not impacted by this vulnerability. Nonetheless, once a new Debian OS patch is available, MDE will release an updated container images so that the vulnerability scanner does not flag this issue. For more information about the vulnerability, see the Debian tracker.

Fixed

• Fix (324563314): Enabling the Dataflow Streaming Engine for the Cloud Storage flows.
• Fix (327369559): Listing the parsers endpoint does not return the disabled parsers.
• Fix (320427422): Upgrading the Spring stack to 3.x.
• Fix (353240181): Security and performance fixes.

Change

Helm charts

• MDE services containers image tag: a04af0d.
• MDE UI image tag: 2.22.0-dev.0.123.4.

Libraries

Release signature

7176648
1.4.0
cf0b7439bf9862b06aedcea13dd783fe

Change

• Improvement (330297418): Enable the three-projects topology for bigquery-sink: the service to be deployed in a project, the Pub/Sub in another project, and the BigQuery datasets in a third project. Find these as dedicated properties in the BigQuery sink helm chart.
• Improvement (320427338): Creating automatic BigQuery views based on the type buckets schema. This provides a stable column-based schema for accessing the types as well as its metadata in a single view. For more information, see Analyticis view.
• Improvement (320427684): Types data field schema validation. The non-numeric types can have a JSON schema that governs the data field of its records, and this schema is now enforced during the processing. For more information, see Data field.
• Improvement (320427624): New real-time stats service for the ingested tags. For more information, see Real-Time service.
• Improvement (324029584): Automatic deletion for excessive metadata instances to prevent the system performance degradation. For more information, see Metadata automatic deletion.
• Improvement (353244779): New Notification API endpoint to display time-sensitive messages to the user such as the metadata instances approach the maximum size limit.
• Improvement (353247809): New API to stop and start the processing of messages.

February 29, 2024

Change

Helm charts

• MDE services containers image tag: 09497d7.
• MDE UI image tag: 2.15.0.0.109.2.

Announcement

Version 1.3.8

This release is not a critical update, unless you are directly impacted by the bug fixes, you don't need to update, and you can wait for future releases before upgrading.

Important: This release introduces a limitation on the execution time of parsers. If the parser execution time exceeds seven seconds, it will be preempted and an error message will be produced in the operations log. If the parser needs more than seven seconds, we recommend to break it into multiple parsers and attach it to the same message class.

Fixed

• Fix (324540732): Adding JVM initialization flags to increase the memory utilisation per pod.
• Fix (324040010): Configuration Manager running out of memory when running a misconfigured parser that impacts the CPU and memory allocations.
• Fix (321911849): Message Mapper running out of memory when running a misconfigured parser that impacts the CPU and memory allocations.
• Fix (322148462): Fixing web interface error when interacting with items in a paginated response with size of more than 100 items.
• Fix (327323565): Cloud Storage Writer out of memory error when ingesting large number of items.
• Fix (328190564): Cloud Storage Reader out of memory error when reading a large number of files.
• Fix (327369559): Listing parsers in the web interface does not show the disabled parsers.

Libraries

Release signature

a94aa0e
1.3.8
e74afda073cd5d52a591763c5ea7cbe1

Change

• Improvement (324563314): Enabling the Dataflow Streaming Engine for the Cloud Storage Flows to improve the scalability behaviour of the Cloud Storage jobs.

January 17, 2024

Breaking

The following Dataflow jobs need to be manually stopped before upgrading:

• eventchange-dataflow-flex-job
• gcs-writer-dataflow-flex-job
• window-dataflow-flex-job
• bigtable-writer-dataflow-flex-job
• gcs-reader-dataflow-flex-job

Change

Helm charts

• MDE services containers image tag: 4b79010.
• MDE UI image tag: 2.15.0.0.109.2.

Change

• Improvement (320427011): All the services now use full names for Pub/Sub subscriptions and topics (For example: projects/PROJECT_ID/subscriptions/configuration-processing-subscription). This allows having the topics or subscriptions in a different project that the one the services are deployed to.
• Improvement (320443808): All dataflow jobs now use private IP by default. For more information, see Routes Firewall.

Libraries

Release signature

7460a29
1.3.7
3ed1b9c3886f123e0d839126a1577c63

Libraries

Library updates

• Updated Apache Beam SDK to version 2.53.0.

Announcement

Version 1.3.7

This release is not a critical update, unless you are directly impacted by the bug fixes, you don't need to update, and you can wait for future releases before upgrading.

December 19, 2023

Announcement

Version 1.3.6

This release is not a critical update, unless you are directly impacted by the bug fixes, you don't need to update, and you can wait for future releases before upgrading.

Libraries

Release signature

1b11952
1.3.6
bfaaf15fad73e78565006891342dbd08

Change

Helm charts

• MDE services containers image tag: 8199764.
• MDE UI image tag: 2.15.0.0.109.2.

Breaking

The following Dataflow jobs need to be manually stopped before upgrading:

• eventchange-dataflow-flex-job
• gcs-writer-dataflow-flex-job
• window-dataflow-flex-job
• bigtable-writer-dataflow-flex-job
• gcs-reader-dataflow-flex-job

Fixed

• Fix (315935780): Pub/Sub attributes on input-messages are filtered and can't be used in the source message class or parser.

Change

• Improvement (311599436): Added a new field to the output of the Group Event Change Transformation to include the message that triggered the change in the transformation
• Improvement (316554360): When parsing a message, add the output of the headers field to the Pub/Sub headers and preserve through the entire pipeline.

Breaking

The following Dataflow jobs need to be manually stopped before upgrading:

• eventchange-dataflow-flex-job
• gcs-writer-dataflow-flex-job
• window-dataflow-flex-job
• bigtable-writer-dataflow-flex-job
• gcs-reader-dataflow-flex-job

November 09, 2023

Change

The following Dataflow jobs need to be stopped before upgrading:

• eventchange-dataflow-flex-job
• gcs-writer-dataflow-flex-job
• window-dataflow-flex-job

Announcement

Version 1.3.5

This release is not a critical update, unless you are directly impacted by the bug fixes, you don't need to update, and you can wait for future releases before upgrading.

Change

Helm charts

• MDE services containers image tag: 7a994ee.
• MDE UI image tag: 2.15.0.0.109.2.

Libraries

Release signature

8584357
1.3.5
db04eff8fb5f0d2267d1653498480ec7

September 29, 2023

Announcement

Version 1.3.4

This release is not a critical update, unless you are directly impacted by the bug fixes, you don't need to update, and you can wait for future releases before upgrading.

Change

• Improvement (301217870): Improved validation messages in the parser test API to better indicate how the input messages timestamps and extra fields are handled.
• Improvement (300220191): Enabling more special characters in the tag name to allow ., $, !, and *.
• Improvement (302490275): The mde::sanitizeTagName() Whistle function is now supporting the allowed special characters in the tag name.
• Multiple documentation updates including specifying the recommended number of types for the small size deployments.

Note: Only GKE services are affected by this release.

Fixed

• Fix (301210983): Materialize Metadata flag on the sinks only works when the BigQuery sink Materialize Metadata flag is enabled.
• Fix (295150919): Multiple security vulnerabilities fixes.
• Fix (294996056): Better service logging and removing unnecessary errors to enhance the troubleshooting experience.
• Fix (300863283): Added missing sid header in the Federation API data fields.

Change

Helm charts

• MDE services containers image tag: e29b2aa.
• MDE UI image tag: "2.15.0.0.104.0-noauth.

Change

• Improvement (301217870): Improved validation messages in the parser test API to better indicate how the input messages timestamps and extra fields are handled.
• Improvement (300220191): Enabling more special characters in the tag name to allow ., $, !, and *.
• Improvement (302490275): The mde::sanitizeTagName() Whistle function is now supporting the allowed special characters in the tag name.
• Multiple documentation updates including specifying the recommended number of types for the small size deployments.

Note: Only GKE services are affected by this release.

Fixed

• Fix (301210983): Materialize Metadata flag on the sinks only works when the BigQuery sink Materialize Metadata flag is enabled.
• Fix (295150919): Multiple security vulnerabilities fixes.
• Fix (294996056): Better service logging and removing unnecessary errors to enhance the troubleshooting experience.
• Fix (300863283): Added missing sid header in the Federation API data fields.

Change

Infrastructure changes: MDE version variable mde_version updated from 1.3.3 to 1.3.4:

    tf
    variable "mde_version" {
      type    = string
      default = "1.3.4"
    }

Change

Helm charts

• MDE services containers image tag: e29b2aa.
• MDE UI image tag: "2.15.0.0.104.0-noauth.

Announcement

Version 1.3.4

This release is not a critical update, unless you are directly impacted by the bug fixes, you don't need to update, and you can wait for future releases before upgrading.

September 15, 2023

Change

Helm charts

• MDE services containers image tag: cdcb206.
• MDE UI image tag: 2.13.2.0.99.7-noauth (same as 1.3.2).

Change

Infrastructure changes: MDE version variable mde_version updated from 1.3.2 to 1.3.3.

    tf
    variable "mde_version" {
      type    = string
      default = "1.3.3"
    }

Change

Infrastructure changes: MDE version variable mde_version updated from 1.3.2 to 1.3.3.

    tf
    variable "mde_version" {
      type    = string
      default = "1.3.3"
    }

Libraries

Release signature

832ef19
1.3.3
48234a49942b0c6031acd67c236ea022

Announcement

Version 1.3.3

This release is not a critical update, unless you are directly impacted by the bug fixes, you don't need to update, and you can wait for future releases before upgrading.

Issue

• Issue (294526878): When updating metadata instances in the UI on a Tag it is not possible to add empty string "" as a value.
• Workaround: Use the API if you need to insert an empty string as a value for a metadata instance attribute.

Change

• Fix (300404657): Malformed input to ops-log not stored in BigQuery due serialization issues.
• Fix (300030725): Bucket references and instances should be optional when parsing messages.

Note: Only GKE services are affected by this release.

September 11, 2023

Announcement

Version 1.3.2

This release is a critical update.

Feature

• MDE UI can now test for a matching message class on the whole list of message classes (like in v1.2) rather than just the individual message class.

Change

Helm charts

• MDE services containers image tag: 3bf1635.
• MDE UI image tag: 2.13.2.0.99.7-noauth.

Change

Helm charts

• MDE services containers image tag: 3bf1635.
• MDE UI image tag: 2.13.2.0.99.7-noauth.

Change

Infrastructure changes

• You must update the file: deployment/terraform/modules/deployment/dataflow-gcs-reader.tf if you have modified the Terraform. The specific change is related to the inputSubscription.
• MDE version variable mde_version updated from 1.3.1 to 1.3.2.
• All Pub/Sub topics no longer have any expiration time set See the Terraform for details. If you have your own Terraform you should also remove the expiration from all relevant topics.
• As a result of Fix (298907994) you may see Terraform plan report that a number of Pub/Sub topics would be created. This only happens if they have expired.

Issue

• Issue (294526878): When updating metadata instances in the UI on a Tag it is not possible to add empty string "" as a value.
  • Workaround: Use the API if you need to insert an empty string as a value for a metadata instance attribute.

Issue

• Issue (294526878): When updating metadata instances in the UI on a Tag it is not possible to add empty string "" as a value.
  • Workaround: Use the API if you need to insert an empty string as a value for a metadata instance attribute.

August 23, 2023

Change

Infrastructure changes

• Terraform IDs are use for reference not the actual names of the resources. For the full set of changes make sure the review the complete terraform code in the package.
• MDE version variable mde_version updated from 1.3.0 to 1.3.1.
• Dead letter topic added to the following subscriptions and topics.
• Pub/Sub subscriptions mentioned earlier have dead letter policy applied.
• GKE lifecycle has the following ignore changes added.

Change

• Fix (294525580): Disable message class doesn't actually disable the message class in the system.
• Fix (294809257, 294995452, 295141703): Missing bucket or natural key references can cause Metadata Manager service to spam the Operations Log tables in BigQuery and potentially overload the ingestion pipeline.
• Fix (294819318): When cloning LookML repository, it could fail.
• Fix (294981315): Helm chart for adding LB to MC update to use the correct resource label.
• Fix (295163432): When deploying MDE in some Google Cloud project, the deployment with terraform apply would recreate the GKE cluster with each apply.
• Fix (295548129): Old validation script from v.1.2.0 removed from package.
• Fix (294807712): MDE web interface had a wrong name in the title.

Change

Helm charts

• Containers image tag: 456bde9.

Breaking

• Fix (294809257, 294995452, 295141703): Introduces breaking changes where the Pub/Sub topics and subscriptions must be destroyed and recreated by Terraform.

Warning: This fix is very important for the stability of MDE and should be applied to all system as soon as possible.

This could cause data loss for any messages that are "in flight" in the processing pipeline. Data already process and stored will not be affected.

Announcement

Version 1.3.1

This release is a critical update.

Issue

• Issue (294526878): When updating metadata instances in the web interface on a Tag it is not possible to add empty string "" as a value.
  • Workaround: Use the API if you need to insert an empty string as a value for a metadata instance attribute.
• Issue (296532636): Pub/Sub Sink doesn't set data for numeric records.
• Issue (297851980): Parsers that use MDE whistle functions, pass an input parameter instead of a String fail to be saved.
  • Workaround: Add null check and return empty string if null for example var santizedTagName: if $root.tagName? then mde::sanitizeTagName($root.tagName) else "";.

August 03, 2023

Change

• Metadata buckets support only a single schema and the schema is a property of the bucket.
• Configuration Manager API has been refactored into configuration and Metadata management API, and the API surface has been simplified to align with Google API design and general REST API best practices.
• BigQuery schema changes:
  • Removed default archetype tables.
  • Removed sfp_data dataset.
  • Improved storage footprint by removing fields from abstract base schemas of types (archetype schemas). See the schema changes in the v1.3 reference.
  • Changed field naming style from camel case to snakecase.
  • Configuration model concepts have been refined and renamed.
  • Simplified proto record schemas by removing redundant fields
  • AVRO files containing processed data stored in the Cloud Storage data sink are stored by type.

Feature

New features

• Added a standalone UI for Manufacturing Data Engine for deployment without Manufacturing Connect.
• Introduced a new archetype family - clustered - to enable user-defined data clustering logic in BigQuery and row-key generation in Bigtable for the Federation API.
• Introduced a Pub/Sub sink to allow for extensibility and custom integration. Support for messages in both JSON and ProtoBuf.
• Extended the BigQuery schema with new fields to simplify and better support analytics use-cases. See the schema changes in the v1.3 BigQuery Schemas documentation.
• Introduced new datasets in BigQuery: mde_data, mde_dimension, and mde_system to improve data table governance.
• Configuration data is now exported to dedicated tables in the mde_system dataset.
• Introduced type versioning.
• Introduced Lookup buckets to enable retrieving reference data in the parser.
• Introduced type - metadata bucket associations and improved metadata validation by providing the ability to require record contextualization with specific bucket metadata.
• Added a new identifier (natural key) to metadata instances to enable retrieval of instances by a business-friendly key, such as the tag name, allowing users to manage shared instances centrally.
• Introduced new custom Whistle functions to look up metadata instances in parsers by natural key.
• Introduced shared cache using Redis.
• Introduced more granular deployment sizes to support varied deployment scenarios.
• Created a MDE LookML library to distribute common MDE LookML components and configuration packages.

Announcement

Version 1.3.0

This release is a critical update.

Issue

• Issue (294526878): When updating metadata instances in the web interface on a Tag it is not possible to add empty string "" as a value.
  • Workaround: Use the API if you need to insert an empty string as a value for a metadata instance attribute.
• Issue (294525580): Disable message class does not actually disable the message class in the system.
  • Workaround: Delete the message class instead using the API.
• Issue (294809257): Missing bucket or natural key references can cause Metadata Manager service to spam the Operations Log tables in BigQuery.
  • Take great care when working with parsers to create the bucket beforehand and double check log for any errors when having deployed a new parser.

Change

• Refined the metadata bucket configuration entity by distinguishing between Tag and Record buckets and allow for data contextualization at different aggregation levels.
• Improved query experience and performance in BigQuery by normalizing metadata, allowing efficient retrieval of metadata without needing to unnest records, and enabling new query patterns.
• Improved support for Bigtable sink, allowing all archetypes to be stored in Bigtable.
• Improved query performance by associating each type with a dedicated table in BigQuery to better take advantage of table partitioning.
• Tag names are now global, and parsers targeting different types can emit records referencing the same tag name.
• Improved Federation API surface that enables new data access patterns for all archetypes.
• The number of required fields in a parser has been reduced.
• Improved logging and introduced the ability to set logging levels.
• Improved observability and monitoring by exposing new Cloud Monitoring metrics.
• Internal performance improvements.
• Improved documentation, deployment and configuration guides, as well as improved access and distribution to documentation content.
• Reduced the cost by 20% for smaller instances of MDE by migrating key data processing workloads from Dataflow to GKE.
• Improved the default configuration package by providing better integration with MCe and enabling the ingestion of any data stream with minimal structural requirements. The configuration package now supports the ingestion of data arrays and complex value objects from MCe.
• Upgraded to Whistle v1.4.0.
• Improved transformation model: added the capability to apply several versions of the same transformation to the same Type.