Dataflow shared responsibility

Security is a shared responsibility. Dataflow secures the scalable infrastructure that you use to run your Dataflow pipelines and provides you tools and security controls to protect your data, code, and models. While not an exhaustive list, this document lists the responsibilities for both Google and the customer.

Google's responsibilities

  • Protect the infrastructure: Google is responsible for providing secure infrastructure for its services, including physical security of data centers, network security, and application security.

  • Secure the platform: Google is responsible for securing its platform, including managing access controls, monitoring for security incidents, and responding to security events. Google also provides customers with tools to manage their own security settings and configurations.

  • Maintain compliance: Google maintains compliance with relevant data protection laws and regulations. Learn more about Google Cloud compliance.

  • Harden and patch images: Google hardens and patches the operating system of base images used by the Dataflow-owned images. Google promptly makes any patches to these images available. Security bulletins are provided for known vulnerabilities.

    For open source dependencies like Apache Beam, Dataflow provides base images and updates the Apache Beam SDK on a best-effort basis as a convenience. Because Dataflow relies on the open source community for these fixes, Google doesn't have full control over fix prioritization or timing. Fixes for Common Vulnerabilities and Exposures (CVEs) are typically included in upcoming Apache Beam releases.

Customer's responsibilities

  • Use and update your environment to the latest versions of Dataflow containers, VM images, and SDKs: Dataflow provides prebuilt containers and VM images to simplify the use of its services. Google creates new versions of these images when vulnerabilities are identified.

    Your responsibilities include:

    • Monitoring for security bulletins and updating your environment promptly when new versions are available.
    • Restarting long-running streaming jobs to pick up the latest VM image patches. Batch jobs automatically use the latest image version each time they start. To use the latest VMs for streaming jobs, restart them by updating the job. For more information, see Upgrade and patch Dataflow VMs. To manage security issues responsively, we recommend that you use custom container images. If you have strict security Service Level Agreement (SLA) requirements, such as those for FedRAMP compliance, using custom containers lets you manage your own Apache Beam versions and patch cycles, rather than relying on the Apache Beam open source release schedule.
    • Regularly updating your Apache Beam SDK versions to ensure you have the latest security fixes and features.
    • Scanning and patching custom images to mitigate vulnerabilities if you are using a custom container or a custom template.
    • Using Distroless base images when possible if you are using a Flex Template base image to ensure security and reduce vulnerability risks.

  • Manage access controls: You are responsible for managing access controls to your own data and services. This includes managing user access, authentication, and authorization controls, and securing your own applications and data. Learn more about Dataflow security and permissions.

  • Secure applications: You are responsible for securing your own applications running on Dataflow, including implementing secure coding practices and regularly testing for vulnerabilities.

    Learn more about Customer-managed encryption keys, networks and VPC Service Controls, and permissions best practices.

  • Monitor for security incidents: You are responsible for monitoring your own applications for security incidents, and reporting any incidents to Google as necessary.

What's next