Create a deployment with TLS encryption on VMs

This document describes how to add TLS encryption to a Spanner Omni deployment on virtual machines (VMs). A deployment with network security features uses Transport Layer Security (TLS) 1.3 to encrypt and authenticate communication within the deployment and with its clients. Spanner Omni provides mutual TLS (mTLS) for enhanced security, where both parties establish authenticity before exchanging data. mTLS is optional between the client and the server, but Spanner Omni servers communicate with each other over mTLS.

Before you begin

Before you begin, ensure your environment meets the following requirements:

  • Make sure you have SSH access to each machine in the deployment. This access lets you download and run the Spanner Omni binary.

  • Your network must allow TCP communication on ports 15000 through 15030.

  • Each machine must have sufficient storage to host the data the deployment handles.

  • Review the System requirements page to ensure your setup meets the requirements.

  • If you run the binaries on the vSphere virtualization platform, disable TSC virtualization. To do this, add the monitor_control.virtual_rdtsc = FALSE setting to the virtual machine's .vmx configuration file.

Step 1: Create a deployment without TLS encryption

Follow the steps in Create a Spanner Omni VM deployment without encryption. Verify that your VM deployment without encryption and security features works properly. This page assumes that you have created a regional deployment with three zones.

Step 2: Generate the certificates

You need to create three sets of certificates:

Certificate type Description
API certificates API certificates help protect the Spanner API server.
Server certificates Server certificates help protect inter-server communication.
Client certificates End users or applications use client certificates to establish their identity and trust with Spanner Omni servers.

A Certification Authority (CA) issues these certificates. Spanner Omni provides tools to create a CA and all three types of certificates. Perform the following steps on one of your machines.

You can create these certificates on your workstation using the Spanner Omni CLI and then transfer the certificate files to each Spanner Omni server. For more information, see the Quickstart using the Spanner Omni CLI.

To generate certificates, you must complete the following steps:

Create a Certification Authority (CA)

This authority is the root CA for all client and server certificates you generate in the following steps.

spanner certificates create-ca --ca-certificate-directory=certs

The certs directory contains the CA certificate. Create a copy of this certificate to use as a CA for API certificates.

cp certs/ca.crt certs/ca-api.crt

The directory $HOME/.spanner/private-keys contains the private key for the CA. Back up and secure this directory. A user with access to the private key can sign arbitrary certificates that clients trusting the self-signed CA trust. Although you can use the same CA for all certificates, it is mandatory that API certificates and client certificates use the same CA. Optionally, you can create an additional CA (or use an externally trusted CA) for the API certificates. Make sure you use the right CA in the following steps while creating certificates. This document uses the same CA for all certificate types.

Generate server certificates

You generate two types of server certificates:

This setup provides more flexible management of these certificates, like a certificate rotation.

Create the Spanner server certificate

Spanner Omni servers use server certificates to encrypt communication with each other (inter-server communication).

Create the server certificate by running the following. Replace SERVER_LIST with a comma separated list of names of Spanner servers or suffixes.

SERVER_NAMES=SERVER_LIST
spanner certificates create-server --hostnames=${SERVER_NAMES} --ca-certificate-directory certs --output-directory certs

This command creates two files, server.crt and server.key, in the certs directory.

Create the API certificate

API certificates encrypt communication from systems interacting with the deployment. Using separate certificates for the API and inter-server communication lets you manage and rotate each type independently.

Create the API certificate by running the following. Replace LB_DNS with the DNS of the load balancer.

SERVER_NAMES=LB_DNS
spanner certificates create-server --filename-prefix=api --hostnames=${SERVER_NAMES} --ca-certificate-directory certs --output-directory certs

This command creates two more files, api.crt and api.key, in the certs directory. If required, you can use an externally trusted CA for the API certificates.

Distribute the certificates to all servers

Copy the certs directory to all other servers in the deployment to start them with network security features.

scp -r certs REMOTE_HOST:SPANNER_DIR/certs

Step 3: Generate client certificates

You can use client certificates to authenticate users and applications in Spanner. Client certificates enable mTLS between the client and server.

Client certificates must be signed by the same CA as the API certificate and must contain a username for authorization. This example uses the admin user, which is the default user for each database. For more information about users, roles, and authentication options, see Authentication and authorization in Spanner Omni.

USERNAME=admin
spanner certificates create-client $USERNAME --output-directory clientcerts --ca-certificate-directory certs

This command creates client.crt and client.key files in the clientcerts directory. Send these files to any machine that connects to servers of the deployment.

If you plan to use the client certificates with the Java client library, you must generate the certificate key in PKCS#8 format. Use the following command:

USERNAME=admin
spanner certificates create-client $USERNAME \
    --output-directory clientcerts \
    --ca-certificate-directory certs \
    --generate-pkcs8-key

Step 4: Start the servers

After you generate the certificates and copy them to all servers in your deployment, start each server. Create a base directory on each server to store data, metadata, and logs. If a server needs to restart, specify the same directory for continuity.

Single server deployment

To start a single-server deployment with encryption, create a password file for the initial admin user and start the server with your supported authentication methods and certificates.

  1. Create the password file with permissions set to 600 so that it's readable by only its owner:

    printf '%s' 'PASSWORD' > admin_password.txt
chmod 600 admin_password.txt

    The password must meet the following requirements:

    • Between 8 and 32 characters in length.
    • At least one uppercase character.
    • At least one lowercase character.
    • At least one number.
    • At least one special character.

  2. Start the single server with TLS and authentication configured:

    nohup spanner start-single-server \
    --base-dir=BASE_DIR \
    --certificate-directory=${HOME}/.spanner/certs \
    --auth-methods=password,client-certificate \
    --password-protocol=opaque \
    --initial-admin-password-file=admin_password.txt &

    On initial startup, spanner start-single-server sets the supported authentication methods and password protocol, and creates the initial adminuser with theroles/spanner.admin` role using the specified password file.

For information about interacting with your deployment, see Step 7: Interact with the deployment.

Scale-out deployment

For scale-out deployments, start the server on each machine. The values for server-address and zone must match the values in the deployment configuration. The network must resolve server-address. Servers use server-address for internal communication. Run the following to start the root server:

nohup spanner start \
    --root \
    --server-address=HOST_NAME \
    --zone=ZONE_NAME \
    --base-dir=BASE_DIR \
    --certificate-directory=${HOME}/.spanner/certs &

The following command shows an example with specific values:

nohup spanner start \
    --root \
    --server-address=rootserver1 \
    --zone=us-central-1a \
    --base-dir=./spanbasedir \
    --certificate-directory=${HOME}/.spanner/certs &

With the servers now running on each machine, you are ready to create the deployment.

Step 5: Create a deployment with TLS encryption

To create the deployment, follow these steps:

  1. Create a password file containing the password for the default admin user with its permissions set to 600 (readable by only its owner):

    printf '%s' 'PASSWORD' > admin_password.txt
chmod 600 admin_password.txt

    The password must meet the following requirements:

    • Between 8 and 32 characters in length.
    • At least one uppercase character.
    • At least one lowercase character.
    • At least one number.
    • At least one special character.

  2. Ensure that deployment.yaml includes your selected authentication methods and password protocol:

    deployment_settings:
  security_settings:
    authentication_methods:
      - AUTHENTICATION_METHOD_PASSWORD
      - AUTHENTICATION_METHOD_CLIENT_CERTIFICATE
    password_authentication_protocol: PASSWORD_AUTHENTICATION_PROTOCOL_OPAQUE

    You can specify one or more of the following authentication methods:

    • AUTHENTICATION_METHOD_PASSWORD: password authentication
    • AUTHENTICATION_METHOD_CLIENT_CERTIFICATE: client certificate authentication

    You can specify the following password protocol:

    • PASSWORD_AUTHENTICATION_PROTOCOL_OPAQUE: OPAQUE protocol

  3. Run the spanner deployment create command from one of the root servers to create the deployment. Specify the base directory with the --base-dir flag (using the same BASE_DIR specified when starting the root server), and provide the administrator password file with the --admin-password-file flag:

    spanner deployment create \
    --config-file=deployment.yaml \
    --base-dir=BASE_DIR \
    --admin-password-file=admin_password.txt

The console for each machine shows messages indicating that the deployment now includes TLS encryption. All servers communicate with each other over an encrypted channel.

Step 6: (Optional) Configure a load balancer

To manage and distribute client traffic across the servers in your deployment, set up a load balancer. Ensure that the load balancer configuration for the health check uses HTTPS instead of HTTP. Use the following configuration details:

Parameter Value
Protocol TCP
Backend IP The IP addresses of your servers.
Port 15000 (This is the default port. If you used a different port in the --server-address flag, use that port.)
Health check URL https://IP_ADDRESS:15012/healthz
Balancing strategy roundrobin (distributes requests sequentially across servers)

Step 7: Interact with the deployment

You can interact with your Spanner Omni deployment from any VM using the Spanner Omni CLI.

You must include the following flag with each command to establish an encrypted connection:

  • --ca-certificate-file=certs/ca-api.crt

If you enabled mTLS for clients, also include the following flag with each command:

  • --client-certificate-directory=clientcerts

To sign in and interact with your deployment, follow these steps:

  1. Sign in to Spanner Omni

    spanner auth login admin \
    --ca-certificate-file=certs/ca-api.crt \
    --deployment-endpoint=ENDPOINT

    When prompted, enter the password configured in admin_password.txt.

    Successfully logged in as "admin"

  2. Create a database

    spanner --deployment-endpoint=ENDPOINT databases create mydb --ca-certificate-file=certs/ca-api.crt

    Creating database...done.

  3. Open the SQL Shell

    spanner sql --database=mydb --ca-certificate-file=certs/ca-api.crt

    Connected.
spanner>

  4. Create a table and add data

    spanner> create table names (nameId INT64 NOT NULL, name String(100)) Primary Key (nameId);
Query OK, 0 rows affected (4.62 sec)

spanner> insert names (nameId, name) values (1, "Jack");
Query OK, 1 rows affected (0.18 sec)

  5. Verify the data

    List the databases:

    spanner databases list --ca-certificate-file=certs/ca-api.crt

    NAME  STATE  VERSION_RETENTION_PERIOD  EARLIEST_VERSION_TIME  KMS_KEY_NAME  ENABLE_DROP_PROTECTION
mydb  READY  1h                        2025-02-07T12:25:30Z                 false

    Get the data from the table:

    spanner sql --database=mydb --ca-certificate-file=certs/ca-api.crt

    Connected.
spanner> show tables;
+----------------+
| Tables_in_mydb |
+----------------+
| names          |
+----------------+
1 rows in set (0.14 sec)

spanner> select * from names;
+--------+--------+
| nameId | name   |
+--------+--------+
| 1      | Jack   |
+--------+--------+
1 rows in set (18.69 msecs)

Step 8: (Optional) Scale the deployment

You can add non-root servers to a zone to scale the capacity of the zone. To do this, generate the server certificate for the non-root servers as Step 2: Generate the certificates explains, and start the server with the following command:

spanner start \
    --server-address=NON_ROOT_MACHINE \
    --join-servers=ROOT_SERVER1,ROOT_SERVER2,ROOT_SERVER3 \
    --zone=us-central1-a \
    --base-dir=./spandir \
    --certificate-directory=${HOME}/.spanner/certs

Next steps