<button class="button button-with-icon cloud-select-dropdown__button button-primary" aria-haspopup="true">Version 1.10 (unsupported)</button>

- [Latest](https://docs.cloud.google.com/anthos/clusters/docs/bare-metal/latest/how-to/vm-workloads)
- [Version 1.16](https://docs.cloud.google.com/anthos/clusters/docs/bare-metal/1.16/how-to/vm-workloads)
- [Earlier versions (unsupported)](https://docs.cloud.google.com/anthos/clusters/docs/bare-metal/1.10/getting-support.md.txt#version-support)

<br />

> [!WARNING]
>
> **Preview**
>
>
> This product or feature is
>
> subject to the "Pre-GA Offerings Terms" in the General Service Terms section
> of the [Service Specific
> Terms](https://docs.cloud.google.com/terms/service-terms#1).
>
> Pre-GA products and features are available "as is" and might have limited support.
>
> For more information, see the
> [launch stage descriptions](https://cloud.google.com/products/#product-launch-stages).

This page shows you how to deploy VMs on Google Distributed Cloud using
Anthos VM Runtime. Anthos VM Runtime uses
[KubeVirt](https://kubevirt.io/)
to orchestrate VMs on clusters, allowing you to work with your VM-based apps and
workloads in a uniform development environment. You can enable Anthos VM Runtime
when creating a new cluster and on existing clusters.

> [!NOTE]
> **Note:** Starting with Google Distributed Cloud release 1.10.0, newly created clusters and clusters upgraded to version 1.10.0 or higher have a cluster-scoped `VMRuntime` custom resource. You use the `VMRuntime` custom resource, instead of the cluster configuration file, to [enable](https://docs.cloud.google.com/anthos/clusters/docs/bare-metal/1.10/how-to/vm-workloads.md.txt#enable_anthos_vm_runtime) and [disable](https://docs.cloud.google.com/anthos/clusters/docs/bare-metal/1.10/how-to/vm-workloads.md.txt#disable_anthos_vm_runtime) Anthos VM Runtime. The `VMRuntime` custom resource is provided for hybrid, standalone, and user clusters only.

## Before you begin

These instructions assume that you have a cluster up and running. If you don't,
you can follow the instructions on
[Google Distributed Cloud quickstart](https://docs.cloud.google.com/anthos/clusters/docs/bare-metal/1.10/quickstart.md.txt)
to quickly set up a cluster on your workstation.

## Enable Anthos VM Runtime

Anthos VM Runtime is disabled, by default. To enable Anthos VM Runtime, edit the
`VMRuntime` custom resource in the cluster. Starting with Google Distributed Cloud
release 1.10.0, the `VMRuntime` custom resource is automatically installed on
clusters.

To enable Anthos VM Runtime:

1. Update the `VMRuntime` custom resource to set `enabled` to `true`.

   ```yaml
   apiVersion: vm.cluster.gke.io/v1
   kind: VMRuntime
   metadata:
     name: vmruntime
   spec:
     enabled: true
     # useEmulation default to false if not set.
     useEmulation: true
     # vmImageFormat default to "qcow2" if not set.
     vmImageFormat: qcow2
   ```
2. If your node doesn't support hardware virtualization, or you aren't sure,
   set `useEmulation` to `true`.

   If available, hardware virtualization provides better performance than
   software emulation. The `useEmulation` field defaults to `false`, if it
   isn't specified.

   ```yaml
   apiVersion: vm.cluster.gke.io/v1
   kind: VMRuntime
   metadata:
     name: vmruntime
   spec:
     enabled: true
     # useEmulation default to false if not set.
     useEmulation: true
     # vmImageFormat default to "qcow2" if not set.
     vmImageFormat: qcow2
   ```
3. You can change the image format used for the VMs you create by setting the
   `vmImageFormat` field.

   The `vmImageFormat` field supports two disk image format values: `raw` and
   `qcow2`. If you don't set `vmImageFormat`, the Anthos VM Runtime uses the
   `raw` disk image format to create VMs. The `raw` format may provide improved
   performance over `qcow2`, a copy on write format, but may use more disk.
   For more information about the image formats for your vm, see
   [Disk image file formats](https://www.qemu.org/docs/master/system/images.html#disk-image-file-formats)
   in the QEMU documentation.

   ```yaml
   apiVersion: vm.cluster.gke.io/v1
   kind: VMRuntime
   metadata:
     name: vmruntime
   spec:
     enabled: true
     # useEmulation default to false if not set.
     useEmulation: true
     # vmImageFormat default to "qcow2" if not set.
     vmImageFormat: qcow2
   ```
4. Save the configuration and verify that the `VMRuntime` custom resource is
   enabled:

   ```bash
   kubectl describe vmruntime vmruntime
   ```

   The details of the `VMRuntime` custom resource include a `Status` section.
   Anthos VM Runtime is enabled and working when `VMRuntime.Status.Ready` is set
   to `true`.

### Upgrading 1.9.x clusters

The `VMRuntime` custom resource is automatically installed on clusters upgraded
to version 1.10.0 or higher. When you upgrade a 1.9.x cluster to version 1.10.0
and higher, the upgrade process checks your cluster settings and configures the
`VMRuntime` custom resource to match the Anthos VM Runtime settings on your
1.9.x cluster. If `spec.kubevirt` is present in the 1.9.x cluster resource, the
upgrade process enables Anthos VM Runtime.

The `VMRuntime` custom resource settings take precedence over legacy Anthos VM
Runtime cluster settings, such as `spec.kubevirt.useEmulation`, in version
1.10.0 or higher clusters. Update the `VMRuntime` custom resource to change the
Anthos VM Runtime settings for your 1.10.0 or higher cluster.

## Install `virtctl`

1. Install the `virtctl` CLI tool as a `kubectl` plugin

   ```bash
   export GOOGLE_APPLICATION_CREDENTIALS="bm-gcr.json"
   sudo -E ./bmctl install virtctl
   ```
2. Verify that `virtctl` is installed:

   ```bash
   kubectl plugin list
   ```

   If `virtctl` is listed in the response, it's successfully installed.

## Create a VM

Once you enable KubeVirt on your cluster and install the `virtctl` plugin for
`kubectl`, you can start creating VMs in your cluster using the
`kubectl virt create vm` command. Before running this command, we recommend
configuring a cloud-init file to ensure that you have console access to the VM
once it's created.

### Create a custom cloud-init file for console access

There are two ways that you can create a custom cloud-init file. The easiest way
is to specify the `--os=<OPERATING_SYSTEM>` flag when creating the VM. This
method automatically configures a simple cloud-init file and works for the
following operating systems.

- Ubuntu
- CentOS
- Debian
- Fedora

Once your VM is created, you can access it for the first time with the following
credentials and then change the password:

```yaml
user: root
password: changeme
```

If your image contains a different Linux-based OS or you need a more advanced
configuration, you can manually create a custom cloud-init file and specify the
path to that file by specifying the `--cloud-init-file=<path/to/file>` flag. In
its most basic form, the cloud-init file is a YAML file that contains the
following:

```yaml
#cloud-config
user: root
password: changeme
lock_passwd: false
chpasswd: {expire: false}
disable_root: false
ssh_authorized_keys:
- <ssh-key>
```

For more advanced configurations, see
[Cloud config examples](https://cloudinit.readthedocs.io/en/latest/topics/examples.html).

Once you've determined which method to use, you are ready to create a VM.

### Run the `kubectl virt create vm` command

You can create VMs from public or custom images.

### Public image

If your cluster has external connection, you can create a VM from a public
image by running:

```bash
kubectl virt create vm VM_NAME \
    --boot-disk-access-mode=MODE \
    --boot-disk-size=DISK_SIZE \
    --boot-disk-storage-class="DISK_CLASS" \
    --cloud-init-file=FILE_PATH \
    --cpu=CPU_NUMBER \
    --image=IMAGE_NAME \
    --memory=MEMORY_SIZE
```

Replace the following:

- <var translate="no">VM_NAME</var> with the name of the VM that you want to create.
- <var translate="no">MODE</var> with the access-mode of the boot disk. Possible values are `ReadWriteOnce` (default) or `ReadWriteMany`.
- <var translate="no">DISK_SIZE</var> with the size you want for the boot disk. The default value is `20Gi`.
- <var translate="no">DISK_CLASS</var> with the storage class of the boot disk. The default value is `local-shared`. For a list of available storage classes, run `kubectl get storageclass`.
- <var translate="no">FILE_PATH</var> with the full path of the customized cloud-init file. Depending on the image, this may be required to gain console access to the VM after it is created. If you plan to automatically configure the cloud-init file with the `--os` flag, then don't specify the `--cloud-init-file` flag. If you're specifying the `--cloud-init-file` flag, the `--os` flag is ignored. Acceptable values for `--os` are `ubuntu`, `centos`, `debian`, and `fedora`.
- <var translate="no">CPU_NUMBER</var> with the number of CPUs you want to configure for the VM. The default value is `1`.
- <var translate="no">IMAGE_NAME</var> with the VM image, which can be `ubuntu20.04` (default), `centos8`, or a URL of the image.
- <var translate="no">MEMORY_SIZE</var> with the memory size of the VM. The default value is `4Gi`.

### Custom image

When creating a VM from a custom image, you can either specify an image from
an HTTP image server or a locally stored image.

#### HTTP image server

You can set up an HTTP server using Apache or nginx and upload the custom
image to its exposed folder. You can then create a VM from the custom image by
running:

```bash
kubectl virt create vm VM_NAME \
    --boot-disk-access-mode=DISK_ACCESS_MODE \
    --boot-disk-size=DISK_SIZE \
    --boot-disk-storage-class=DISK_CLASS \
    --cloud-init-file=FILE_PATH \
    --cpu=CPU_NUMBER \
    --image=http://SERVER_IP/IMAGE_NAME \
    --memory=MEMORY_SIZE
```

Replace the following:

- <var translate="no">VM_NAME</var> with the name of the VM you want to create.
- <var translate="no">DISK_ACCESS_MODE</var> with the access-mode of the boot disk. Possible values are `ReadWriteOnce` (default) or `ReadWriteMany`.
- <var translate="no">DISK_SIZE</var> with the size you want for the boot disk. The default value is `20Gi`.
- <var translate="no">DISK_CLASS</var> with the storage class of the boot disk. The default value is `local-shared`. For a list of available storage classes, run `kubectl get storageclass`.
- <var translate="no">FILE_PATH</var> with the full path of the customized cloud-init file. Depending on the image, this may be required to gain console access to the VM after it is created. If you plan to automatically configure the cloud-init file with the `--os` flag, then don't specify the `--cloud-init-file` flag. If you're specifying the `--cloud-init-file` flag, the `--os` flag is ignored. Accectable values for `--os` are `ubuntu`, `centos`, `debian`, and `fedora`.
- <var translate="no">CPU_NUMBER</var> with the number of CPUs you want to configure for the VM. The default value is `1`.
- <var translate="no">SERVER_IP</var> with the IP address of the server hosting the image.
- <var translate="no">IMAGE_NAME</var> with the file name of the custom image.
- <var translate="no">MEMORY_SIZE</var> with the memory size of the VM. The default value is `4Gi`.

#### Locally stored image

You can store the custom image locally and create a VM from it by running:

```bash
kubectl virt create vm VM_NAME \
    --boot-disk-access-mode=DISK_ACCESS_MODE \
    --boot-disk-size=DISK_SIZE \
    --boot-disk-storage-class=DISK_CLASS \
    --cloud-init-file=FILE_PATH \
    --cpu=CPU_NUMBER \
    --image=IMAGE_PATH \
    --memory=MEMORY_SIZE \
```

Replace the following:

- <var translate="no">VM_NAME</var> with the name of the VM you want to create.
- <var translate="no">DISK_ACCESS_MODE</var> with the access-mode of the boot disk. Possible values are `ReadWriteOnce` (default) or `ReadWriteMany`.
- <var translate="no">DISK_SIZE</var> with the size you want for the boot disk. The default value is `20Gi`.
- <var translate="no">DISK_CLASS</var> with the storage class of the boot disk. The default value is `local-shared`.
- <var translate="no">FILE_PATH</var> with the full path of the customized cloud-init file. Depending on the image, this may be required to gain console access to the VM after it is created. If you plan to automatically configure the cloud-init file with the `--os` flag, then don't specify the `--cloud-init-file` flag. If you're specifying the `--cloud-init-file` flag, the `--os` flag is ignored. Accectable values for `--os` are `ubuntu`, `centos`, `debian`, and `fedora`.
- <var translate="no">CPU_NUMBER</var> with the number of CPUs you want to configure for the VM. The default value is `1`.
- <var translate="no">IMAGE_PATH</var> with the local file path to the custom image.
- <var translate="no">MEMORY_SIZE</var> with the memory size of the VM. The default value is `4Gi`.

#### Change the default values for flags

The `kubectl virt create vm` command uses default values to auto-fill
unspecified flags when the command is executed. You can change these default
values by running:

```bash
kubectl virt config default FLAG
```

Replace <var translate="no">FLAG</var> with the flag of the parameter that you want to change the
default value for.

**Example** : The following command changes the default CPU configuration from
the initial default of `1` to `2`:

```bash
kubectl virt config default --cpu=2
```

For a list of supported flags and their current default values, run:

```bash
kubectl virt config default -h
```

The default configurations are stored client side as a local file called
`~/.virtctl.default`. You can change the default configurations by editing
this file as well.

## Access your VM

You can access VMs using the following methods:

### Console access

To access a VM from the console, run:

```bash
kubectl virt console VM_NAME
```

Replace <var translate="no">VM_NAME</var> with the name of the VM that you want to access.

### VNC access

To access a VM using VNC, run:

```bash
# This requires remote-viewer from the virt-viewer package and a graphical desktop from where you run virtctl
kubectl virt vnc VM_NAME
```

Replace <var translate="no">VM_NAME</var> with the name of the VM that you want to access.

### Internal access

The IP addresses of your cluster VMs can be directly accessed by all other
pods in the cluster. To find the IP address of a VM, run:

```bash
kubectl get vmi VM_NAME
```

Replace <var translate="no">VM_NAME</var> with the name of the VM that you want to access.

The command returns something like:

```bash
NAME     AGE   PHASE     IP              NODENAME
vm1      13m   Running   192.168.1.194   upgi-bm002
```

### External access

VMs created in your cluster have pod network addresses that can be accessed
only from within the cluster. To access cluster VMs externally:

1. Expose the VM as a load balancer service:

   ```bash
   kubectl virt expose vm VM_NAME \
       --port=LB_PORT \
       --target-port=VM_PORT \
       --type=LoadBalancer \
       --name=SERVICE_NAME
   ```

   Replace the following:
   - <var translate="no">VM_NAME</var> with the name of the VM that you want to access.
   - <var translate="no">LB_PORT</var> with the port of the load balancer service that is exposed.
   - <var translate="no">VM_PORT</var> with the port on the VM that you want to access through the load balancer service.
   - <var translate="no">SERVICE_NAME</var> with the name you want to give to this load balancer service.
2. Get the external IP address of the load balancer service:

   ```bash
   kubectl get svc SERVICE_NAME
   ```

   Replace <var translate="no">SERVICE_NAME</var> with the name of the load balancer service that
   exposes the VM.

   You can access the target-port of the VM through the IP address listed in
   the `EXTERNAL-IP` field of the response.

#### Example

If you have a VM named `galaxy` that you wanted to access it from outside the
cluster using SSH, you would run:

```bash
kubectl virt expose vm galaxy \
   --port=25022 \
   --target-port=22 \
   --type=LoadBalancer \
   --name=galaxy-ssh
```

Then get the load balancer IP address:

```bash
kubectl get svc galaxy-ssh
```

The command returns something like:

```bash
NAME        TYPE          CLUSTER-IP     EXTERNAL-IP   PORT(S)           AGE
galaxy-ssh  LoadBalancer  10.96.250.76   21.1.38.202   25000:30499/TCP   4m40s
```

Now you can access the VM using SSH through `21.1.38.202:25022` (VIP:port)
from outside the cluster:

```bash
ssh root@21.1.38.202:22 -p 25022
```

## Inspect VM telemetry and console logs

VM telemetry and console logs have been integrated into Google Cloud console.
Telemetry information and log data are critical to monitoring the status of your
VMs and troubleshooting any problems with your cluster VMs.

### VM telemetry

The **Anthos clusters VM status** dashboard provides live telemetry data for
your cluster VMs.

To view telemetry information for your cluster VMs:

1. In the Google Cloud console, select **Monitoring**, or click the following
   button:

   [Go to Monitoring](https://console.cloud.google.com/monitoring)
2. Select **Dashboards**.

   ![Anthos cluster VM status dashboard in the Monitoring dashboards list](https://docs.cloud.google.com/static/anthos/clusters/docs/bare-metal/1.10/images/vm-status-menu.png)
3. Click **Anthos clusters VM status** in the **All Dashboards** list.

   ![Anthos cluster VM status details](https://docs.cloud.google.com/static/anthos/clusters/docs/bare-metal/1.10/images/vm-status-details.png)

### VM console logs

VM serial console logs are streamed to Cloud Logging and can be viewed in
Logs Explorer.

![Logs Explorer showing Anthos cluster VM data](https://docs.cloud.google.com/static/anthos/clusters/docs/bare-metal/1.10/images/console-logs.png)

## Delete VMs and their resources

### Deleting only the VM

```bash
kubectl virt delete vm VM_NAME
```

Replace <var translate="no">VM_NAME</var> with the name of the VM that you want to delete.

### Deleting only VM disks

```bash
kubectl virt delete disk DISK_NAME
```

Replace <var translate="no">DISK_NAME</var> with the name of the disk that you want to delete. If
you try to delete a VM disk before deleting the VM, the disk is marked for
deletion pending the deletion of the VM.

### Deleting VM \& resources

```bash
kubectl virt delete vm VM_NAME --all
```

Replace <var translate="no">VM_NAME</var> with the name of the VM that you want to delete.

If you want to check the resources used by the VM that would be deleted, you
can specify the `--dry-run` flag together with `--all`.

## Disable Anthos VM Runtime

Before you can disable Anthos VM Runtime in a cluster, you must ensure that all
VMs in that cluster have been deleted. Disabling Anthos VM Runtime removes all
KubeVirt related deployments, such as pods and services in the KubeVirt and CDI
namespaces.

1. Check for any existing VMs in the cluster:

   ```bash
   kubectl get vm
   ```

   If the command shows that there are still VMs in your cluster, then you must
   [delete](https://docs.cloud.google.com/anthos/clusters/docs/bare-metal/1.10/how-to/vm-workloads.md.txt#deleting_vms_and_its_resources)
   them before proceeding.
2. Update the `VMRuntime` custom resource to set `enabled` to `false`.

   ```yaml
   apiVersion: vm.cluster.gke.io/v1
   kind: VMRuntime
   metadata:
     name: vmruntime
   spec:
     enabled: false
     # useEmulation default to false if not set.
     useEmulation: true
     # vmImageFormat default to "qcow2" if not set.
     vmImageFormat: qcow2
   ```
3. Save the configuration and verify that the `VMRuntime` custom resource is
   disabled:

   ```bash
   kubectl describe vmruntime vmruntime
   ```

   The details of the `VMRuntime` custom resource include a `Status` section.
   Anthos VM Runtime is enabled and working when `VMRuntime.Status.Ready` is set
   to `false`.

## What's next

- Learn about Anthos VM Runtime [known issues](https://docs.cloud.google.com/anthos/clusters/docs/bare-metal/1.10/troubleshooting/known-issues.md.txt#anthos_vm_runtime).
- See the [KubeVirt user guide](https://kubevirt.io/user-guide/).