> [!WARNING]
>
> **Preview**
>
>
> This feature is
>
> subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the
> [Service Specific
> Terms](https://docs.cloud.google.com/terms/service-terms#1).
>
> Pre-GA features are available "as is" and might have limited support.
>
> For more information, see the
> [launch stage descriptions](https://cloud.google.com/products/#product-launch-stages).

Select a documentation version: <button class="button button-with-icon cloud-select-dropdown__button button-primary" aria-haspopup="true">18.1.0</button>

- [Current (18.4.0)](https://docs.cloud.google.com/alloydb/omni/containers/current/docs/ai/generate-multimodal-embeddings.md.txt)
- [18.4.0](https://docs.cloud.google.com/alloydb/omni/containers/18.4.0/docs/ai/generate-multimodal-embeddings.md.txt)
- [18.3.0](https://docs.cloud.google.com/alloydb/omni/containers/18.3.0/docs/ai/generate-multimodal-embeddings.md.txt)
- [18.1.0](https://docs.cloud.google.com/alloydb/omni/containers/18.1.0/docs/ai/generate-multimodal-embeddings.md.txt)

<br />

> [!NOTE]
> **Note:** This experimental launch is a Pre-GA offering.

You can generate multimodal embeddings in AlloyDB Omni using the supported
Vertex AI multimodal model, `multimodalembedding@001`.

You can use the Vertex AI multimodal embedding models referred to
in [Supported models](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/embeddings/get-multimodal-embeddings.md.txt#supported-models).

This page assumes that you're familiar with AlloyDB Omni and
[generative AI concepts](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/embeddings/get-multimodal-embeddings.md.txt#supported-models). For more information, see
[What are embeddings](https://docs.cloud.google.com/alloydb/omni/containers/18.1.0/docs/ai/work-with-embeddings.md.txt#what-are-embeddings).

## Before you begin

Before you use multimodal embeddings, do the following:

1. [Configure user access to Vertex AI models](https://docs.cloud.google.com/alloydb/omni/containers/18.1.0/docs/install-with-alloydb-ai.md.txt#with-cloud).
2. Verify that the latest version of `google_ml_integration` is installed.
   1. To check the installed version, run the following
      command:

      ```sql
      SELECT extversion FROM pg_extension WHERE extname = 'google_ml_integration';
      extversion
      ---
      1.5.2
      (1 row)
      ```
   2. If the extension isn't installed or if the installed version is
      earlier than 1.5.2, update the extension.

      ```sql
      CREATE EXTENSION IF NOT EXISTS google_ml_integration;
      ALTER EXTENSION google_ml_integration UPDATE;
      ```

      If you experience issues when you run the preceding commands, or if the
      extension isn't updated to version 1.5.2 after you run the preceding
      commands, contact [Google Cloud support](https://cloud.google.com/support).
3. To use the AlloyDB AI query engine functionality, set the
   `google_ml_integration.enable_ai_query_engine` flag to `on`.

   1. Use a text editor to set the following flag in the `postgresql.conf` configuration file for your installation of AlloyDB Omni:

      ```
      google_ml_integration.enable_ai_query_engine = on
      ```
   2. After you save the `postgresql.conf` file, restart the AlloyDB Omni service for the changes to take effect.

      ```
      docker restart CONTAINER_NAME
      ```
4. [Access data in Cloud Storage to generate multimodal embeddings](https://docs.cloud.google.com/alloydb/omni/containers/18.1.0/docs/ai/generate-multimodal-embeddings.md.txt#access-data-in-storage).

### Access data in Cloud Storage to generate multimodal embeddings

- To generate multimodal embeddings, refer to content in Cloud Storage using a `gs://` URI.
- Access Cloud Storage content through your current project's Vertex AI service agent. By default, the Vertex AI service agent already has permission to access the bucket in the same project. For more information, see [IAM roles and permissions index](https://docs.cloud.google.com/iam/docs/roles-permissions/aiplatform.md.txt#aiplatform.serviceAgent).
- To access data in a Cloud Storage bucket in another Google Cloud project,
  run the following gcloud CLI command to grant the [Storage Object Viewer role (`roles/storage.objectViewer`)](https://docs.cloud.google.com/storage/docs/access-control/iam-roles.md.txt) to the Vertex AI service agent of your
  Google Cloud project.

  ```bash
  gcloud projects add-iam-policy-binding <ANOTHER_PROJECT_ID> \
  --member="serviceAccount:service-<PROJECT_ID>@gcp-sa-aiplatform.iam.gserviceaccount.com" \
  --role="roles/storage.objectViewer"
  ```

  For more information, see [Set and manage IAM policies on buckets](https://docs.cloud.google.com/storage/docs/access-control/using-iam-permissions.md.txt).

To generate multimodal embeddings, select one of the following schemas.
<button value="ai" default="">ai schema</button> <button value="google_ml">google_ml schema</button>

## Generate multimodal embeddings

To generate text embeddings for a `multimodalembedding@001` model endpoint, run the following statement:

```sql
SELECT
  ai.text_embedding(
    model_id => 'multimodalembedding@001',
    content => 'TEXT');
```

Replace `TEXT` with the text to generate the embedding for.

To generate image embeddings for a registered `multimodalembedding@001` model endpoint where the image mimetype is default `image/jpeg`, run the following statement:

```sql
SELECT
  ai.image_embedding(
    model_id => 'multimodalembedding@001',
    image => 'IMAGE_PATH_OR_TEXT',
    mimetype => 'MIMETYPE');
```

Replace the following:

- `IMAGE_PATH_OR_TEXT` with the Cloud Storage URI of the image, for example, `gs://my-bucket/embeddings/flowers.jpeg`, or the base64 string of the image.

- `MIMETYPE` with the mimetype of the image, for example, `image/jpeg`. For the full list of supported mimetypes, see the
  [Multimodal embeddings API](https://docs.cloud.google.com/gemini-enterprise-agent-platform/reference/models/multimodal-embeddings-api.md.txt#image).

To generate video embeddings for a registered `multimodalembedding@001` model endpoint, run the following statement:

```sql
SELECT
  ai.video_embedding(
    model_id => 'multimodalembedding@001',
    video => 'VIDEO_URI');
```

Replace `VIDEO_URI` with the Cloud Storage URI of the target video, for example, `gs://my-bucket/embeddings/supermarket-video.mp4`, or the base64 string of the video.
These are two-dimensional arrays that can be accessed as `my_array[0][5]` like syntax.
For more information, see [Arrays](https://www.postgresql.org/docs/17/arrays.html).

## What's next

- [Register a model endpoint with model endpoint management](https://docs.cloud.google.com/alloydb/omni/containers/18.1.0/docs/ai/register-model-endpoint.md.txt).

- [Query using AI powered SQL operators](https://docs.cloud.google.com/alloydb/omni/containers/18.1.0/docs/ai/evaluate-semantic-queries-ai-operators.md.txt).