> [!WARNING]
>
> **Preview**
>
>
> This feature is
>
> subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the
> [Service Specific
> Terms](https://docs.cloud.google.com/terms/service-terms#1).
>
> Pre-GA features are available "as is" and might have limited support.
>
> For more information, see the
> [launch stage descriptions](https://cloud.google.com/products/#product-launch-stages).

> [!NOTE]
> **Note:** This experimental launch is a Pre-GA offering.

Multimodal embeddings in AlloyDB for PostgreSQL let you represent text, images, and
videos in a shared vector space for cross-modal semantic search and
retrieval-augmented generation (RAG). You can generate embeddings for text and
for images or video segments in Cloud Storage by calling a supported
Gemini Enterprise Agent Platform multimodal model---for example, the
`multimodalembedding@001` model---from SQL. For a list of available models, see
[Supported models](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/provisioned-throughput/supported-models.md.txt).

To learn about [generative AI concepts](https://docs.cloud.google.com/docs/generative-ai) and vector
embeddings in AlloyDB, see
[What are embeddings](https://docs.cloud.google.com/alloydb/docs/ai/work-with-embeddings.md.txt#what-are-embeddings).

## Before you begin

Before you use multimodal embeddings, do the following:

- [Verify that the `google_ml_integration` extension is installed](https://docs.cloud.google.com/alloydb/docs/ai/configure-vertex-ai.md.txt#verify-installed-extension).
- [Verify that the `google_ml_integration.enable_model_support` flag is set to `on`](https://docs.cloud.google.com/alloydb/docs/instance-configure-database-flags.md.txt).
- [Integrate with Agent Platform](https://docs.cloud.google.com/alloydb/docs/ai/configure-vertex-ai.md.txt).
- [Access data in Cloud Storage to generate multimodal embeddings](https://docs.cloud.google.com/alloydb/docs/ai/generate-multimodal-embeddings.md.txt#access-data-in-storage).

### Integrate with Agent Platform and install the extension

1. [Configure user access to Agent Platform models](https://docs.cloud.google.com/alloydb/docs/ai/configure-vertex-ai.md.txt).
2. Verify that the latest version of `google_ml_integration` is installed.
   1. To check the installed version, run the following
      command:

      ```sql
      SELECT extversion FROM pg_extension WHERE extname = 'google_ml_integration';
      extversion 
      ---
      1.5.2
      (1 row)
      ```
   2. If the extension isn't installed or if the installed version is
      earlier than 1.5.2, update the extension.

      ```sql
      CREATE EXTENSION IF NOT EXISTS google_ml_integration;
      ALTER EXTENSION google_ml_integration UPDATE;
      ```

      If you experience issues when you run the preceding commands, or if the
      extension isn't updated to version 1.5.2 after you run the preceding
      commands, contact [Google Cloud support](https://cloud.google.com/support).
3. To use the AlloyDB AI query engine functionality, set the
   `google_ml_integration.enable_ai_query_engine` flag to `on`.

   ### SQL

   1. Enable the AI query engine for the current session.  

      ```sql
      SET google_ml_integration.enable_ai_query_engine = on;
      ```
   2. Enable features for a specific database across sessions.  

      ```sql
      ALTER DATABASE DATABASE_NAME SET google_ml_integration.enable_ai_query_engine = 'on';
      ```
   3. Enable the AI query engine for a specific user across sessions and databases.  

      ```sql
      ALTER ROLE postgres SET google_ml_integration.enable_ai_query_engine = 'on';
      ```

   ### Console

   To modify the value of the `google_ml_integration.enable_ai_query_engine` flag, follow the steps
   in [Configure an instance's database flags](https://docs.cloud.google.com/alloydb/docs/instance-configure-database-flags.md.txt#console).

   ### gcloud


   To use the gcloud CLI, you can
   [install and initialize](https://docs.cloud.google.com/sdk/docs/install-sdk.md.txt) the Google Cloud CLI, or you
   can use [Cloud Shell](https://docs.cloud.google.com/shell/docs/using-cloud-shell.md.txt).

   <br />

   You can modify the value of the `google_ml_integration.enable_ai_query_engine` flag.
   For more information, see [Configure an instance's database flags](https://docs.cloud.google.com/alloydb/docs/instance-configure-database-flags.md.txt#console).

   ```console
   gcloud alloydb instances update INSTANCE_ID \
     --database-flags google_ml_integration.enable_ai_query_engine=on \
     --region=REGION_ID \
     --cluster=CLUSTER_ID \
     --project=PROJECT_ID
   ```

### Access data in Cloud Storage to generate multimodal embeddings

- To generate multimodal embeddings, refer to content in Cloud Storage using a `gs://` URI.
- Access Cloud Storage content through your current project's Agent Platform service agent. By default, the Agent Platform service agent already has permission to access the bucket in the same project. For more information, see [Identity and Access Management (IAM) roles and permissions index](https://docs.cloud.google.com/iam/docs/roles-permissions/aiplatform.md.txt#aiplatform.serviceAgent).
- To access data in a Cloud Storage bucket in another Google Cloud project,
  run the following gcloud CLI command to grant the [Storage Object Viewer role (`roles/storage.objectViewer`)](https://docs.cloud.google.com/storage/docs/access-control/iam-roles.md.txt) to the Agent Platform service agent of your
  AlloyDB project.

  ```bash
  gcloud projects add-iam-policy-binding <ANOTHER_PROJECT_ID> \
  --member="serviceAccount:service-<PROJECT_ID>@gcp-sa-aiplatform.iam.gserviceaccount.com" \
  --role="roles/storage.objectViewer"
  ```

  For more information, see [Set and manage IAM policies on buckets](https://docs.cloud.google.com/storage/docs/access-control/using-iam-permissions.md.txt).

To generate multimodal embeddings, select one of the following schemas.
<button value="ai" default="">ai schema</button> <button value="google_ml">google_ml schema</button>

> [!NOTE]
> **Note:** AlloyDB doesn't charge for AI function calls. However, when you call Gemini Enterprise Agent Platform, third-party, or self-hosted models and services (such as for hybrid search) from your database, the model or infrastructure provider charges you directly. For more information about Gemini Enterprise Agent Platform costs, see [Pricing](https://cloud.google.com/products/gemini-enterprise-agent-platform/pricing).

## Generate multimodal embeddings

To generate text embeddings for a `multimodalembedding@001` model endpoint, run the following statement:

```sql
SELECT
  ai.text_embedding(
    model_id => 'multimodalembedding@001',
    content => 'TEXT');
```

Replace `TEXT` with the text to generate the embedding for.

To generate image embeddings for a registered `multimodalembedding@001` model endpoint where the image mimetype is default `image/jpeg`, run the following statement:

```sql
SELECT
  ai.image_embedding(
    model_id => 'multimodalembedding@001',
    image => 'IMAGE_PATH_OR_TEXT',
    mimetype => 'MIMETYPE');
```

Replace the following:

- `IMAGE_PATH_OR_TEXT`: the Cloud Storage URI of the image, for example, `gs://my-bucket/embeddings/flowers.jpeg`, or the base64 string of the image.

- `MIMETYPE`: the mimetype of the image, for example, `image/jpeg`. For the full list of supported mimetypes, see the
  [Multimodal embeddings API](https://docs.cloud.google.com/gemini-enterprise-agent-platform/reference/models/multimodal-embeddings-api.md.txt#image).

To generate video embeddings for a registered `multimodalembedding@001` model endpoint, run the following statement:

```sql
SELECT
  ai.video_embedding(
    model_id => 'multimodalembedding@001',
    video => 'VIDEO_URI');
```

Replace `VIDEO_URI` with the Cloud Storage URI of the target video, for example, `gs://my-bucket/embeddings/supermarket-video.mp4`, or the base64 string of the video.
These are two-dimensional arrays that can be accessed as `my_array[0][5]` like syntax.
For more information, see [Arrays](https://www.postgresql.org/docs/17/arrays.html).

## What's next

- [Register a model endpoint with model endpoint management](https://docs.cloud.google.com/alloydb/docs/ai/register-model-endpoint.md.txt).

- [Query using AI powered SQL operators](https://docs.cloud.google.com/alloydb/docs/ai/evaluate-semantic-queries-ai-operators.md.txt).