Mulai menggunakan Gemini Live API menggunakan WebSockets

Tutorial ini menunjukkan cara terhubung ke Gemini Live API menggunakan WebSocket. Dalam tutorial ini, Anda akan membangun aplikasi multimodal real-time dengan frontend JavaScript vanilla dan server Python yang menangani autentikasi dan proxy.

Sebelum memulai

Selesaikan langkah-langkah berikut untuk menyiapkan lingkungan Anda.

  1. Login ke akun Google Cloud Anda. Jika Anda baru menggunakan Google Cloud, buat akun untuk mengevaluasi performa produk kami dalam skenario dunia nyata. Pelanggan baru juga mendapatkan kredit gratis senilai $300 untuk menjalankan, menguji, dan men-deploy workload.

  2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Roles required to select or create a project

    • Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
    • Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

    Go to project selector

  3. Verify that billing is enabled for your Google Cloud project.

  4. Enable the Gemini Enterprise Agent Platform API, if it is not already enabled.

    Roles required to enable APIs

    To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.

    Enable the API

  5. Install the Google Cloud CLI.

  6. If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

  7. To initialize the gcloud CLI, run the following command: 

    gcloud init

  8. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Roles required to select or create a project

    • Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
    • Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

    Go to project selector

  9. Verify that billing is enabled for your Google Cloud project.

  10. Enable the Gemini Enterprise Agent Platform API, if it is not already enabled.

    Roles required to enable APIs

    To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.

    Enable the API

  11. Install the Google Cloud CLI.

  12. If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

  13. To initialize the gcloud CLI, run the following command: 

    gcloud init
  14. Instal Git.
  15. Instal Python 3.

Meng-clone aplikasi demo

Buat clone repositori aplikasi demo dan buka direktori tersebut:

git clone https://github.com/GoogleCloudPlatform/generative-ai.git &&
cd generative-ai/gemini/multimodal-live-api/native-audio-websocket-demo-apps/plain-js-demo-app

Struktur project

Aplikasi ini mencakup file berikut:

/
├── server.py            # WebSocket proxy + HTTP server
├── requirements.txt     # Python dependencies
└── frontend/
    ├── index.html       # UI
    ├── geminilive.js    # Gemini API client
    ├── mediaUtils.js    # Audio/video streaming
    ├── tools.js         # Custom tool definitions
    └── script.js        # App logic

Menjalankan server backend

Backend (server.py) menangani autentikasi dan bertindak sebagai proxy WebSocket antara klien dan Gemini Live API.

Untuk menjalankan server backend, jalankan perintah berikut:

  1. Instal dependensi:

    pip3 install -r requirements.txt

  2. Lakukan autentikasi dengan Google Cloud:

    gcloud auth application-default login

  3. Mulai server:

    python3 server.py

Buka UI frontend dan hubungkan dengan Gemini

Frontend mengelola perekaman dan pemutaran audio dan video. File geminilive.js menangani koneksi WebSocket ke backend.

const client = new GeminiLiveAPI(proxyUrl, projectId, model);
client.addFunction(toolInstance); // Add custom tools
client.connect(accessToken); // Connect (token optional with proxy)

Untuk membuka UI frontend dan terhubung dengan Gemini, lakukan hal berikut:

  1. Buka browser Anda dan buka http://localhost:8000.
  2. Di kolom Project ID, masukkan ID project Google Cloud Anda.
  3. Klik Hubungkan.

Berinteraksi dengan Gemini

Coba lakukan hal berikut:

  • Input teks: Anda dapat menulis pesan teks ke Gemini dengan memasukkan pesan Anda di kolom pesan dan mengklik Kirim. Gemini membalas pesan menggunakan audio.
  • Input suara: Untuk berbicara dengan Gemini, klik Mulai mikrofon. Gemini merespons perintah menggunakan audio.
  • Input video: Agar Gemini dapat melihat melalui kamera Anda, klik Mulai kamera. Anda dapat berbicara dengan Gemini tentang apa yang dilihatnya melalui kamera Anda.

Langkah berikutnya