Skip to content

Commit

Permalink
fix: Gemini 2.0 live api audio not working
Browse files Browse the repository at this point in the history
  • Loading branch information
LivioGama committed Dec 17, 2024
1 parent f1b9eeb commit c80f234
Show file tree
Hide file tree
Showing 8 changed files with 649 additions and 711 deletions.
1 change: 1 addition & 0 deletions .github/CODEOWNERS
Validating CODEOWNERS rules …
Original file line number Diff line number Diff line change
Expand Up @@ -80,3 +80,4 @@
/generative-ai/gemini/sample-apps/e2e-gen-ai-app-starter-pack @eliasecchig @lspatarog @mariagpuyol @GoogleCloudPlatform/generative-ai-devrel
/generative-ai/vision/use-cases/hey_llm @tushuhei @GoogleCloudPlatform/generative-ai-devrel
/generative-ai/gemini/sample-apps/llamaindex-rag/backend/indexing/ @Lionel-Lim @GoogleCloudPlatform/generative-ai-devrel
/generative-ai/gemini/multimodal-live-api/websocket-demo-app/ @LivioGama @GoogleCloudPlatform/generative-ai-devrel
13 changes: 9 additions & 4 deletions gemini/multimodal-live-api/websocket-demo-app/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@ While some web development experience, particularly with localhost, port numbers

### File Structure


- backend/main.py: The Python backend code
- backend/requirements.txt: Lists the required Python dependencies

Expand Down Expand Up @@ -61,7 +60,11 @@ python3 backend/main.py
```

5. Start the frontend:
Make sure to open a **new** terminal window to run this command. Keep the backend server running in the first terminal.

- Navigate to `script.js` on line 9, `const PROXY_URL = "wss://[THE_URL_YOU_COPIED_WITHOUT_HTTP]";` and replace `PROXY_URL` value with `ws://localhost:8000`. It should look like: `const PROXY_URL = "ws://localhost:8000;";`. Note the absence of the second "s" in "wss" since it stands for secure.
- Right below on line 10, update `PROJECT_ID` with your Google Cloud project ID.
- Save the changes you've made to `script.js`
- Now make sure to open a **separate** terminal window from the backend to run this command (keep the backend server running in the first terminal).

```sh
cd frontend
Expand All @@ -74,14 +77,16 @@ python3 -m http.server
Run the following command in a terminal with gcloud installed to set your project, and to retrieve your access token.

```sh
gcloud components update
gcloud components install beta
gcloud config set project YOUR-PROJECT-ID
gcloud auth print-access-token
```

8. Copy the access token from the previous step into the UI that you have open in your browser.

9. Enter the model ID in the UI:
Replace `YOUR-PROJECT-ID` in the input with your credentials
Replace `YOUR-PROJECT-ID` in the input with your Google Cloud project ID.

10. Connect and interact with the demo:

Expand All @@ -90,7 +95,7 @@ gcloud auth print-access-token
11. To interact with the app, you can do the following:

- Text input: You can write a text prompt to send to the model by entering your message in the box and pressing the send arrow. The model will then respond via audio (turn up your volume!).
- Voice input: Press the pink microphone button and start speaking. The model will respond via audio. If you would like to mute your microphone, press the button with a slash through the microphone.
- Voice input: Press the microphone button to stop speaking. The model will respond via audio. If you would like to mute your microphone, press the button with a slash through the microphone.
- Video input: The model will also capture your camera input and send it to Gemini. You can ask questions about current or previous video footage. For more details on how this works, visit the [documentation page for the Multimodal Live API](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/multimodal-live).

### Setup in Cloud Shell
Expand Down
118 changes: 60 additions & 58 deletions gemini/multimodal-live-api/websocket-demo-app/frontend/cookieJar.js
Original file line number Diff line number Diff line change
@@ -1,65 +1,67 @@
class CookieJar {
/**
* @class CookieJar
* @classdesc A utility class for managing cookies associated with HTML input elements.
*/
/**
* @class CookieJar
* @classdesc A utility class for managing cookies associated with HTML input elements.
*/

/**
* @static
* @method init
* @memberof CookieJar
* @description Initializes the CookieJar for a given element. Loads saved value from cookie and sets up auto-saving on value change.
* @param {string} elementId - The ID of the HTML input element.
*/
static init(elementId) {
const element = document.getElementById(elementId);
if (!element) {
console.error(`❌ Element with ID '${elementId}' not found.`);
return;
}

const cookieName = `CookieJar_${elementId}`;
/**
* @static
* @method init
* @memberof CookieJar
* @description Initializes the CookieJar for a given element. Loads saved value from cookie and sets up auto-saving on value change.
* @param {string} elementId - The ID of the HTML input element.
*/
static init(elementId) {
const element = document.getElementById(elementId);
if (!element) {
console.error(`❌ Element with ID '${elementId}' not found.`);
return;
}

// Load existing value from cookie
const savedValue = CookieJar.getCookie(cookieName);
if (savedValue) {
console.log(`🍪 Found cookie for ${elementId}. Loading value: ${savedValue}`);
element.value = savedValue;
}
const cookieName = `CookieJar_${elementId}`;

// Save on value change
element.addEventListener('input', () => {
console.log(`🍪 Saving value for ${elementId} to cookie...`);
CookieJar.setCookie(cookieName, element.value);
});
// Load existing value from cookie
const savedValue = CookieJar.getCookie(cookieName);
if (savedValue) {
console.log(`🍪 Found cookie for ${elementId}. Loading value: ${savedValue}`);
element.value = savedValue;
}

/**
* @static
* @method setCookie
* @memberof CookieJar
* @description Sets a cookie with the given name, value, and optional expiration days.
* @param {string} name - The name of the cookie.
* @param {string} value - The value to store in the cookie.
* @param {number} [days=365] - The number of days until the cookie expires. Defaults to 365.
*/
static setCookie(name, value, days = 365) {
const expires = new Date();
expires.setTime(expires.getTime() + days * 24 * 60 * 60 * 1000);
document.cookie = `${name}=${encodeURIComponent(value)};expires=${expires.toUTCString()};path=/`;
console.log(`🍪 Cookie '${name}' set successfully!`);
}
// Save on value change
element.addEventListener("input", () => {
console.log(`🍪 Saving value for ${elementId} to cookie...`);
CookieJar.setCookie(cookieName, element.value);
});
}

/**
* @static
* @method getCookie
* @memberof CookieJar
* @description Retrieves the value of a cookie with the given name.
* @param {string} name - The name of the cookie to retrieve.
* @returns {string|null} The value of the cookie if found, otherwise null.
*/
static getCookie(name) {
const cookieValue = document.cookie.match(`(^|;)\\s*${name}\\s*=\\s*([^;]+)`);
return cookieValue ? decodeURIComponent(cookieValue.pop()) : null;
}
}
/**
* @static
* @method setCookie
* @memberof CookieJar
* @description Sets a cookie with the given name, value, and optional expiration days.
* @param {string} name - The name of the cookie.
* @param {string} value - The value to store in the cookie.
* @param {number} [days=365] - The number of days until the cookie expires. Defaults to 365.
*/
static setCookie(name, value, days = 365) {
const expires = new Date();
expires.setTime(expires.getTime() + days * 24 * 60 * 60 * 1000);
document.cookie = `${name}=${encodeURIComponent(
value,
)};expires=${expires.toUTCString()};path=/`;
console.log(`🍪 Cookie '${name}' set successfully!`);
}

/**
* @static
* @method getCookie
* @memberof CookieJar
* @description Retrieves the value of a cookie with the given name.
* @param {string} name - The name of the cookie to retrieve.
* @returns {string|null} The value of the cookie if found, otherwise null.
*/
static getCookie(name) {
const cookieValue = document.cookie.match(`(^|;)\\s*${name}\\s*=\\s*([^;]+)`);
return cookieValue ? decodeURIComponent(cookieValue.pop()) : null;
}
}
Loading

0 comments on commit c80f234

Please sign in to comment.