feat: add Gemini-pro-1.5 to GeminiTextGenerator Tuning and Support score() method in Gemini-pro-1.5 #1208

shuoweil · 2024-12-11T18:51:47Z

feat: add Gemini-pro-1.5 to GeminiTextGenerator Tuning and Support score() method in Gemini-pro-1.5

Bug: b/381936588 and b/344891364

merge to main branch to avoid conflict

Update ll.TextEmbeddingGenerator to 005, update feature, docs, and test. The default vlaue is kept as text-embedding-004 for better customer experience. Bug:381935784

merge to main branch to avoid conflict

…ethod in Gemini-pro-1.5 \n Bug: b/381936588 and b/344891364

GarrettWu

PR title has typos.

It should be adding new features instead of "update".

GarrettWu · 2024-12-11T22:58:51Z

bigframes/ml/llm.py

@@ -892,8 +892,12 @@ def fit(
        Returns:
            GeminiTextGenerator: Fitted estimator.
        """
-        if self._bqml_model.model_name.startswith("gemini-1.5"):
-            raise NotImplementedError("Fit is not supported for gemini-1.5 model.")
+        # Support gemini-1.5 and gemini-pro


move the consts to the top of the file for better organization.

GarrettWu · 2024-12-11T23:00:12Z

bigframes/ml/llm.py

+        supported_models = ["gemini-pro", "gemini-1.5-pro-002", "gemini-1.5-flash-002"]
+        if self.model_name not in supported_models:
+            raise NotImplementedError(
+                "Score is not supported models other than gemini-pro or gemini-1.5 model."


be more explicit to "gemini-1.5-pro-002" and "gemini-1.5-flash-002".

Since other gemini 1.5 endpoints aren't supported.

GarrettWu · 2024-12-11T23:01:07Z

bigframes/ml/llm.py

+        supported_models = ["gemini-pro", "gemini-1.5-pro-002", "gemini-1.5-flash-002"]
+        if self.model_name not in supported_models:
+            raise NotImplementedError(
+                "Score is not supported models other than gemini-pro or gemini-1.5 model."


This is in fit() in stead of score().

I am working on two bugs, b/381936588 and b/344891364, both fit() and score() needed to be updated.

I mean the message is incorrect

GarrettWu · 2024-12-11T23:05:20Z

bigframes/ml/llm.py

+        if self.model_name not in supported_models:
+            raise NotImplementedError(
+                "Score is not supported models other than gemini-pro or gemini-1.5 model."
+            )


I couldn't leave a comment at unchanged lines. Need to update line 905 to each endpoint respectively. (still use gemini-1.0-pro-002 for gemini-pro, but issuing a warning for that case maybe more appropriate)

GarrettWu · 2024-12-11T23:06:02Z

bigframes/ml/llm.py

@@ -1009,7 +1013,7 @@ def score(
            "text_generation", "classification", "summarization", "question_answering"
        ] = "text_generation",
    ) -> bpd.DataFrame:
-        """Calculate evaluation metrics of the model. Only "gemini-pro" model is supported for now.
+        """Calculate evaluation metrics of the model. Only "gemini-pro" and "gemini-1.5" models are supported for now.


same, more explicit to "gemini-1.5-pro-002" and "gemini-1.5-flash-002"

GarrettWu · 2024-12-11T23:06:40Z

bigframes/ml/llm.py

-        if self._bqml_model.model_name.startswith("gemini-1.5"):
-            raise NotImplementedError("Score is not supported for gemini-1.5 model.")
+        # Support gemini-1.5 and gemini-pro
+        supported_models = ["gemini-pro", "gemini-1.5-pro-002", "gemini-1.5-flash-002"]


Move and share consts to top.

Not seeing the change? I mean add const variables to the top of the file like https://github.com/googleapis/python-bigquery-dataframes/blob/main/bigframes/ml/llm.py#L67

GarrettWu · 2024-12-11T23:08:27Z

bigframes/ml/llm.py

+        supported_models = ["gemini-pro", "gemini-1.5-pro-002", "gemini-1.5-flash-002"]
+        if self.model_name not in supported_models:
+            raise NotImplementedError(
+                "Score is not supported models other than gemini-pro or gemini-1.5 model."


Same, more explicit.

Also the sentence seems awkward.

The message is ill-formed. Maybe just "score() method only supports xxx models".

GarrettWu · 2024-12-11T23:10:41Z

tests/system/small/ml/test_llm.py

-@pytest.mark.flaky(retries=2)
-def test_llm_gemini_pro_score(llm_fine_tune_df_default_index):
-    model = llm.GeminiTextGenerator(model_name="gemini-pro")
+# test score() function for "gemini-pro" and "gemini-1.5" model


nit: test name already indicates it. The comment seems redundant.

GarrettWu · 2024-12-11T23:14:53Z

tests/system/small/ml/test_llm.py

+        "gemini-1.5-flash-002",
+    ),
+)
+def test_llm_gemini_score(llm_fine_tune_df_default_index, model_name):


add for next test "test_llm_gemini_pro_score_params" as well

This reverts commit 9de2c0e.

GarrettWu · 2024-12-13T18:12:17Z

bigframes/ml/llm.py

+    _GEMINI_1P5_PRO_002_ENDPOINT,
+    _GEMINI_1P5_FLASH_002_ENDPOINT,
+)
+_GEMINI_SCORE_ENDPOINTS = (


Can we merge the 2 lists? They are identical, don't think they will diverge.

… since they are identical

…ey are identical

…since they are identical" This reverts commit 205e173.

Shuowei Li added 10 commits December 3, 2024 00:16

docs(bigquery): update minor parts in base.py

97d9259

docs(bigquery): update minor changes for bigframes/ml/base.py

c9318d0

Merge branch 'shuowei-doc-update'

4f9370c

merge to main branch to avoid conflict

Merge branch 'shuowei-doc-update'

8d5e0ed

merge to main branch to avoid conflict

Merge branch 'shuowei-update-textembedding005'

e5413a1

Update ll.TextEmbeddingGenerator to 005, update feature, docs, and test. The default vlaue is kept as text-embedding-004 for better customer experience. Bug:381935784

Merge branch 'shuowei-doc-update'

7b65227

merge to main branch to avoid conflict

Merge branch 'shuowei-update-textembedding005'

9d6376b

Merge branch 'shuowei-text-generator-tuning'

f315f54

feat: Update lUpdate GeminiTextGenerator Tuning and Support score() m…

e9f28f4

…ethod in Gemini-pro-1.5 \n Bug: b/381936588 and b/344891364

feat: Update lUpdate GeminiTextGenerator Tuning and Support score() m…

5d2a807

…ethod in Gemini-pro-1.5 \n Bug: b/381936588 and b/344891364

shuoweil requested a review from GarrettWu December 11, 2024 18:51

shuoweil self-assigned this Dec 11, 2024

shuoweil requested review from a team as code owners December 11, 2024 18:51

product-auto-label bot added size: s Pull request size is small. api: bigquery Issues related to the googleapis/python-bigquery-dataframes API. labels Dec 11, 2024

Merge branch 'main' into shuowei-text-generator-tuning

dc765ec

GarrettWu reviewed Dec 11, 2024

View reviewed changes

shuoweil changed the title ~~feat: Update lUpdate GeminiTextGenerator Tuning and Support score() method in Gemini-pro-1.5~~ feat: add Gemini-pro-1.5 to GeminiTextGenerator Tuning and Support score() method in Gemini-pro-1.5 Dec 11, 2024

shuoweil and others added 3 commits December 11, 2024 23:23

Merge branch 'main' into shuowei-text-generator-tuning

7a40315

update testcase and docs for better clarification

8be16d3

update endpoint to corresponding endpoint for fine tuning.

8a39a02

product-auto-label bot added size: m Pull request size is medium. and removed size: s Pull request size is small. labels Dec 12, 2024

shuoweil and others added 6 commits December 12, 2024 01:35

Merge branch 'main' into shuowei-text-generator-tuning

361a734

Merge branch 'main' into shuowei-text-generator-tuning

b213753

docs(bigquery): update minor parts in base.py

9de2c0e

fix syntax issue

ed001b8

Revert "docs(bigquery): update minor parts in base.py"

9928f10

This reverts commit 9de2c0e.

Merge branch 'main' into shuowei-text-generator-tuning

ba80d10

GarrettWu approved these changes Dec 13, 2024

View reviewed changes

Shuowei Li added 3 commits December 13, 2024 19:05

merge gemini_fine_tune_endpoints and gemini_score_endpoints together,…

241ae73

… since they are identical

merge genimi_fine_tune_endpoints and genimi_score_endpoints, since th…

205e173

…ey are identical

Revert "merge genimi_fine_tune_endpoints and genimi_score_endpoints, …

6a44e7b

…since they are identical" This reverts commit 205e173.

shuoweil added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Dec 13, 2024

bigframes-bot removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Dec 13, 2024

shuoweil added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Dec 13, 2024

bigframes-bot removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Dec 13, 2024

shuoweil merged commit 298fc73 into main Dec 13, 2024
22 checks passed

shuoweil deleted the shuowei-text-generator-tuning branch December 13, 2024 23:44

release-please bot mentioned this pull request Dec 13, 2024

chore(main): release 1.30.0 #1215

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add Gemini-pro-1.5 to GeminiTextGenerator Tuning and Support score() method in Gemini-pro-1.5 #1208

feat: add Gemini-pro-1.5 to GeminiTextGenerator Tuning and Support score() method in Gemini-pro-1.5 #1208

shuoweil commented Dec 11, 2024 •

edited

Loading

GarrettWu left a comment

GarrettWu Dec 11, 2024

shuoweil Dec 11, 2024

GarrettWu Dec 11, 2024

shuoweil Dec 11, 2024

GarrettWu Dec 11, 2024

shuoweil Dec 11, 2024

GarrettWu Dec 12, 2024

shuoweil Dec 12, 2024

GarrettWu Dec 11, 2024

shuoweil Dec 12, 2024

GarrettWu Dec 11, 2024

shuoweil Dec 11, 2024

GarrettWu Dec 11, 2024

shuoweil Dec 11, 2024

GarrettWu Dec 12, 2024

shuoweil Dec 12, 2024

GarrettWu Dec 11, 2024

shuoweil Dec 11, 2024

GarrettWu Dec 12, 2024

shuoweil Dec 12, 2024

GarrettWu Dec 11, 2024

shuoweil Dec 11, 2024

GarrettWu Dec 11, 2024

shuoweil Dec 11, 2024

GarrettWu Dec 13, 2024

shuoweil Dec 13, 2024

feat: add Gemini-pro-1.5 to GeminiTextGenerator Tuning and Support score() method in Gemini-pro-1.5 #1208

feat: add Gemini-pro-1.5 to GeminiTextGenerator Tuning and Support score() method in Gemini-pro-1.5 #1208

Conversation

shuoweil commented Dec 11, 2024 • edited Loading

GarrettWu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shuoweil commented Dec 11, 2024 •

edited

Loading