feat: add support for value zero on rating questions (#4864)

# Description This PR is the feature branch with all the changes to support zero value on rating questions. Closes #4822 **Type of change** (Please delete options that are not relevant. Remember to title the PR according to the type of change) - [ ] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Refactor (change restructuring the codebase without changing functionality) - [x] Improvement (change adding some improvement to an existing functionality) - [ ] Documentation update **How Has This Been Tested** (Please describe the tests that you ran to verify your changes. And ideally, reference `tests`) - [x] Adding and changing tests. **Checklist** - [ ] I added relevant documentation - [ ] follows the style guidelines of this project - [ ] I did a self-review of my code - [ ] I made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I filled out [the contributor form](https://tally.so/r/n9XrxK) (see text above) - [ ] I have added relevant notes to the CHANGELOG.md file (See https://keepachangelog.com/) # Argilla Community Growers Thanks for your contribution! As part of our Community Growers initiative 🌱, we're donating Justdiggit bunds in your name to reforest sub-Saharan Africa. To claim your Community Growers certificate, please contact David Berenstein in our Slack community or fill in this form https://tally.so/r/n9XrxK once your PR has been merged. # Pull Request Templates Please go the the `Preview` tab and select the appropriate sub-template: * [🐞-bug](?expand=1&template=bug.md) * [📚-documentation](?expand=1&template=docs.md) * [🆕-features](?expand=1&template=features.md) # Generic Pull Request Template Please include a summary of the changes and the related issue. Please also include relevant motivation and context. List any dependencies that are required for this change. Closes #<issue_number> **Type of change** (Please delete options that are not relevant. Remember to title the PR according to the type of change) - [ ] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Refactor (change restructuring the codebase without changing functionality) - [ ] Improvement (change adding some improvement to an existing functionality) - [ ] Documentation update **How Has This Been Tested** (Please describe the tests that you ran to verify your changes. And ideally, reference `tests`) - [ ] Test A - [ ] Test B **Checklist** - [ ] I added relevant documentation - [ ] follows the style guidelines of this project - [ ] I did a self-review of my code - [ ] I made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I filled out [the contributor form](https://tally.so/r/n9XrxK) (see text above) - [ ] I have added relevant notes to the CHANGELOG.md file (See https://keepachangelog.com/)
argilla-io · May 23, 2024 · b433ebc · b433ebc
1 parent 1bb067e
commit b433ebc
Show file tree

Hide file tree

Showing 11 changed files with 84 additions and 24 deletions.
diff --git a/argilla-server/CHANGELOG.md b/argilla-server/CHANGELOG.md
@@ -18,6 +18,7 @@ These are the section headers that we use:
 
 ### Added
 
+- Added support for rating questions to include `0` as a valid value. ([#4858](https://github.com/argilla-io/argilla/pull/4858))
 - Added `POST /api/v1/token` endpoint to generate a new API token for a user. ([#138](https://github.com/argilla-io/argilla-server/pull/138))
 - Added `GET /api/v1/me` endpoint to get the current user information. ([#140](https://github.com/argilla-io/argilla-server/pull/140))
 - Added `GET /api/v1/users` endpoint to get a list of all users. ([#142](https://github.com/argilla-io/argilla-server/pull/142))

diff --git a/argilla-server/src/argilla_server/schemas/v1/questions.py b/argilla-server/src/argilla_server/schemas/v1/questions.py
@@ -53,9 +53,9 @@
 RANKING_OPTIONS_MAX_ITEMS = 50
 
 RATING_OPTIONS_MIN_ITEMS = 2
-RATING_OPTIONS_MAX_ITEMS = 10
-RATING_LOWER_VALUE_ALLOWED = 1
-RATING_UPPER_VALUE_ALLOWED = 10
+RATING_OPTIONS_MAX_ITEMS = 11
+RATING_VALUE_GREATER_THAN_OR_EQUAL = 0
+RATING_VALUE_LESS_THAN_OR_EQUAL = 10
 
 SPAN_OPTIONS_MIN_ITEMS = 1
 SPAN_MIN_VISIBLE_OPTIONS = 3
@@ -124,30 +124,22 @@ class RatingQuestionSettingsOption(BaseModel):
     value: int
 
 
+class RatingQuestionSettingsOptionCreate(BaseModel):
+    value: int = Field(ge=RATING_VALUE_GREATER_THAN_OR_EQUAL, le=RATING_VALUE_LESS_THAN_OR_EQUAL)
+
+
 class RatingQuestionSettings(BaseModel):
     type: Literal[QuestionType.rating]
-    options: conlist(item_type=RatingQuestionSettingsOption)
+    options: List[RatingQuestionSettingsOption]
 
 
 class RatingQuestionSettingsCreate(UniqueValuesCheckerMixin):
     type: Literal[QuestionType.rating]
-    options: conlist(
-        item_type=RatingQuestionSettingsOption,
+    options: List[RatingQuestionSettingsOptionCreate] = Field(
         min_items=RATING_OPTIONS_MIN_ITEMS,
         max_items=RATING_OPTIONS_MAX_ITEMS,
     )
 
-    @validator("options")
-    def check_option_value_range(cls, options: List[RatingQuestionSettingsOption]):
-        """Validator to control all values are in allowed range 1 <= x <= 10"""
-        for option in options:
-            if not RATING_LOWER_VALUE_ALLOWED <= option.value <= RATING_UPPER_VALUE_ALLOWED:
-                raise ValueError(
-                    f"Option value {option.value!r} out of range "
-                    f"[{RATING_LOWER_VALUE_ALLOWED!r}, {RATING_UPPER_VALUE_ALLOWED!r}]"
-                )
-        return options
-
 
 class RatingQuestionSettingsUpdate(UpdateSchema):
     type: Literal[QuestionType.rating]

diff --git a/argilla-server/tests/factories.py b/argilla-server/tests/factories.py
@@ -320,6 +320,7 @@ class RatingQuestionFactory(QuestionFactory):
     settings = {
         "type": QuestionType.rating.value,
         "options": [
+            {"value": 0},
             {"value": 1},
             {"value": 2},
             {"value": 3},

diff --git a/argilla-server/tests/unit/api/v1/datasets/questions/test_create_dataset_question.py b/argilla-server/tests/unit/api/v1/datasets/questions/test_create_dataset_question.py
@@ -112,3 +112,63 @@ async def test_create_dataset_multi_label_selection_question_with_options_order_
 
         assert response.status_code == 422
         assert (await db.execute(select(func.count(Question.id)))).scalar_one() == 0
+
+    async def test_create_dataset_rating_question(
+        self, db: AsyncSession, async_client: AsyncClient, owner_auth_header: dict
+    ):
+        dataset = await DatasetFactory.create()
+
+        response = await async_client.post(
+            self.url(dataset.id),
+            headers=owner_auth_header,
+            json={
+                "name": "name",
+                "title": "title",
+                "settings": {
+                    "type": QuestionType.rating,
+                    "options": [
+                        {"value": 0},
+                        {"value": 1},
+                        {"value": 2},
+                        {"value": 3},
+                        {"value": 4},
+                        {"value": 5},
+                        {"value": 6},
+                        {"value": 7},
+                        {"value": 8},
+                        {"value": 9},
+                        {"value": 10},
+                    ],
+                },
+            },
+        )
+
+        question = (await db.execute(select(Question))).scalar_one()
+
+        assert response.status_code == 201
+        assert response.json() == {
+            "id": str(question.id),
+            "name": "name",
+            "title": "title",
+            "description": None,
+            "required": False,
+            "settings": {
+                "type": QuestionType.rating,
+                "options": [
+                    {"value": 0},
+                    {"value": 1},
+                    {"value": 2},
+                    {"value": 3},
+                    {"value": 4},
+                    {"value": 5},
+                    {"value": 6},
+                    {"value": 7},
+                    {"value": 8},
+                    {"value": 9},
+                    {"value": 10},
+                ],
+            },
+            "dataset_id": str(dataset.id),
+            "inserted_at": question.inserted_at.isoformat(),
+            "updated_at": question.updated_at.isoformat(),
+        }
diff --git a/argilla-server/tests/unit/api/v1/test_datasets.py b/argilla-server/tests/unit/api/v1/test_datasets.py
@@ -326,6 +326,7 @@ async def test_list_dataset_questions(self, async_client: "AsyncClient", owner_a
                     "settings": {
                         "type": "rating",
                         "options": [
+                            {"value": 0},
                             {"value": 1},
                             {"value": 2},
                             {"value": 3},

diff --git a/argilla-server/tests/unit/api/v1/test_questions.py b/argilla-server/tests/unit/api/v1/test_questions.py
@@ -69,6 +69,7 @@
             {
                 "type": "rating",
                 "options": [
+                    {"value": 0},
                     {"value": 1},
                     {"value": 2},
                     {"value": 3},

diff --git a/argilla-server/tests/unit/api/v1/test_records.py b/argilla-server/tests/unit/api/v1/test_records.py
@@ -790,7 +790,7 @@ async def test_create_record_response_with_extra_question_responses(
                         "rating_question_1": {"value": "wrong-rating-value"},
                     },
                 },
-                "'wrong-rating-value' is not a valid rating for rating question.\nValid ratings are: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]",
+                "'wrong-rating-value' is not a valid rating for rating question.\nValid ratings are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]",
             ),
             (
                 create_label_selection_questions,

diff --git a/argilla/CHANGELOG.md b/argilla/CHANGELOG.md
@@ -16,6 +16,10 @@ These are the section headers that we use:
 
 ## [Unreleased]()
 
+### Added
+
+- Added support for rating questions to include `0` as a valid value. ([#4860](https://github.com/argilla-io/argilla/pull/4860))
+
 ### Fixed
 
 - Fix issue when record does not have vectors related ([#4856](https://github.com/argilla-io/argilla/pull/4856))

diff --git a/argilla/src/argilla/client/feedback/schemas/questions.py b/argilla/src/argilla/client/feedback/schemas/questions.py
@@ -126,15 +126,15 @@ class RatingQuestion(QuestionSchema, LabelMappingMixin):
             modified.
         values: The list of integer values of the rating question. There is not need
             for the values to be sequential, but they must be unique, contain at least two
-            unique integers in the range [1, 10].
+            unique integers in the range [0, 10].
 
     Examples:
         >>> from argilla.client.feedback.schemas.questions import RatingQuestion
         >>> RatingQuestion(name="rating_question", title="Rating Question", values=[1, 2, 3, 4, 5])
     """
 
     type: Literal[QuestionTypes.rating] = Field(QuestionTypes.rating.value, allow_mutation=False, const=True)
-    values: List[int] = Field(..., unique_items=True, ge=1, le=10, min_items=2)
+    values: List[int] = Field(..., unique_items=True, ge=0, le=10, min_items=2)
 
     @property
     def server_settings(self) -> Dict[str, Any]:

diff --git a/argilla/tests/unit/client/feedback/schemas/test_questions.py b/argilla/tests/unit/client/feedback/schemas/test_questions.py
@@ -76,13 +76,13 @@ def test_text_question(schema_kwargs: Dict[str, Any], server_payload: Dict[str,
             },
         ),
         (
-            {"name": "a", "title": "A", "description": "a", "required": False, "values": [1, 2, 3]},
+            {"name": "a", "title": "A", "description": "a", "required": False, "values": [0, 1, 2, 3]},
             {
                 "name": "a",
                 "title": "A",
                 "description": "a",
                 "required": False,
-                "settings": {"type": "rating", "options": [{"value": 1}, {"value": 2}, {"value": 3}]},
+                "settings": {"type": "rating", "options": [{"value": 0}, {"value": 1}, {"value": 2}, {"value": 3}]},
             },
         ),
     ],
@@ -100,7 +100,7 @@ def test_rating_question(schema_kwargs: Dict[str, Any], server_payload: Dict[str
         ({"name": "a", "values": ["a", "b"]}, ValidationError, "value is not a valid integer"),
         ({"name": "a", "values": [1, 1, 1]}, ValidationError, "the list has duplicated items"),
         ({"name": "a", "values": [1]}, ValidationError, "ensure this value has at least 2 items"),
-        ({"name": "a", "values": [0, 1]}, ValidationError, "ensure this value is greater than or equal to 1"),
+        ({"name": "a", "values": [-1, 0, 1]}, ValidationError, "ensure this value is greater than or equal to 0"),
         ({"name": "a", "values": [1, 11]}, ValidationError, "ensure this value is less than or equal to 10"),
     ],
 )

diff --git a/docs/_source/practical_guides/create_update_dataset/create_dataset.md b/docs/_source/practical_guides/create_update_dataset/create_dataset.md
@@ -90,7 +90,7 @@ You can define your questions using the Python SDK and set up the following conf
 
 The following arguments apply to specific question types:
 
-- `values`: In the `RatingQuestion` this will be any list of unique integers that represent the options that annotators can choose from. These values must be defined in the range [1, 10]. In the `RankingQuestion`, values will be a list of strings with the options they will need to rank. If you'd like the text of the options to be different in the UI and internally, you can pass a dictionary instead where the key is the internal name and the value is the text to display in the UI.
+- `values`: In the `RatingQuestion` this will be any list of unique integers that represent the options that annotators can choose from. These values must be defined in the range [0, 10]. In the `RankingQuestion`, values will be a list of strings with the options they will need to rank. If you'd like the text of the options to be different in the UI and internally, you can pass a dictionary instead where the key is the internal name and the value is the text to display in the UI.
 - `labels`: In `LabelQuestion`, `MultiLabelQuestion` and `SpanQuestion` this is a list of strings with the options for these questions. If you'd like the text of the labels to be different in the UI and internally, you can pass a dictionary instead where the key is the internal name and the value will be the text to display in the UI.
 - `field`: A `SpanQuestion` is always attached to a specific field. Here you should pass a string with the name of the field where the labels of the `SpanQuestion` should be used.
 - `allow_overlapping`: In a `SpanQuestion`, this value specifies whether overlapped spans are allowed or not. It is set to `False` by default. Set to `True` to allow overlapping spans.