Allow RatingQuestion to take value 0

[MoritzLaurer]

small feedback on rg.RatingQuestion: at the moment, RatingQuestions are not allowed to take the value 0. There has been some recent literature on LLM-as-a-judge where people use cumulative prompts/instructions and you award points to a response for each fulfilled criterion. These prompt types can also result in 0 points if no criterion is fulfilled. At the moment, it's not possible to use this type of prompt with Argilla's RatingQuestions, because the minimum value always has to be at least 1. Is there added value to enforcing a minimum value of 1? Being able to set values=[0, 1, 2, 3, 4, 5] for these types of instructions would be useful.

dataset_argilla = rg.FeedbackDataset(
    fields=[
        rg.TextField(name="content", use_markdown=True, required=True),
    ],
    questions=[
        rg.RatingQuestion(
            name="rating_criterion_1",
            description="Some quality rating on a scale from 0 to 5.",
            required=True,
            values=[0, 1, 2, 3, 4, 5]
        )
    ]
)

ValidationError: 1 validation error for RatingQuestion
values -> 0
  ensure this value is greater than or equal to 1 (type=value_error.number.not_ge; limit_value=1)

Tasks

Preview Give feedback

1. [TASK] Support zero as value for rating questions on argilla-server #4857
   +0
   
2. [TASK] Support zero as value for rating questions on argilla old SDK #4859
   +0
   
3. [TASK] Support zero as value for rating questions on UI #4861
   +0
   
4. [TASK] Support zero as value for rating questions on argilla new SDK #4862
   +0
   
5. [DOCS] Change docs to show zero values as supported for rating questions #4863
   +0
   

[burtenshaw]

Thanks for the proposal @MoritzLaurer .

Out of interest, could 0 be communicated by a None answer in a required=False rating question? The advantage would be that it saves your annotator time.

[MoritzLaurer]

mh.. I think explicitly selecting "0 points" is important for these prompts. Here is an example:

eval_prompt_cumulative = f"""\
Your task is to evaluate the quality of the image \
based on the color scheme and how well the colors harmonize with the content of the image.

First reason step by step for your evaluation and then return a quality score.

Instructions for scoring on a cumulative scale from 0 to 2: 
- Award zero points if the colors neither harmonize with each other nor with the content of the image.  
- Add one point if the the colors harmonize well with each other
- Add one point if the colors harmonize well with the content of the image

Use the following JSON schema:\n{schema_simplified}
"""

0 points is an explicit option, so I think that would be best for annotators to select it explicitly.

[burtenshaw]

Thanks for the clarification @MoritzLaurer

I agree that it makes sense for the annotator to specify 0 and that because it is 'cumulative' it should be a RatingQuestion' and not LabelQuestion` with 0 as a label.

@nataliaElv We should clarify in our docs the difference between a 0 and null response. We could also give examples like this one compared to a 1-5 rating.