Reviewer role and workflows

[Amelie-V]

Summary

Summarize the functionality or added value of the feature:

Including workflows to check the quality of the responses of the annotation team, including a role for team members who will be focusing on reviewing.

User needs

Set out the problem to be solved and the preferences or needs of users:

• Bigger teams often need to train other team members and control that they adhere to annotation guidelines.
• This can be done by project managers or by more experienced annotators.
• In the case of the experienced annotators, they can review but shouldn't be modifying the settings of the dataset.
• As an admin / project manager, I want the annotation team to be able to see each other's responses, so they can discuss and solve disagreements.

Proposed solution

Outline the initial solution to the problems or needs of the user:

• Add a new user role: reviewer.
• This user can see the responses from other team members and validate them.
• This user doesn't have permissions to manage or modify datasets / workspaces.

Further context

Add any additional information or details needed to understand the feature:

Related issues

Preview Give feedback

1. [FEATURE] progress visibility for the annotation progress argilla#3882
   +0
   

[davidberenstein1957]

According to GSK, it would also be useful as an annotator to see an overview of the used record labels and potential label overlap to assess the progress of the annotation task. This could for example be a barchart or a timeline of the applied labels/mistakes over time.

[vmehta-cohere]

This is really awesome for us. Hope to see this fleshed out as soon as possible

[dbuades]

This would also be very useful for us, since we have annotators with different levels of expertise in the team.

A good workflow could be to add new status for records (pending - discarded- submitted - reviewed - approved).
A reviewer would go through all their assigned submitted records and either approve them (the status would change to approved) or leave comments in them for the original annotator to modify (the status would change to reviewed).

Thank you for your work!

[dbuades]

Hello @Amelie-V !

Now that Argilla 2.0 has been released, do you have plans to work on this ? I'd be happy to collaborate with you to get this done, let me know how can I help. Thanks!

[cceyda]

This would also help my case argilla-io/argilla#5516

[cceyda]

When in the roadmap might this be? I as admin would really like to be able to see the submitted records by all annotators while annotation is in progress... but I'm basically flying blind and have to call the records into a dataset through the SDK everytime I wanna check something and do manual loops&hoops (which kinda defeats the visual goodness of argilla )

[vmehta-cohere]

Yes I totally agree aswell! I resonate with the fact that even with "visual goodness" argilla becomes a hard pitch for us since it's bascially adding more hoops for admins to access annotator progress.

[JonnyTran]

When in the roadmap might this be? I as admin would really like to be able to see the submitted records by all annotators while annotation is in progress... but I'm basically flying blind and have to call the records into a dataset through the SDK everytime I wanna check something and do manual loops&hoops (which kinda defeats the visual goodness of argilla )

I’m working on an argilla fork that can retrieve the submitted responses of other annotators to show up in the Suggestion tab. I recently had a demo to folks at work and they found it was a good way review consensus/differences in everyone’s answers. It’s not finished yet but happy to show how I did this.

[davidberenstein1957]

@nataliaElv @Amelie-V, perhaps you can have a chat with @JonnyTran about his cool-sounding fork? It would be great if we could align this reasoning and make something basic like this work alongside our expectations for "visual goodness".

[nataliaElv]

@JonnyTran I'd love to see this in action! What's the best way to try it out?

[DustinEagar]

This feature would be really helpful for my organization - we currently have to rely on a cumbersome workaround to enable review of annotated documents.

[JonnyTran]

@JonnyTran I'd love to see this in action! What's the best way to try it out?

Currently I have an instance running on HF Spaces that have this feature, although it can only view one Suggestion tab (from another annotator’s response) at a time for each Question. Soon I’ll have it so multiple Suggestions can be viewed for each Question.

Let me get back to you next week for a demo. I’m currently out on vacation but would be happy to set up a discord/zoom meeting starting next week