Evaluation, Reproducibility, Benchmarks Meeting 28

Minutes of Meeting 28

Date: 25th September 2024

Nick presented on the overall plan for the dataset review project for lighthouse challenges

The goals are
- Report on quality
- Identify errors that could be addressed
- Ensure the challenges are adhering to their proposals
The project is roughly divided into
- Summary statistics
- Technical review
- Re-annotation
- Expert feedback
Discussion from the group included:
- How do we recruit/incentivize experts to participate?
  - Authorship on challenge report
  - Authorship (or at least acknowledgement) on challenge papers
- Relationship to MONAI
  - This project could serve as a visible example to demonstrate how the MONAI framework can be used for quality checks – should release source code somewhere visible
- Be sure to include a summary of what metadata is made available, and how it is distributed
- It will be challenging to decide at what point we deem a dataset to be “good enough” for lighthouse status, and what will we do if major issues are found?
  - Organizers were made aware that their lighthouse status is contingent upon providing high quality data and adhering to what they proposed
  - If there are challenges where what was proposed is clearly not being followed, we will have to address these with the organizers on a case-by-case basis

There was a discussion of the versioning and development process for the Metrics Reloaded repository
Carole provided an overview of the metrics reloaded software paper
- Ongoing Work
  - Identifying cases where we need tests
  - Cases where we want to warn the user
- Sections to add to paper
  - Want to have tutorials for each use case
  - Need to talk about the impact
- Carole not yet ready to share a draft for feedback, but will when ready
- Carole and Annika will coordinate how tasks may be split between the Carole’s Hackathon and Annika’s community day

Maybe we should have a dedicated planning meeting – use next meeting for this purpose (October 23 – Nick to send a reminder a week in advance)
Roadmap would be helpful for involving others
Prioritize things that will reach more of the community

Copyright (c) MONAI Consortium