Writing a System description Paper for SemEval 2025 Task 11
The content blogpost is based on content that you will find in the references as well as feedback received when running paper writing tutorials for SemEval2023 Task 12 and SemEval2024 Task 1.
Some of these points are general so they can be applied to any paper you write.
In order to be included in our official ranking, you need to write a system description paper.
Note that you will not have to pay any fees for your paper to get published unless you would like to attend the SemEval workshop. In this case, you will have to check the website of the conference with which SemEval will be co-located.
Yes!
We would like to hear from all of you even if your results are negative or if your system does not outperform other systems. Please share details about your model implementation and note that in our task description paper, we will have special mentions of reproducible systems even if they are not the top-performing ones.
Before the system description papers are due, we will share a preprint of our dataset paper. You can use the shared information to analyse your results and conduct ablation studies, further discussion, etc.
SemEval provides some writing tips here. Check also their FAQs page.
Use the ACL template, see https://acl-org.github.io/ACLPUB/formatting.html. Your paper should include 5 pages in the ACL format System paper submissions for a single task and up to 8 pages if you are participating in multiple tasks (E.g., SemEval task 11 and SemEval task 1). Acknowledgments, references, and appendices do NOT count toward page limits.
- The title should be "Team Name at SemEval-2024 Task N: Descriptive Title".
- "at", not "@" "SemEval" and the year are separated by a hyphen and no spaces,
- E.g., "SemEval-2020" task number is followed by a colon not a dash colon that has a space after it but not before it.
- You can write one paper for multiple tasks:
- E.g. 1, ``Team Name at SemEval-YYYY Tasks N1 and N2: Descriptive Title" (2 tasks).
- E.g. 2, ``Team Name at SemEval-YYYY Tasks N1, N2, and N3: Descriptive Title" (3 or more tasks) .
- Papers are not anonymous when submitted.
You should start with an introduction followed by related work, data, model description, results, analysis, and conclusion. Note that this is an example and you can change this structure depending on what you would like to present. In addition, the length of each section depends on what you would like to highlight.
| Section | Points covered |
|---|---|
| Abstract | Summary of your paper. You can highlight the key takes/results. |
| Introduction | - What is the task about? Why is it relevant/important? - What is your approach to solving the problem? - What did you find out or observe while participating in the task? - What are the key results? - What are the struggles? - Have you released the code/model? (You are highly encouraged to do so.) - Add a URL in this case. |
| Related Work/Background | - Summarise the most important details about the task. - Summarise work related to yours (e.g., approaches that inspired you to go with the presented solution). - Situate your work: - Explain how it is new/different. - What does it build on? - See some work on our GitHub repository. (Check references.) E.g., see this paper. - Present key details of publicly available resources used. |
| System Overview | - Key algorithms and modeling decisions/steps in detail. - The resources beyond training data (e.g., lexicons, other data,...). - The maths behind your model and its explanation in plain English. - All the variants of your model |
| Experimental Setup | - Data splits (train/dev/test) and how they are used. - E.g., have you used the dev set to train the final model? - All the details about preprocessing, parameter tuning, ... - You can add more to the appendix. - External tools libraries used for pre/post-processing. |
| Results | Key findings - How well did your system perform according to the official metrics? - How does it rank in the competition? - Any analysis beyond the overall scores? - Positive/negative insight? - Any quantitative analysis and ablation studies. - Any error analysis and negative insights from the results - E.g., did you notice some patterns in the generated false positives/negatives? - Investigate further and report your findings! - Observations about the data. - Human evaluation if any. - E.g., see this paper. |
| Conclusion | - Summary of the work and future work. - E.g., see all previous papers. |
| Ethical considerations (Encouraged) | - Ethical implications of the task/model. - Can your model be misused? If so, how? (Potential misuses?) - Any potential biases? - See https://aclrollingreview.org/ethicsreviewertutorial as well the ethics sheet for emotion analysis. |
| Acknowledgment (Optional) | - Anyone you would like to thank? - Any funding to acknowledge? |
| Appendix (Optional) | - Any figures/examples/results that are not crucial to the main points in your paper but may help. |
-
Examples of good system description papers that we received for SemEval2024 Task 1 organised last year: 1, 2, 3, 4, 5, 6, and others.
-
Further, you may refer to the papers that received best paper awards at previous SemEval workshops:
- Best papers at SemEval 2024: https://semeval.github.io/SemEval2024/awards
- Best papers at SemEval 2023: https://semeval.github.io/SemEval2023/awards
- Best papers at SemEval 2022: https://semeval.github.io/SemEval2022/awards
-
SemEval Guidelines: https://semeval.github.io/system-paper-template.html
-
Writing an abstract for your research paper: https://writing.wisc.edu/handbook/assignments/writing-an-abstract-for-your-research-paper/
-
Presenting your research: Writing NLP papers by Christopher Potts: https://web.stanford.edu/class/cs224u/2021/slides/cs224u-2021-presenting-part2-handout.pdf
-
Ethics Reviewer Tutorial by ARR: https://aclrollingreview.org/ethicsreviewertutorial
-
How to write an okay research paper by Sasha Rush: https://www.youtube.com/watch?v=qNlwVGxkG7Q
-
Tips for Writing NLP Papers by Vered Schwartz: https://medium.com/@vered1986/tips-for-writing-nlp-papers-9c729a2f9e1f