Add plagiarism hook to coderunner questions

[good-moon-oooo]

Similar to MDL-32226 (https://tracker.moodle.org/browse/MDL-32226), this MR add the plagiarism hooks to Coderunner.

[trampgeek]

Hi Chanqihui

Many thanks for your interest in coderunner and your pull request. Sorry for not responding earlier but we're nearing the end of our teaching semester and I have very little spare time for software development.

Sorry, but I'm not prepared to merge what seems to be a significant new feature into the main Moodle 3 branch until I understand it properly and have tested it myself. I know nothing about the plagiarism detection systems in Moodle and I worry about some of the implications of this change. There are approaching 3000 sites with CodeRunner installed and I will be the one held responsible if a new feature causes problems in even 1 or 2 percent of the sites. My usual policy, in fact, is not to push any change to the main branch (either 3.11 or 4.0) until the code has been running for several weeks on our own production system.

Can you clarify some of the following concerns, please:

1. I see that if plagiarism is turned on, as it is for example on our main University Moodle server, it seems that CodeRunner submissions would all be checked. But lots of our CodeRunner questions ask for trivial bits of code where multiple identical submissions are to be expected. Indeed often we ask students to provide just one missing line of code. I certainly wouldn't want plagiarism checks to be applied to those. How could I prevent that?
2. What are the performance implications of this? We have exams with 600 students stitting at a time. What would be the load on the server when the quiz closes?
3. You've titled the pull request "Add check for coderunner questions Add check for coderunner questions turnitin/moodle-plagiarism_turnitin#638". Does this mean it's intended specifically to work with Turnitin? That seems like an inappropriate choice of plagiarism checker as it looks for common chunks of text, which are almost inevitable in programs. It's also trivial to defeat such checkers by simply changing identifiers. Our in house plagiarism checkers looks for structural equivalence, not textual equivalence. Are there specific code plagiarism plugins available that would be more suitable?
4. Can different plagiarism plugins be used for say essay questions and for coding questions?
5. Am I right in thinking that plagiarism checks are being performed on each question answer individually, rather than on an entire quiz submission? I share Tim Hunt's sentiments expressed in https://tracker.moodle.org/browse/MDL-32226 that, at least for the sort of quizzes we use, checks should be made at the quiz level, not the individual question level.
6. Do you have any test code for the new functionality?

I would also like to hear from Tim Hunt, who knows almost infinitely more about this topic area than me, whether this approach is suitable for coding questions. Are you there, Tim? And if so, how would you suggest we proceed with the development? For a new development branch perhaps?

Regards

Richard

[good-moon-oooo]

Hi @trampgeek, thanks for your note. Sorry for the late reply. We were working on this feature with our clients and have tested it on staging, but not in the prod environment yet. I will answer your questions above.

1. Plagiarism checks can be optional in quiz level. When edit coderunner quiz setting, we can choose to enable plagiarism or not for this quiz. But if enabled in quiz setting, all questions in this quiz would be checked.
2. Our client who would like this function has about 19,000+ users on their prod. But this function hasn't gone live. We are not sure about the load on the server for the time being.
3. Currently, this is intended to work with Turnitin (plagiarism/turnitin). If our client or our team requires it to work with other software in the future, we will investigate and develop that. Your thoughts are reasonable. When testing, we found if the code answer is short and simple in some questions, TII is not able to generate the similarity score and produce the report.
4. I haven't used other plagiarism plugins. But the plagiarism turnitin can be manually enabled/disabled for each quiz type and I didn't see a sign it will trouble other plagiarism plugins. So, I guess you can use different ones for different quiz types.
5. The TII plagiarism has a default setting in the site about if enabled or not for quiz/assign/forum, etc. But for each quiz type, you can decide if you keep using the default setting or change it by editing the quiz setting from UI.
6. We haven't written the test code for it.

Hope my answer can clarify things a bit. Thanks for your note again.
Kind regards,
Qihui

[trampgeek]

Hi QihuiChan

Thanks for reporting back and my apologies for not myself being more proactive on the topic.

I had an interesting email exchange with Dan Marsden on this topic and he explained the background and argued along the same lines you. However, I replied back to him:

"After much soul-searching, I've decided to hold off adding the plagiarism hooks to CodeRunner, for now at least. Although you're of course right that if someone uses the available functionality and has performance problems it's not really my problem, I fear that it would inevitably lead to forum postings and requests for improved behaviour, all of which I would have to field. So, it wouldn't be zero cost for me. Furthermore, I have concerns about the whole plagiarism-detection industry and its impacts on tertiary education. I really don't want to encourage its use by making it available through CodeRunner.

"I doubt that the ability to detect directly copied code improves educational outcomes. I've seen too many teachers focus on detecting cheating rather than on getting the best outcomes for the class as a whole. Too much emphasis on cheat-detection poisons a class's attitude towards a teacher and their course and demotivates students. And lastly - how are we going to deal with students who use tools like OpenAI to write code for them? This won't be detected as plagiarism.

"I feel we need to focus more on providing good motivation and rewarding successful outcomes than on trying to stamp out cheating, which is the ambulance at the bottom of the cliff."

However, I realise this is an evolving space, so I'm certainly not saying never, no way. I'm impressed that your client has 19,000 users on their product although that might be just a single university? Please feel free to keep this channel open and report on experiences. I'm assuming that you have no problem supporting this extension yourself and don't need me to merge it into my master.

Regards

Richard