Editors comments:
Although this short paper is rejected both reviewers very much like the work submitted but believe that the paper would be much better re-developed into a full submission which provides full methodological information as well as detailed synthesis of results. Both are essential for a convincing SLR. We recommend that you consider submitting a fresh long version of the paper to IST for review.
Reviewer #1: -> Summary of the submission. This short communication paper presents a grey literature-like study in which 50 internet websites are analyzed in order to extract knowledge on the developer's practices concerned to testing of infrastructure code (IaC). The methodology adopted to pursue the goal is an open-coding procedure conducted by two researchers, which lead to the emergence of six tactics currently used by practitioners for IaC testing.
-> Evaluation. The paper revolves around an extremely timely and relevant problem, which has not been yet investigated by the research community. Given the state of research on infrastructure code, it does make sense to analyze the other side of the spectrum, namely the grey literature available. Despite these valuable points, I think the paper should not really be considered suitable for a short communication. Let me argument my concern. According to the guidelines for preparing short communication submissions [A], IST specifies that:
"Different from full research papers, [a short communication] does not need to cover neither detailed background information nor a comprehensive evaluation. Instead, the focus is on quick publication of preliminary or other results [...]".
While I see the potential value of the research proposed in this paper, I do not believe that the work done can be considered preliminar. For many reasons:
(1) Short communications should not cover "neither detailed background information nor a comprehensive evaluation". On the one hand, infrastructure code quality/testing is a relatively new concept that should be appropriately introduced, for instance by presenting background information on what infrastructure code actually is, why it is relevant nowadays, and why quality aspects are important in this context. On the other hand, the research proposed in this paper represents a substantial step forward the currently available literature - which is, by the way, mostly conducted through grey literature reviews, highlighting the fact that the methodology employed should not be really considered as preliminary - that would deserve much more space than a short communication.
(2) As mentioned above, the methodology is far more elaborated than what expected from a short communication. The paper proposes a systematic grey literature review: this is, up to now, likely the only way to investigate aspects of infrastructure code (given the lack of research on it). By nature, systematic reviews require space to discuss the necessarily details that would enable their replicability. Unfortunately, words/page limitation does not help. This paper is not the expection. The rationale behind the selection of the search query, the stop criterion used when analyzing Google results, the systematic filtering procedure employed to discard unrelevant sources, the quality assessment of the retrieved sources, the analysis methodology: all these aspects would deserve a larger detail that may allow readers to correctly understand the methodology adopted other than replicability of the study.
(3) One of the key points of a systematic review consists of providing an extensive set of implications/actionable items for further research. Again, the page limitation plays against the quality of the paper. In Section 3, a short, concluding paragraph is dedicated to the implications of the work: at the end of the read, what remains unclear is (1) what are the tools available in research that may be useful or even adapted for testing infrastructure code, (2) how is research currently supporting the various challenges mentioned in Section 2, (3) how complete actually are the list of practices described in the paper, (4) what research should do to better support the practitioner's activities, and more.
In other words, the completeness of the paper is questionable because several pieces of information are missing.
In conclusion, let me be clear. I am not against the publication of this piece of work, rather I believe this may be extremely impactful. I only believe that something else would be more appropriate: for instance, the IST special issue on grey literature and multivocal literature reviews [B] would be the perfect fit for a paper like this one.
-> References.
[A] Guidelines for short communication papers: https://www.journals.elsevier.com/information-and-software-technology/short-communications/information-and-software-technology-now-publishing-short-com
Reviewer #2: This is a short communication paper that summarizes practitioners-reported testing practices for infrastructure as code scripts. The research methodology uses a systematic analysis process, and I think it is sound. The topic, testing IaC scripts, is an important topic that deserves further attention, and I think that this paper can have implications in other researchers to consider this topic an important one. I only have a few minor typos as a suggestion for improving the paper:
Abstract; "guideline" -> "guidelines" (I believe it is an uncountable noun). Line 9; remove "." After the footnote. Line 18; "necessitates" -> "necessitate"
I will be looking forward to reading a full extended version of this paper.