Reviewer Instructions 

The effectiveness of the workshop program depends on a knowledgeable, experienced, and ethical group of reviewers. It utilises a two-phase, double-blind review process where submissions are evaluated by a minimum of three reviewers - two of whom must be program committee members, and at least one must be a PhD student. All reviewers adhere to the ethics guidelines set forth by IEEE.

Good Review Practices 

Reviewers assess the trustworthiness, significance, and clarity of manuscripts. The results must be, primarily, accurate (trustworthy) and, secondarily, significant. A manuscript that is deemed trustworthy may be accepted even if it lacks significance. Conversely, a manuscript that is not trustworthy cannot be accepted, regardless of its apparent significance. Manuscripts that possess both trustworthiness and significance may receive priority. Furthermore, manuscripts must be sufficiently clear to facilitate the evaluation of their trustworthiness and significance. Reviewers are encouraged to undertake the following:

• Reflect upon and clearly state their limitations and biases.

• Distinguish between necessary changes and suggested modifications. It is preferable to delineate them distinctly.

• Identify sections of the paper that cannot be accurately evaluated or were not reviewed.

Reviewing Criteria 

The workshop expects significant, impactful, and potentially transformative contributions. Submissions will be evaluated based on their scientific quality and originality, potential impact, the extent to which the evidence substantiates the findings, the suitability of the methodology employed, and the rigour of the research process conducted.

• Originality and level of innovativeness - The degree to which the submission introduces new ideas, methods, perspectives, or insights that are not simply extensions of existing work (15%)

• Description of the research method - Was the research method presented transparently and at an appropriate level of detail? (20%)

• Validity and reasoning - Method appropriateness, validity of methodology relative to published literature or standard practice, reference to the literature, technical correctness, the study is reproducible/replicable, empirical validation, valid sample, discussion of threats to validity, the conclusions follow from the evidence (30%)

• Applicability and impact - Evaluate the relevance, potential influence, and real-world applicability of the work. Consider whether the research has practical value, reflects on its potential impact, includes industrial or societal perspectives, or shows how it could be applied in a meaningful way (20%)

• Clarity of exposition - Assess the quality and clarity of the writing, structure, and overall presentation. Consider how well the paper communicates its contributions, how logically ideas are developed, the rigour of the narrative, and whether the layout and formatting support readability (15%)

In addition to the reviewing criteria, we strongly encourage reviewers and authors to use the general checklist of essential quality attributes to evaluate manuscripts and report research, respectively.

• States a purpose, problem, objective, or research question 

• Explains why the purpose, problem, etc., is important (motivation) 

• Defines jargon, acronyms and key concepts 

• Name the methodology or methodologies used 

• Methodology is appropriate (not necessarily optimal) for the stated purpose, problem, etc. 

• Describes in detail what, where, when and how data were collected 

• Describes in detail how the data were analysed 

• Presents results 

• Results directly address research questions 

• Enumerates and validates assumptions of statistical tests used (if any)

• Discusses implications of the results

• Discloses all major limitations

• States clear conclusions which are linked to the research question (or purpose, etc.) and supported by explicit evidence (data/observations) or arguments 

• Contributes in some way to the collective body of knowledge

• Language is not misleading; any grammatical problems do not substantially hinder understanding

• Acknowledges and mitigates potential risks, harms, burdens or unintended consequences of the research

• Visualisations/graphs are not misleading

• Complies with all applicable empirical standards

These essential quality attributes are necessary conditions for publishing the work. Without a compelling justification, a study that does not meet one or more essential attributes should not be accepted. Whenever a criterion is not met, ask yourself two questions:

• Does deviating from the standards in this way make sense in the context of this study? and

• Would this problem be easy to fix in the camera-ready copy? Justified deviations and easy-to-fix mistakes should not preclude acceptance.

 As a consequence, whether to accept or reject a paper is a consequence of the recommended decisions based on the following categories:

• REJECT if:

  • The paper does not have one or more essential attributes AND

  • One or more deviations are not justified AND

  • One or more unjustified deviations cannot be fixed, or at least not without repeating data collection

• REVISE if:

  • The paper does not have one or more essential attributes AND

  • One or more deviations are not justified AND

  • All of the unjustified deviations can be fixed without repeating data collection

• ACCEPT if:

  • The paper has all the essential attributes OR

  • All deviations are justified or trivially fixed

We strongly encourage authors and reviewers to use this checklist of essential quality criteria because it:

• Makes peer review less biased,

• Reduces review loads by reducing the need for major revisions and overall reviewing work,

• Increases acceptance rate because expectations are know in advance,

• Reduces the reviewers' extraneous cognitive load associated with generating ad hoc criteria for each review,

• Actively prevents reviewers from imposing their own, non-standard criteria, and

• Mitigates the frustration caused by differences between author's and reviewers' expectations,

Form and Shape of Review 

• Constructive criticism: The review should be constructive and give indicators for improvement.

• Factual statements: Statements made by the reviewer should be factual and backed up / justified.

• Clarity: The statements and recommendations should be clear and detailed.

• Transparency: The decision/recommendation to the chair should become evident from the statements and arguments in the review.

• Reasonable suggestions: Suggestions made should be reasonable and feasible.

• Comprehensive: Review should be in-depth rather than being a short, non-helpful statement.

• Soundness: Review should be sound and coherent.

• Professional tone: Review should be written in a professional, non-emotional tone.

Conduct and Attitude 

• Accuracy: The reviewer took care to read the paper and provide a review that is factually correct relative to the contents of the paper

• Fairness: The reviewer should be fair and carefully consider both negative and positive points

• Expertise: The reviewer should have clear expertise on the topic

• Openness: The reviewer should be objective and open to new ideas that might not fit the reviewer's belief system

• Focus on soundness: The reviewer should focus on methodological soundness rather than on actual results

• Honesty: The reviewer should be honest about views and expectations

Reviewing Antipatterns 

Invalid Criticisms